Logging Updates Backups and Recovery

A tunnel and a DMZ are only dependable when their operators can explain what happened and restore the intended state. For this homelab, the Netgate 2100 protects LAN4 (192.168.60.0/24), the Pi 5 is 192.168.60.10, Nginx serves 127.0.0.1:8080, and cloudflared is the public web connector. Tailscale is the separate private administration path. Record that boundary in every runbook and backup.

A backup is not a recovery plan. A configuration file that has never been restored proves only that an export completed. Keep encrypted copies, protect credentials separately, and rehearse a restore on hardware or an isolated replacement before an outage makes the exercise urgent.

Collect evidence from every boundary

Source	Useful evidence	Protect it from
Netgate 2100	Rule changes, aliases, DHCP leases, states, blocked DMZ egress, IPv6 events, system and configuration events	Unauthorized firewall access and short local retention
Pi and Nginx	Service starts, package actions, SSH authentication, access and error logs, kernel and storage errors	Log injection, secrets in URLs, and a full disk
cloudflared	Connector registration, tunnel disconnects, ingress selection, authentication and restart events	Exposed tokens and unbounded verbose logs
Tailscale	Device enrollment, key expiry, route and policy changes, connection diagnostics	Stale devices and unauthorized tailnet administrators
DNS and certificate service	Record changes, certificate issuance and renewal, failed validation	Unexpected zone edits and missing AAAA review

Synchronize the Netgate, Pi, and administrator devices to trustworthy time sources. Use UTC in exported logs or record the offset. A remote collector can improve survivability, but a DMZ-to-LAN logging exception must be explicit, authenticated, encrypted where supported, and limited to the collector and port. Never make the DMZ generally reachable just to centralize logs.

Patch in an order that preserves a rollback

Read the vendor and project release notes, support status, and compatibility requirements. Check that the planned versions support the Netgate hardware and Pi operating system.
Export and verify an encrypted Netgate configuration backup. Record the active rule and alias set, installed package versions, and a known-good test result.
Back up application data and configuration, then update the Pi operating system and security packages. Reboot only with console or Tailscale recovery available.
Update Nginx and the application, validate with nginx -t, and test the loopback origin before restarting cloudflared.
Update cloudflared from the official source and check the connector service and token or credentials file permissions. Do not replace a working credential with a token pasted into a shell transcript.
Review Tailscale client and policy changes separately. Verify the administrator device still has the intended least-privilege access.
Run the outside, IPv4/IPv6, private-access, and DMZ-deny tests. Keep the previous package or image available for the documented rollback window.

Do not automatically apply a major firewall, OS, web-server, and connector change at the same moment. One change at a time makes logs, counters, and rollback meaningful.

Back up state and credentials as different classes

Netgate: export an encrypted configuration backup after each approved policy change. Store it away from the firewall and record the software version, hardware model, interface mapping, and restore password in a protected credential manager.
Pi: back up application data, Nginx site configuration, package and service definitions, firewall configuration, scheduled jobs, and a list of installed packages. Exclude caches and regenerateable data. Encrypt the backup before it leaves the DMZ.
Tunnel: protect the named-tunnel credentials file or token separately with least-privilege permissions. A backup operator should not automatically have permission to publish arbitrary routes. Revoke and rotate it if copied to an untrusted location.
Tailscale: retain the tailnet owner and recovery information in the organization’s protected credential process, not in the Pi backup. Remove old device authorizations and avoid exporting machine state unless the current Tailscale recovery procedure requires it.
SSH and application secrets: keep private keys, password-manager recovery codes, database credentials, and API tokens in a dedicated encrypted secret store. Never include them in a public document root, ordinary log archive, or unencrypted configuration export.

Use at least two independently accessible backup copies with documented retention and deletion. Verify that the backup job reports success, has enough space, and produces a file that can be decrypted by the recovery operator—not merely that a scheduler ran.

Run a restore drill

Schedule a maintenance window and record the current known-good state. Disconnect the test replacement from production WAN and LAN, or use isolated hardware and a test DNS name.
Restore the Netgate configuration to the same or supported hardware. Confirm interface assignments, DMZ address, aliases, rule ordering, default-deny behavior, IPv6 posture, and management lockout boundaries before connecting production links.
Reinstall or rebuild the Pi from a supported image. Restore data and configuration without restoring stale host keys or unknown executables; regenerate secrets and certificates when the service requires it.
Enroll Tailscale using the current controlled process and verify only the intended administrator can connect. Restore the tunnel using a rotated or specifically approved credential, then check its ingress catch-all.
Run a local origin test, an external DNS/TLS test, an Access allow/deny test if used, an SSH non-publication test, and DMZ egress-deny tests. Compare logs with the expected evidence.
Document elapsed time, missing dependencies, failed assumptions, and the exact rollback or cleanup. Update the runbook while the details are fresh.

Define recovery priorities

Priority	Restore first	Acceptance check
1	Safe Netgate management and DMZ isolation	Trusted management works; WAN management, WAN SSH, and DMZ-to-LAN paths remain closed.
2	Pi operating system, time, DNS, and updates	Pi is patched, synchronized, and reachable only through the intended local or Tailscale path.
3	Nginx origin and application data	`127.0.0.1:8080` returns the expected response without exposing secrets.
4	Cloudflare Tunnel and DNS	Expected public hostname works; stale A/AAAA records and unrelated hostnames do not.

Keep local console access and an onsite recovery contact available if Tailscale, Cloudflare, DNS, or Starlink is unavailable. A private administration plan that depends on the same public service it is meant to repair is not a recovery plan.

Sequence navigation

Previous: Restrict Netgate 2100 DMZ Egress · Next: Test the Complete Homelab from Outside