Run a Persistent Reverse SSH Tunnel Safely

A persistent tunnel is a service with an owner, a bounded destination, a restart policy, and observable failure. It is not a command copied into a shell profile. Prove the foreground tunnel first, then supervise it with the platform’s service manager. The examples use systemd and keep the remote listener on bastion loopback.

Persistence increases both availability and incident impact. Use a dedicated unprivileged local identity, a dedicated bastion account and key, host-key verification, short-lived access approval, narrow egress, and a tested disable procedure. Never place a private key, token, passphrase, or secret in a unit file, environment value, command line, or log.

Separate service identity from administrator identity

Create a local account used only by this service and a remote forwarding account used only on the bastion. Neither account needs a login shell, sudo, agent forwarding, X11 forwarding, or a personal administrator’s key. Keep the private key in a protected file or an approved host secret store; a systemd unit should refer to a path or credential mechanism, not contain key material.

getent passwd ssh-tunnel
sudo -u ssh-tunnel ssh -V
sudo -u ssh-tunnel ssh -G tunnel-bastion \
  | grep -E '^(hostname|user|identityfile|userknownhostsfile|exitonforwardfailure|serveralive)'

Use the installed OpenSSH client’s path and service conventions. On some systems the binary is not /usr/bin/ssh; discover it with command -v ssh and set the unit accordingly. Do not copy a private key into /etc/systemd/system.

Put non-secret SSH policy in a config file

# /etc/ssh/ssh_config.d/tunnel-bastion.conf
Host tunnel-bastion
    HostName bastion.example.net
    User tunnel
    IdentityFile /var/lib/ssh-tunnel/.ssh/id_ed25519
    IdentitiesOnly yes
    UserKnownHostsFile /var/lib/ssh-tunnel/.ssh/known_hosts
    StrictHostKeyChecking yes
    ForwardAgent no
    RequestTTY no
    ExitOnForwardFailure yes
    ServerAliveInterval 30
    ServerAliveCountMax 3

Use a service-specific config path if the distribution does not load the global drop-in. Protect it from unprivileged edits. Install the bastion host key through an out-of-band verification process; do not generate a “known host” entry from an unauthenticated first connection.

Use a system service with least privilege

After the foreground test, create a unit appropriate to the installed systemd release:

# /etc/systemd/system/ssh-reverse-tunnel.service
[Unit]
Description=Approved loopback-only reverse SSH tunnel
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=ssh-tunnel
Group=ssh-tunnel
ExecStart=/usr/bin/ssh -F /etc/ssh/ssh_config -N -T \
  -R 127.0.0.1:2222:127.0.0.1:22 tunnel-bastion
Restart=on-failure
RestartSec=15s
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ProtectControlGroups=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

[Install]
WantedBy=multi-user.target

The unit contains no private key or secret. Confirm that ProtectHome, file modes, and any distribution sandboxing still allow the service to read only its intended SSH config, host-key file, and private key. Some systemd versions do not support every hardening directive; run systemd-analyze verify, remove only unsupported directives with an explicit review, and retain the account, key, destination, and network restrictions.

sudo systemd-analyze verify /etc/systemd/system/ssh-reverse-tunnel.service
sudo systemctl daemon-reload
sudo systemctl enable --now ssh-reverse-tunnel.service
sudo systemctl status ssh-reverse-tunnel.service --no-pager
ss -lntp | grep ':2222'

Use the service name that the distribution documents. Verify both the service log and the bastion log; a running process is not evidence that the requested forward exists. Keep ExitOnForwardFailure=yes in the effective client configuration so a restart cannot report success while missing its listener.

Consider a user service carefully

A systemd user service can be suitable when the tunnel belongs to one logged-in, non-root user and lingering is an intentional policy. It must still use a dedicated identity and protected key:

# ~/.config/systemd/user/ssh-reverse-tunnel.service
[Unit]
After=network-online.target

[Service]
ExecStart=/usr/bin/ssh -F %h/.ssh/config -N -T \
  -R 127.0.0.1:2222:127.0.0.1:22 tunnel-bastion
Restart=on-failure
RestartSec=15s
NoNewPrivileges=yes

[Install]
WantedBy=default.target

Run systemctl --user as that user. Do not enable lingering merely to make an undocumented tunnel survive logout; if lingering is required, document who can stop it, where the key is stored, and how it is audited. A system service is generally easier to inventory and revoke for a machine-level tunnel.

Use AutoSSH only when it solves a reviewed gap

AutoSSH is an optional maintained dependency, not a magic security layer. systemd’s restart policy is often sufficient. If a supported distribution package and current upstream documentation justify AutoSSH, use it with the same OpenSSH policy and keep systemd as the supervisor; do not use a second competing watchdog without a reason. Check the installed version and its manual before selecting monitor options:

command -v autossh
autossh -V
man autossh

Never hide keys in AUTOSSH_* environment variables or add an unreviewed monitor port. Pin and patch the package through the normal update process, record its owner, and test the failure and shutdown paths.

Observe health without leaking secrets

  • Watch systemctl status, journalctl -u ssh-reverse-tunnel.service, sshd authentication/forwarding logs, listener state, and the private host’s egress counters.
  • Alert on repeated reconnects, changed bastion host keys, a listener outside 127.0.0.1, a new destination, an unexpected process owner, or a service running after its expiry.
  • Use rate limits at the bastion and egress firewall. MaxStartups protects pre-authentication sshd capacity, not application traffic through an established tunnel; current OpenSSH releases may also provide per-source controls, which must be checked in the installed manual.
  • Keep logs access-controlled and redact key paths, account details, and application data before sharing diagnostics. Never log command lines that include private material.

Rotate and stop deliberately

  1. Schedule a short validity period and a change owner. Before rotation, add the replacement public key with the same restrictions and test it.
  2. Stop and disable the service, remove the old key from the bastion and private host, and verify the remote listener disappears.
  3. For an emergency, stop the service, revoke the bastion key or certificate, terminate the associated SSH session, block the destination egress, preserve logs, and investigate before reconnecting.
  4. After reboot, network loss, bastion restart, and destination failure, verify that the service reconnects only with the intended listener and does not broaden its bind or destination.

Continue with Use a Bastion Host for Reverse SSH Access Behind CGNAT for the Starlink and DMZ scenario.

Sequence navigation

Previous: Build a Loopback-Only Reverse SSH Tunnel · Next: Use a Bastion Host for Reverse SSH Access Behind CGNAT

Official references