Log Audit Troubleshoot and Recover SSH

SSH incidents are easiest to resolve when authentication, authorization, network state, and service configuration are examined separately. This runbook is for systems you own or administer. It favors read-only inspection first, keeps a second session or local console available, and treats logs as sensitive operational data rather than as an unlimited record of user activity.

Define the evidence and privacy boundary

Record the host, UTC time, software versions, source network, destination address, username, client version, and change ticket. SSH logs can contain usernames, source IP addresses, key IDs, certificate principals, and command or subsystem details; they may be personal data. Limit readers by role, protect log transport, set retention to the approved period, redact copied examples, and do not paste private keys, passwords, tokens, or full home-directory paths into tickets.

date -u
ssh -V
sshd -V 2>&1 || true
hostname --fqdn
ip -br address
ip route
ss -lntp | grep -E '(:22[[:space:]]|sshd)' || true

Command output varies by distribution and privilege. Do not infer that a missing process-list entry proves a service is absent; check the service manager and socket activation configuration next.

Find authentication and service evidence

On systemd systems, start with the journal and narrow by time. On Debian-family systems, authentication records are commonly in /var/log/auth.log; Red Hat-family systems commonly use /var/log/secure. Use the path present on the host rather than creating a second log source:

sudo journalctl -u ssh --since "30 min ago" --no-pager
sudo journalctl -u sshd --since "30 min ago" --no-pager
sudo journalctl _COMM=sshd --since "30 min ago" --no-pager
sudo tail -n 200 /var/log/auth.log 2>/dev/null || sudo tail -n 200 /var/log/secure

Useful patterns include “Failed password” or “Invalid user” (authentication attempts), “Accepted publickey” (successful authentication), “User ... not allowed” (account or policy), “Connection closed” (the session ended), and host-key, certificate, KEX, or algorithm errors (negotiation). Correlate one event with the firewall, bastion, Tailscale, or Netgate record; a source address in a proxy or relay log is not necessarily the original client.

Increase diagnostic detail only during a controlled, short test. A temporary LogLevel VERBOSE or DEBUG1 can reveal key and certificate decisions, but DEBUG levels can expose more metadata and create volume. Set it back to the approved level and document the interval. Protect and rotate logs if sensitive data was collected.

Audit the effective configuration

Inspect the parsed configuration rather than assuming that a later drop-in did not override an earlier line. -C evaluates conditional Match blocks for a representative connection:

sudo sshd -t
sudo sshd -T | sort | less
sudo sshd -T -C user=alice,addr=192.0.2.25,host=server.example.invalid,laddr=192.0.2.10,lport=22 \
  | grep -E '^(allow|deny|authenticationmethods|authorizedkeysfile|authorizedprincipalsfile|trustedusercakeys|revokedkeys|passwordauthentication|kbdinteractiveauthentication|permitrootlogin|pubkeyauthentication|port|listenaddress)'
sudo systemctl cat ssh 2>/dev/null || sudo systemctl cat sshd

Check file ownership and permissions for sshd_config, drop-ins, authorized keys, CA public keys, KRLs, and the user’s .ssh directory. Avoid broad recursive permission changes. On a host that uses an include directory, list the files in lexical order and identify the owner of every change.

Use safe rate limits

First restrict reachability at the network boundary: allow TCP/22 only from the intended management interface, Tailscale path, bastion, or trusted-LAN break-glass host. On the host, use current OpenSSH controls such as MaxStartups, PerSourceMaxStartups, and LoginGraceTime only after checking man sshd_config for the installed release. Keep enough capacity for two administrators and an emergency console workflow.

# Example values require site sizing; place in an approved sshd_config.d drop-in.
MaxStartups 10:30:60
PerSourceMaxStartups 3
LoginGraceTime 30

Do not mistake rate limiting for authorization. A host firewall, a tailnet grant or ACL, valid SSH authorization, and account policy must still deny unapproved sources. Fail2ban or another detector can add temporary bans, but tune it against real logs, exclude the documented break-glass source, protect its state, and test that an administrator cannot lock out every recovery path. Avoid permanent IP allowlists for mobile administrators when an identity-aware private overlay is available.

Troubleshoot by layer

ssh -vvv -o ConnectTimeout=10 user@host
ssh -G user@host | sort
nc -vz -w 5 host 22

Use -vvv only for a controlled test: it can disclose usernames, hostnames, paths, and negotiation details. Redact it before sharing. nc proves a TCP connection only; it does not prove successful SSH authentication.

Recover without creating a new exposure

1. Keep the existing session open and use the local console, hypervisor console, or documented trusted-LAN break-glass host. Do not improvise a public port forward.
2. Copy the current configuration and logs to protected incident storage, record hashes if required, then identify the smallest changed file. Restore the last known-good drop-in only after reviewing the change.
3. Run sshd -t. If valid, reload with systemctl reload ssh or systemctl reload sshd, using the service name shown by the host. If invalid, fix the reported line or restore the known-good copy; do not reboot as a test.
4. Verify two fresh logins: one normal administrator and one explicitly documented recovery account or certificate principal. Then verify password and root login remain at policy values.
5. If credentials may be exposed, revoke the certificate or key, rotate the affected key, remove unauthorized principals, and inspect successful-login logs. Preserve evidence before deleting accounts or logs.

A trusted-LAN break-glass path is a narrow, tested recovery route, not permission for every LAN device. Restrict it by source, account, key, and firewall rule; log use; review it after every incident; and keep the console path independent of the LAN.

Audit and rollback checklist

• Test a permitted source, an unapproved source, an expired or revoked certificate, a disabled account, and a malformed configuration in a maintenance window.
• Check log forwarding, clock synchronization, retention, access controls, and rate-limit counters. Confirm that privacy requests and deletion rules are honored.
• Back up configuration, authorized principals, CA public keys, KRLs, host keys, firewall policy, and recovery notes; encrypt backups and perform a restore test.
• Rollback by restoring the reviewed configuration and firewall state, validating syntax, reloading, and repeating the test matrix. Remove temporary debug logging and rate-limit exceptions.

For certificate-specific issuance and KRL work, see Use OpenSSH Certificates and Key Revocation. For the complete Pi recovery context, see Operate and Recover SSH on the Raspberry Pi 5.

Official references

• Debian OpenSSH sshd manual ↗
• Debian OpenSSH sshd_config manual ↗
• systemd journalctl documentation ↗