Process Management

Understanding processes is what separates people who can actually debug a stuck system from people who just reboot it. Everything running on a Unix box is a process — it has a PID, an owner, a parent, a working directory, and a set of open file descriptors. Once you can see that clearly, a lot of mysterious system behaviour starts making sense.

What's Running: ps and top

$ ps aux             # everything running, BSD-style flags
$ ps -ef             # everything running, POSIX-style
$ ps aux | grep nginx   # find a specific process

The columns that matter: PID (process ID), USER (who owns it), %CPU, %MEM, STAT (process state), and COMMAND.

Process states in the STAT column:

R  running or runnable
S  sleeping (waiting for something)
D  uninterruptible sleep (usually I/O) — this one can be a problem
Z  zombie (finished but parent hasn't collected it yet)
T  stopped

top gives you a live view. htop is friendlier if you have it installed — tree view, mouse support, easier to read.

Process Hierarchy

Every process has a parent. When you open a terminal and run a command, the shell forks a child to run it. PID 1 is the init process (systemd on most modern Linux) — it's the ancestor of everything.

$ pstree -p           # show the tree with PIDs
$ ps -o pid,ppid,cmd  # show PID and parent PID

Signals

Signals are how you communicate with a running process. Most people only know kill -9, which works but is the sledgehammer — it can't be caught or ignored, so the process has no chance to clean up.

$ kill -SIGTERM 1234    # polite termination, process can handle it
$ kill -SIGKILL 1234    # force kill, cannot be ignored
$ kill -SIGHUP 1234     # hangup — often causes daemons to reload config
$ kill -SIGSTOP 1234    # pause a process
$ kill -SIGCONT 1234    # resume a paused process

You can also use numeric signals: kill -15 (SIGTERM), kill -9 (SIGKILL), kill -1 (SIGHUP).

$ pkill nginx           # kill by name instead of PID
$ killall firefox       # kill all processes with this name

Signal handling in a shell script — trap specific signals to do cleanup before dying:

trap 'echo "Caught SIGINT, cleaning up..."; rm -f /tmp/mylock; exit 1' SIGINT SIGTERM

Job Control

Job control is how you manage multiple things in a single shell session. More useful than most people realize.

$ long_command &        # run in background immediately
$ long_command
^Z                      # Ctrl+Z suspends the foreground process
$ bg                    # continue it in background
$ fg                    # bring it back to foreground
$ jobs                  # list all jobs in this session
$ fg %2                 # bring job 2 to foreground specifically

Jobs are per-shell-session. Once you close the terminal they're gone, even if the processes keep running. Use nohup or a terminal multiplexer (tmux, screen) if you need things to outlive the session.

$ nohup ./myscript.sh &     # run immune to hangup, output to nohup.out
$ disown %1                 # detach job 1 from the shell without nohup

Finding and Killing Stuck Processes

The workflow when something is misbehaving:

# Find by name
$ pgrep -l nginx
$ pidof nginx

# Find what's using a port
$ lsof -i :8080
$ ss -tlnp | grep 8080

# Find what's using a file
$ lsof /var/log/app.log
$ fuser /mnt/usb         # useful when you can't unmount something

Processes in state D (uninterruptible sleep) are usually stuck waiting on I/O — typically a network filesystem or a dying disk. You often can't kill them; sometimes you have to reboot. kill -9 won't work on a D-state process.

Monitoring Over Time

$ top                   # live CPU/memory overview
$ htop                  # nicer version of top
$ vmstat 1              # system stats every second
$ iostat -x 1           # I/O stats per device, every second
$ sar -u 1 10           # CPU usage, 10 samples at 1-second intervals

When a system is slow, the sequence I usually follow: check top for obvious CPU/memory hogs, then iostat to see if it's I/O-bound, then lsof if I suspect a specific process is holding resources.

Process Priority: nice and renice

Nice values range from -20 (highest priority) to 19 (lowest). Regular users can only lower priority (raise the nice value). Root can raise priority.

$ nice -n 10 ./heavy_job.sh      # start a process with lower priority
$ renice +10 -p 1234             # lower priority of a running process
$ renice -5 -p 1234              # raise priority (needs root)

Useful when you're running something CPU-intensive and don't want it to murder interactive performance.