Working with Files and Text

Unix tools are especially effective when a file is plain text: a sequence of bytes interpreted as characters and separated into lines. The same tools exist across POSIX-oriented Unix systems, Linux distributions, the BSDs, and macOS, but GNU utilities sometimes have options that BSD/macOS utilities do not. The commands here use common forms; check man locally before depending on less common flags.

Bytes, text, and terminals

Every file is bytes. “Text” means software can interpret those bytes as characters and line breaks. UTF-8 is a common encoding, but it is not the only one. An image, archive, executable, or random data file can contain control bytes that alter a terminal display, so do not assume every file is safe to print.

$ file -- report.txt
$ file -- download
$ cat report.txt             # suitable only when it is short, known text

cat concatenates files and writes them to standard output. It is handy for a short, known text file or when joining files, but it is not a general viewer. For longer files use a pager:

$ less -- report.txt

In less, press Space to move forward, b to move back, /word to search, n for the next match, and q to quit. Many systems also provide more, but less is usually more capable. If it is not installed, use more or install a system package through your normal administrator-approved process.

Looking at parts and counting

Use head for the beginning and tail for the end. POSIX supports -n number; the old shorthand such as -20 is less clear.

$ head -n 5 -- access.log
$ tail -n 20 -- access.log
$ wc -- lines.txt
      42      98     611 lines.txt
$ wc -l -- lines.txt          # count newline-terminated lines

wc reports lines, words, and bytes by default. A final line without a newline may be readable text but is not counted by wc -l. To watch a log as lines are appended, tail -f -- log.txt is widely available; stop with Ctrl-C. Only follow logs you are allowed to read, and remember that a rotated log may require reopening.

Sorting and removing adjacent duplicates

sort orders lines. uniq only groups adjacent equal lines, so sort first when the input is unsorted.

$ sort -- names.txt
$ sort -- names.txt | uniq
$ sort -- names.txt | uniq -c
$ sort -u -- names.txt

Output order and locale-sensitive collation can vary by system and language settings. If a script needs bytewise, reproducible ordering, set an appropriate locale intentionally (often LC_ALL=C) after understanding the impact on non-ASCII text.

Selecting and translating fields

cut extracts simple delimiter-separated fields or character positions. It is not a CSV parser: quoted commas and embedded newlines require a CSV-aware tool.

$ cut -d ':' -f 1,3 -- users.txt
$ cut -c 1-20 -- short-lines.txt
$ tr '[:lower:]' '[:upper:]' < title.txt
$ tr -d '\r' < windows-lines.txt > unix-lines.txt

The first command prints fields 1 and 3 from colon-separated lines. tr transforms or deletes individual characters; it does not understand words or regular expressions. Removing carriage returns can convert CRLF line endings to LF, but make a copy first if preserving the original format matters.

Searching with grep

grep prints lines that match a pattern. Quote patterns so the shell does not reinterpret characters such as *, spaces, or brackets.

$ grep -n -- 'error' application.log
$ grep -i -- 'warning' application.log
$ grep -F -- 'price (USD)' catalogue.txt
$ grep -E -- '^(WARN|ERROR):' application.log

Without a pattern option, grep uses basic regular expressions (BRE). In BRE, characters such as ., *, ^, $, and bracket expressions have special meanings, while grouping and alternation are spelled differently than in extended regular expressions. grep -E selects extended regular expressions (ERE), where |, +, ?, and parentheses are useful. grep -F performs a fixed-string search: it does not interpret regex metacharacters and is the right choice for a literal phrase.

grep -r recursively searches a tree on common systems, but its link-following and exclusion options vary. Start in a small directory you own and avoid treating filenames as line-oriented data. For a status check in a script, use grep -q -- 'pattern' file; it produces no matching lines and communicates success through its exit status.

Small, reviewable edits with sed

sed edits a stream. By default it writes transformed text to standard output and does not alter the input file, which makes it safe for a preview.

$ sed 's/draft/final/' -- letter.txt
$ sed 's/draft/final/g' -- letter.txt
$ sed -n '1,10p' -- letter.txt
$ sed 's/draft/final/g' -- letter.txt > letter-new.txt

The first substitution changes the first draft on each line; the g flag changes every occurrence on a line. The -n form suppresses normal output, then explicitly prints lines 1 through 10. In-place editing is deliberately omitted here: GNU and BSD/macOS sed -i use incompatible backup-argument conventions. Writing a new file, reviewing it with diff, then replacing deliberately is portable and safer.

Compare before replacing

diff shows line-level differences between two text files and returns a nonzero status when they differ. That nonzero result is useful in a terminal but must be accounted for in scripts.

$ diff -u -- settings.old settings.new
$ diff -u -- original.txt edited.txt > changes.patch

The unified format from -u gives surrounding context and is easier to review. A difference is not an error; a read failure is. For important work, preserve the original or use version control rather than assuming a text transformation is reversible.

tee: save a copy of a pipeline

tee copies standard input to standard output and to a file. It is useful when you want to see pipeline output while recording it.

$ grep -F -- 'ERROR' application.log | tee errors.txt
$ printf '%s\n' "checked $(date)" | tee -a review.log

-a appends; without it, tee truncates the output file. Keep output paths under a directory you own. Do not use privileged redirection tricks to edit system configuration; make and review a normal user-owned copy, then follow your system’s documented administration process if a change is justified.

Choose an editor you can use

Editors are interactive programs, so the best beginner choice is the one available on your system and comfortable for you. Terminal choices may include vi/vim, nano, or emacs; graphical editors are also fine. Before saving, check the filename and directory. For a configuration or valuable document, save a new version, inspect it, and compare it with diff. Do not edit binary files with a text editor.

Encoding, newlines, and unusual filenames

Unix text traditionally uses LF (\n) to end lines; files from Windows commonly use CRLF (\r\n). Modern macOS and Unix use LF. An editor may detect and convert line endings, but automatic conversion can be undesirable for generated files or protocols. UTF-8 is common on current systems, while older data may use another encoding. If text looks garbled, identify the file and encoding before bulk conversion; never blindly rewrite a source file.

Spaces, tabs, newlines, and a leading dash are legal in Unix filenames. Always quote a literal path and use -- where the utility supports it:

$ less -- "meeting notes.txt"
$ mv -- "meeting notes.txt" "meeting-notes.txt"
$ rm -i -- "-old.txt"

-- marks the end of options, preventing -old.txt from being parsed as an option. It is common in modern GNU, BSD, and macOS utilities, but not mandated for every POSIX utility; ./-old.txt is a practical alternative. Avoid loops such as for f in $(ls): they break on whitespace and other valid names. Prefer shell globbing with quoted variable uses, or tools designed to pass paths safely.

Safe exercise: make a small inventory and log

Create a uniquely named disposable workspace. This exercise reads only files it creates, writes only there, and does not require sudo. Run the commands individually, then inspect each output.

$ (
> work=$(mktemp -d "${TMPDIR:-/tmp}/text-tools.XXXXXX") || exit 1
> [ -d "$work" ] || exit 1
> printf 'Practice directory: %s\n' "$work"
> cd "$work" || exit 1
> printf '%s\n' "apple" "pear" "apple" "orange" > fruit.txt
> printf '%s\n' "INFO: started" "WARN: low space" "ERROR: retry" "ERROR: retry" > service.log
> sort -- fruit.txt | uniq -c | tee fruit-counts.txt
> grep -E -n -- '^(WARN|ERROR):' service.log | tee attention.log
> cut -d ':' -f 1 -- service.log | sort | uniq -c > level-counts.txt
> sed 's/retry/retry later/g' -- service.log > service-reviewed.log
> diff -u -- service.log service-reviewed.log
> wc -l -- *.txt *.log
> less -- attention.log

Notice that the quoted '^(WARN|ERROR):' reaches grep intact, tee both displays and saves results, and diff lets you review an edit without overwriting the source. When finished, leave the workspace first and remove it only if you have inspected the target:

> cd "$HOME" || exit 1
> rm -ri -- "$work"
> )

The subshell stops the lab safely if temporary-directory creation or cd fails. The unique path prevents this exercise from silently reusing an older directory. The interactive recursive removal is intentional: read the path it prints and answer only for the practice directory. Keeping the directory is equally valid. mktemp is common but not POSIX; check its local manual on unusual systems.

A practical checklist

  • Use file before displaying an unfamiliar file.
  • Use less for reading, and head/tail for a bounded view.
  • Choose grep -F for literal text; choose BRE or grep -E only when you intend a regular expression.
  • Write transformed output to a new file and compare it before replacing anything.
  • Quote paths and variables, and protect leading-dash filenames with -- or ./.