Git: Internals — Index, Pack Files, and Protocols

This article covers Git's internal storage formats and network protocols. It's not required knowledge for day-to-day Git use, but understanding it helps when debugging unusual situations, optimizing large repositories, or working with Git's transfer layer. It also covers all remaining low-level commands not covered elsewhere.

Commands Covered

git update-index · git pack-objects · git unpack-objects · git index-pack · git verify-pack · git commit-graph · git multi-pack-index · git count-objects · git bugreport · git diagnose · git scalar · git backfill

The Index (Staging Area)

Git's core data flow. The Index (staging area) acts as a buffer between the local Working Tree and the permanent Object Database, which is then synchronized with Remotes via pack files over network protocols.

The index is a binary file at .git/index that records the state of the working tree that will go into the next commit. It stores mode, SHA, and path for every tracked file, and during merges it holds all three stages (base, ours, theirs) for each conflicted file.

git update-index

The low-level command for modifying the index directly. Rarely used — git add and git rm call it internally.

# Mark a file as "assume unchanged" — Git stops checking it for modifications
# (useful for large config files you need to keep but not commit changes to)
$ git update-index --assume-unchanged config/local.php
# Undo:
$ git update-index --no-assume-unchanged config/local.php

# Mark a file as "skip worktree" — similar but for sparse checkouts and
# intentionally modified tracked files you don't want to commit
$ git update-index --skip-worktree config/local.php
$ git update-index --no-skip-worktree config/local.php

# Force-add a file even if it matches .gitignore
$ git update-index --add ignored-but-needed.txt

# Refresh the index (recheck mtime of all files without hashing content)
$ git update-index --refresh

# Re-add a file using an already-stored blob SHA
$ git update-index --cacheinfo 100644 BLOB_SHA path/to/file

# Show what's currently in the index
$ git ls-files --stage

Pack Files

Git stores objects in two ways: as loose files (.git/objects/ab/cdef...) and as pack files (.git/objects/pack/*.pack). Pack files compress multiple objects together using delta compression — storing only the differences between similar objects. Large repos use pack files almost exclusively.

git pack-objects

# Pack all loose objects into a pack file
$ git pack-objects --all .git/objects/pack/my-pack < /dev/null
# Creates my-pack.pack and my-pack.idx

# Pack objects reachable from HEAD with delta compression
$ git rev-list --all | git pack-objects --stdout > repo.pack

# Pack with maximum compression (slow)
$ git pack-objects -q --compression=9 .git/objects/pack/compressed \
    <<< $(git rev-list --all)

git unpack-objects

# Unpack objects from a pack file into loose objects
$ git unpack-objects < repo.pack
# Useful for inspecting pack contents or importing objects

git index-pack

# Build an index file (.idx) for an existing pack file
$ git index-pack .git/objects/pack/repo.pack
# Creates repo.idx — the index that allows fast lookup inside the pack

# Verify a pack and its index
$ git index-pack --verify .git/objects/pack/repo.pack

git verify-pack

# Verify a pack file's integrity
$ git verify-pack .git/objects/pack/pack-*.pack

# Verbose — list all objects in the pack with sizes
$ git verify-pack -v .git/objects/pack/pack-*.pack

# Find the largest objects in the pack (useful for diagnosing repo bloat)
$ git verify-pack -v .git/objects/pack/pack-*.pack \
    | sort -k3 -n \
    | tail -20 \
    | awk '{print $1}' \
    | xargs git cat-file -p \
    | head

# Find the 10 largest objects and their paths
$ git verify-pack -v .git/objects/pack/pack-*.pack \
    | sort -k3 -n \
    | tail -10 \
    | awk '{print $1}' \
    | while read sha; do
        git log --all --oneline --find-object=$sha 2>/dev/null | head -1
    done

git commit-graph

Writes a commit-graph file (.git/objects/info/commit-graph) that caches commit metadata for fast graph traversal. Significantly speeds up operations like git log and git merge-base in large repos.

# Write the commit-graph
$ git commit-graph write

# Write with reachability index (makes some operations even faster)
$ git commit-graph write --reachable

# Incrementally update (only adds new commits)
$ git commit-graph write --reachable --changed-paths

# Verify the commit-graph
$ git commit-graph verify

# Read info from the commit-graph
$ git commit-graph read

git multi-pack-index

Writes a multi-pack-index (MIDX) file that indexes objects across multiple pack files. Makes object lookup fast without repacking everything into one file.

# Write the MIDX
$ git multi-pack-index write

# Verify the MIDX
$ git multi-pack-index verify

# Expire packs not referenced by MIDX
$ git multi-pack-index expire

# Repack based on MIDX coverage
$ git multi-pack-index repack

git count-objects

# Count loose objects and their size
$ git count-objects

# Verbose — includes pack file statistics
$ git count-objects -v
# count:        0         (loose objects)
# size:         0         (KB in loose objects)
# in-pack:      12345     (objects in pack files)
# packs:        3         (number of pack files)
# size-pack:    45678     (KB in pack files)
# prune-packable: 0       (loose objects duplicated in packs)
# garbage:      0         (garbage files in object directory)
# size-garbage: 0

# Human-readable sizes
$ git count-objects -vH

Git Transfer Protocols

Git supports several protocols for transferring data between repositories:

Protocol	URL format	Notes
Local	`/path/to/repo` or `file:///path`	Fastest; same machine only
SSH	`git@host:user/repo.git`	Authenticated; most common for private repos
HTTPS	`https://host/user/repo.git`	Works through firewalls; uses username/password or token
Git protocol	`git://host/repo.git`	Unauthenticated; fast; rarely used now (port 9418)

git daemon

# Run a simple Git server (unauthenticated read-only access)
$ git daemon --reuseaddr --base-path=/srv/git /srv/git/
# Clients can clone: git clone git://yourhost/project.git

# For a single repo to be exportable, create a magic file:
$ touch /srv/git/project.git/git-daemon-export-ok

# Run for export only (no push)
$ git daemon --export-all --base-path=/srv/git /srv/git/

git http-backend

# git http-backend implements the "smart" HTTP protocol.
# It's a CGI program — configure with nginx or Apache.

# Nginx config snippet:
# location ~ /git(/.*) {
#     fastcgi_pass  unix:/var/run/fcgiwrap.socket;
#     fastcgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend;
#     fastcgi_param GIT_HTTP_EXPORT_ALL "";
#     fastcgi_param GIT_PROJECT_ROOT /srv/git;
#     fastcgi_param PATH_INFO $1;
#     include fastcgi_params;
# }

Protocol Version 2

# Git protocol v2 (default since Git 2.26) is more efficient:
# - Server only sends refs the client asks for (not all refs)
# - Significant speedup for repos with many branches/tags

# Enable explicitly (usually already default):
$ git config --global protocol.version 2

# Check which version is being used:
$ GIT_TRACE_PACKET=1 git ls-remote origin 2>&1 | head -5
# Look for: packet:          git< version 2

git scalar

A tool for managing very large Git repositories. It configures several performance-related settings in one shot: partial clone, sparse checkout, commit-graph, maintenance, MIDX, and more.

# Register a repo with scalar (enables all optimizations)
$ scalar register

# Clone a large repo with all scalar optimizations
$ scalar clone https://github.com/microsoft/vscode.git

# Run scalar maintenance
$ scalar run all

# Reconfigure an existing repo
$ scalar reconfigure

# What scalar enables:
# - git maintenance start (background maintenance)
# - commit-graph with reachability index
# - multi-pack-index
# - fetch.writeCommitGraph = true
# - index.threads = true
# - pack.useBitmaps = true (faster clone/fetch)

git backfill

Downloads missing objects in a partial clone (a clone made with --filter=blob:none or similar). Partial clones skip downloading certain objects; backfill retrieves them on demand or in bulk.

# Create a partial clone (no blobs downloaded)
$ git clone --filter=blob:none https://github.com/large/repo.git

# Later, backfill all missing blobs
$ git backfill
$ git backfill --min-batch-size=50000   # fetch in large batches

# Backfill for specific paths only
$ git backfill -- src/

git bugreport and git diagnose

# Generate a bug report with system info (for filing Git issues)
$ git bugreport
# Creates: git-bugreport-YYYY-MM-DD-HHMMSS.txt

# Generate a comprehensive diagnostic zip archive
$ git diagnose
# Creates: git-diagnostics-YYYY-MM-DD-HHMMSS.zip
# Contains: version info, config, pack info, ref info, fsck output