x86-64 System V and Windows ABI Practice

An ABI, or application binary interface, is the agreement that lets separately compiled code cooperate. It covers argument locations, return values, preserved registers, stack alignment, data layout, and unwinding—not merely “which register holds argument one.” This lesson compares the System V AMD64 ABI used by many Unix-like x86-64 systems with Microsoft’s x64 ABI for 64-bit Windows. “x64” in the latter name is Microsoft’s conventional label for the x86-64/AMD64/Intel 64 instruction family; it is not a claim that 64-bit x86 is an unrelated architecture.

Start with the function type

Before reading registers, establish the declaration, target, compiler options, and language. Integer/pointer and floating-point parameters use different register classes; aggregates may be split, passed indirectly, or use special rules; C++ methods and exceptions add language conventions. The following table is a useful entry point for simple, non-variadic scalar calls, not a replacement for either ABI specification.

Item	System V AMD64	Microsoft x64
Integer/pointer argument registers	RDI, RSI, RDX, RCX, R8, R9	RCX, RDX, R8, R9
Floating scalar argument registers	XMM0–XMM7	XMM0–XMM3, position-correlated with first four slots
Integer/pointer result	RAX	RAX
Typical FP scalar result	XMM0	XMM0
Stack at a call boundary	16-byte alignment requirement	16-byte alignment requirement outside prolog/epilog
Special caller stack area	None reserved for every call	32-byte shadow space/home area

For example, an ordinary long combine(long a, long b, long c) normally receives a, b, c in RDI, RSI, RDX under SysV and RCX, RDX, R8 under Microsoft x64. In both, a scalar integer result uses RAX. It follows that this illustrative output has a target-specific meaning:

; SysV AMD64, owned example: return a + 2*b + c
lea     rax, [rdi + rsi*2]
add     rax, rdx
ret

; Microsoft x64, same source-level prototype
lea     rax, [rcx + rdx*2]
add     rax, r8
ret

Neither snippet proves an ABI by itself: a function may be inlined, use a custom convention, or have its signature changed by a language feature. But the mapping gives a concrete hypothesis to check against symbols and source.

Volatile and preserved state

A caller may expect nonvolatile (callee-saved) registers to retain their values across a call. A callee that uses one must save it and restore it in an ABI-conforming way. Under SysV AMD64, RBX, RBP, and R12–R15 are commonly preserved general-purpose registers; Microsoft x64 preserves RBX, RBP, RDI, RSI, RSP, and R12–R15. Microsoft x64 also treats XMM6–XMM15 as nonvolatile, while SysV normally treats XMM registers as caller-saved. RSP is necessarily preserved in the sense that a function must return with the caller’s stack pointer restored.

These lists are why a disassembly may begin with pushes or stack stores that seem unrelated to local variables. They may protect a value on behalf of the caller. Conversely, a small leaf function can use volatile registers and return with no frame at all. Do not demand a frame pointer: modern optimization commonly omits RBP as a frame pointer, and source-level locals can live in registers or disappear.

Alignment, red zone, and shadow space

The stack convention is a contract at calls. Because call pushes an 8-byte return address, a callee’s entry alignment and the caller’s pre-call alignment are related but not identical. Compilers arrange their allocation so calls they make meet the ABI. A hand-written boundary must follow the specification exactly, including stack arguments and padding; subtracting a plausible-looking number is not sufficient.

System V AMD64 gives leaf functions a 128-byte red zone below the current RSP. In normal user-mode ABI conditions, asynchronous signal handlers do not overwrite it, so a leaf function may use it without moving RSP. It is not a general spare stack area: code that makes a call cannot leave live local state there expecting the callee to respect it, and kernels, interrupt contexts, and different targets have their own rules. Toolchain options can disable it.

Microsoft x64 instead requires each caller to reserve 32 bytes of shadow space for the first four register arguments before calling. The callee may home those arguments there. This reservation exists even if the callee never stores them, and it belongs to the caller’s call frame. It is not the SysV red zone and should never be substituted for it. Additional stack arguments follow the home area according to the ABI’s layout and alignment requirements.

Returns, aggregates, and hidden parameters

Simple integer returns use RAX and simple floating returns commonly use XMM0. Wider or aggregate results may use register pairs, classification rules, or a hidden caller-provided result address. A function that appears to take one source-level argument may therefore receive another machine-level argument. The SysV ABI classifies aggregates in eight-byte chunks; Microsoft x64 has its own size- and type-based handling. Read the ABI’s aggregate chapter and the compiler’s generated declarations before writing an interoperation wrapper.

Returning from a function also means restoring the stack and preserved state, then executing ret in ordinary code. A return instruction does not by itself make arbitrary control transfers safe or ABI-correct. Compiler-generated control-flow protections, stack probes, and cleanup are observable implementation choices that should be retained in production builds.

Unwinding is metadata plus disciplined prologs

Exceptions, debuggers, profilers, and crash reporters need to recover callers even when code omitted a frame pointer. ELF toolchains commonly emit DWARF call-frame information, often in .eh_frame, describing how to compute the canonical frame address and saved registers. Windows PE/COFF uses structured unwind information, notably .pdata and .xdata, associated with constrained prolog and epilog forms. The Windows requirements are particularly important: arbitrary hand-written stack manipulation may be impossible for the system unwinder to describe.

For portable owned code, prefer the compiler’s normal function generation. If a project genuinely requires assembly, use its platform assembler directives and verify the emitted unwind metadata with the platform’s documented tools. Do not “test” unwinding by intentionally corrupting stacks or executing unknown binaries. A clean normal exception or debugger backtrace in a test program you own is sufficient evidence for a learning exercise.

Varargs: where simple tables stop

Variadic functions require the callee to locate arguments whose types are known only through a format or protocol. SysV AMD64 has a register save area and a va_list structure tracking general-purpose and floating register consumption as well as stack overflow arguments. A notable call-site detail is that AL communicates the number of vector registers used for certain variadic calls. Microsoft x64 uses its slot convention and requires floating-point values in the corresponding general-purpose register as well as XMM register for unprototyped or variadic calls. These rules are why casting a variadic function pointer or inventing a call sequence is unsafe.

Use the language’s va_start, va_arg, and va_end facilities in C, and pass a correctly declared prototype at every call site. Format-string APIs deserve extra care: pass a literal format whenever practical and match the conversion specifier to the argument type. ABI knowledge explains the generated code; it is not a reason to bypass type checking.

Tail calls provide another useful caution. A compiler may replace a call followed immediately by a compatible return with a jump, but only when it can preserve the observable ABI obligations. Stack arguments, cleanup, instrumentation, visibility, exception behavior, and optimization settings can prevent that transformation. Seeing a jmp near the end of a function is therefore a clue to investigate, not proof that a source call vanished incorrectly. Likewise, register allocation can move a value through several volatile registers before a call; the ABI constrains the boundary, not every temporary inside a function.

Cross-language interfaces should start from an explicitly documented C-compatible boundary. Confirm integer widths, structure layout, ownership, callback lifetime, error handling, and whether the compiler supports the requested calling-convention annotation on the selected target. The same source spelling can be ignored, rejected, or have target-specific meaning. A tiny compile-and-inspect test is useful, but a supported foreign-function interface is safer than reproducing compiler output by hand.

Practice workflow

Create an owned file containing one fixed-argument function, one function returning a small struct, and one variadic wrapper that delegates to the standard library safely. Compile only the fixed function first with clang -target x86_64-pc-linux-gnu -O1 -S -masm=intel and, where a suitable SDK is installed, clang -target x86_64-pc-windows-msvc -O1 -S -masm=intel. Treat the second command as a cross-compilation observation, not an invitation to link foreign system libraries. Label argument registers, stack adjustment, and every saved register. Then compare metadata: readelf --debug-dump=frames file.o on ELF and Microsoft’s documented dump tools on a PE/COFF object.

Exercises

Write a two-argument pure C function and make a two-column annotation of its SysV and Microsoft x64 register locations.
Find a compiler-produced function that saves one nonvolatile register. Identify the save, restore, and the ABI rule requiring them.
Explain in one sentence why a SysV leaf function’s red-zone use cannot be copied into a Windows x64 function.
Change an owned fixed-argument function into a variadic one. Read the compiler output, but use va_arg rather than attempting a manual register save area.

Review register widths and RIP-relative addressing, then continue to compiler output and object files. These cross-links describe the same x86-64 code at the instruction, ABI, and linker layers.