x86 Assembly and Calling Conventions

Assembly is a human notation for encoded instructions, not one universal language. A CPU executes bytes; an assembler accepts a syntax for a selected processor mode, object format, and ABI. Intel syntax writes mov rax, rbx; AT&T syntax commonly writes movq %rbx, %rax. The notation differs in operand order, register prefix, immediate prefix, and size suffix. Neither notation changes the instruction’s underlying encoding.

State and decoding

x86-64 general-purpose registers include RAX, RBX, RCX, RDX, RSI, RDI, RBP, RSP, and R8–R15, plus RIP and RFLAGS. EAX selects the low 32 bits of RAX; a 32-bit write zero-extends into RAX in 64-bit mode. AX, AH, and AL are narrower historical portions. Vector, control, debug, segment, and floating-point state have separate purposes. Instructions cover moves, arithmetic, comparisons, branches, calls, returns, loads/stores through effective addresses, and SIMD operations.

x86 instructions are variable length. A disassembler needs the right mode and entry address: bytes decoded from the wrong boundary can look like valid but unrelated instructions. Treat output as evidence to correlate with symbols, relocations, source, and compiler flags, not as an unquestionable explanation of intent.

Calls and the stack

call normally records a return address on the stack and transfers control; ret resumes through that address. Functions use the stack for saved registers, local storage, spill locations, and arguments not placed in registers. It conventionally grows downward, but optimized code may omit a frame pointer, inline a function, or use a permitted red zone. A drawn stack frame is a useful model, not a promise about every compiler build.

Rule	System V AMD64	Windows x64
First integer/pointer arguments	RDI, RSI, RDX, RCX, R8, R9	RCX, RDX, R8, R9
Integer return	RAX	RAX
Caller stack area	Eligible leaf code may use a 128-byte red zone	Caller reserves 32-byte shadow space
Unwinding	Usually DWARF call-frame information	PE/COFF unwind metadata

Floating-point/vector arguments, aggregates, varargs, alignment, and preserved registers add important detail. Consult the actual ABI before writing an interface. Do not use the SysV red zone in Windows code or omit Windows shadow space when following its convention.

Calling C and reading output safely

A C declaration is not enough to guess assembly. Target ABI determines locations; type determines extension and register class; optimization changes layout; linking may add thunks. A safe experiment is a pure function: long add_bias(long x) { return x + 7; }. A SysV AMD64 build commonly receives x in RDI; Windows x64 commonly uses RCX; the result is in RAX. The compiler may use lea, add, or inline it, so generated instructions are evidence rather than the language rule.

Build code you own with debug information, then inspect a copy: objdump -d -Mintel file.o selects Intel-style output on GNU binutils; -S asks a compiler for assembly before assembly and linking. Linked executables can add startup code, relocation stubs, and dynamic-linking machinery. The related command-line disassembly article shows a basic workflow. Do not execute unknown bytes, disable safeguards, or treat a disassembler as a safe sandbox.

Instructions, syntax, and object files

Machine-code encodings use prefixes, opcodes, ModR/M and sometimes SIB bytes, displacement, and immediate fields. The assembler selects an encoding after resolving labels and addressing choices; the linker later resolves references between object files and shared libraries. This is why a call shown in an object file can have an unresolved relocation rather than its final target. Position-independent code commonly uses RIP-relative addressing on x86-64 and may call a procedure-linkage stub for an external symbol. These mechanisms are normal linking machinery, not necessarily indirection added by a compiler “inefficiency.”

Condition codes deserve care. cmp performs a subtraction for flags without retaining its numeric result; conditional jumps choose signed or unsigned interpretations through different flag combinations. ja and jg do not mean the same comparison. Likewise, an address calculation with lea does not dereference memory, despite its name. Read operand width and signedness from surrounding code and types before translating a few instructions into a high-level claim.

Calling conventions protect separately compiled code. The caller and callee agree on which registers are volatile, which are preserved, where arguments live, how the stack is aligned, and how exceptions can unwind. Violating an ABI can seem to work in a tiny test and fail after a library call, optimization change, or exception. Inline assembly needs declared inputs, outputs, clobbers, and memory effects so the compiler can maintain that agreement; it is not a safe shortcut around understanding it.

For learning, compare an unoptimized build with an optimized one, then inspect source interleaving and symbols. Avoid treating compiler-generated stack canaries, control-flow protection, or bounds checks as noise: they are observable security and ABI-relevant design choices. Only debug executables and inputs you are authorized to run.

Three related, different x86 contexts

The word “x86” spans several execution environments. Keeping them separate prevents many plausible but incorrect explanations. The original 8086-era 16-bit model used AX, BX, CX, DX, IP, FLAGS, and segment:offset addressing. Real mode’s familiar address calculation and 64 KiB segment windows are historical context, not the normal address model for a protected-mode IA-32 application. Its conventions, object formats, and even available instructions reflect a different era.

IA-32 protected-mode practice is the focus of this category. General-purpose registers acquire their 32-bit E names—EAX through EDI—while EIP names the instruction pointer and EFLAGS holds conditions and control bits. Protected-mode segment registers are selectors interpreted through descriptors; ordinary 32-bit user processes commonly use a flat arrangement where code and data descriptor bases are zero. Paging and privilege remain operating-system responsibilities. FS or GS may identify thread- or platform-defined data, so a segment prefix is meaningful even in code described as flat.

x86-64 is a later extension, not merely “IA-32 with bigger registers.” It adds R8–R15, changes instruction-pointer-relative addressing, and has its own System V AMD64 and Windows x64 ABIs. A 32-bit write has a special zero-extending consequence in long mode; the stack argument registers, red zone, Windows shadow space, and 64-bit calling rules in the opening table apply only to the named x86-64 ABI. For those details, use the dedicated x86-64 track rather than copying them into an i386 function.

Question	16-bit historical context	IA-32 protected mode	x86-64 reference
Instruction pointer	IP with CS	EIP with CS selector	RIP
Typical pointer width	16-bit offsets, model-dependent	32 bits	64 bits
Ordinary user addressing	segment:offset model	usually flat descriptors plus paging	long-mode rules, often RIP-relative code
Do ABI rules transfer?	No: historically varied	Use i386 target ABI	No: use AMD64 or Windows x64 ABI

IA-32 calls in practical terms

In a common 32-bit cdecl interface, arguments are placed on the stack and the caller removes them after the call; EAX commonly returns an integer result. EAX, ECX, and EDX are typically volatile, while EBX, ESI, EDI, and EBP are typically preserved. These are common i386 ABI patterns, not an architectural property of call. stdcall usually makes the callee remove fixed stack arguments, and fastcall is compiler-specific: a document must identify the compiler’s version before saying which arguments use ECX or EDX. Types, aggregate returns, variadic arguments, alignment, x87/SSE choices, and unwinding impose additional rules.

A conventional frame begins push ebp; mov ebp, esp, putting a saved EBP at [ebp], return address at [ebp+4], and first stack argument at [ebp+8]. It is educational but optional: optimized code can omit EBP as a frame pointer. Compilers can produce debugging/unwind metadata instead. Do not use a pictured frame chain as a substitute for symbols, ABI documentation, or authorized debugging.

Encoding and tool boundaries

Whether source says mov eax, [ebp-4] or movl -4(%ebp), %eax, the resulting IA-32 instruction is selected from variable-length fields: opcode, potentially ModR/M and SIB addressing fields, a displacement, and possibly an immediate. Little-endian order stores the low byte of a multi-byte value first. NASM normally presents Intel syntax; GNU as normally presents AT&T syntax. Syntax is not an ABI and does not select 16-, 32-, or 64-bit mode on its own.

A safe starting point is an ELF32 relocatable object. nasm -f elf32 example.asm -o example.o or as --32 example.s -o example.o, followed by readelf -h -s -r example.o and objdump -dr -Mintel example.o, requires assemblers and binutils but no multilib runtime. Relocations in the object indicate addresses the linker will later supply. Do not execute a file just to inspect it, and do not interpret bytes from an unknown source as a lesson.

For a guided IA-32 foundation, start with registers, modes, and instruction encoding, then continue to stack frames and calling conventions and NASM and GNU as. Linux int 0x80, often seen in old 32-bit Linux material, is a historical, Linux-specific kernel entry convention—not a portable x86 calling convention—and is unnecessary for these object-only exercises.

The durable lesson is to name the mode, ABI, and toolchain before generalizing from a listing.