Lukas' Notes

Calling Convention (x86-64)

2 min read

computer-architecture

Definition

Calling Convention (x86-64)

The System V AMD64 calling convention fixes how a function call passes arguments and return values across the regfile and the stack on 64-bit Linux (and other System V systems). The first six integer/pointer arguments are passed in registers RDI, RSI, RDX, RCX, R8, R9; any further arguments are pushed on the stack with the first remaining argument at the lowest address; the return value is placed in RAX. The call instruction pushes the address of the next instruction onto the stack before jumping, and ret pops it back into RIP.

Argument Passing

Where each argument goes

  • in registers — up to six integer/pointer arguments use, in order, RDI, RSI, RDX, RCX, R8, R9;
  • on the stack — any argument beyond the sixth is pushed before the call, first remaining argument at the lowest address (so arguments flow top-to-bottom in address order);
  • return value — in RAX (a second return value can use RDX).

Saved State and Transfer of Control

What the caller and callee must preserve

  • The call <address> instruction pushes the return address (the address of the instruction after call) onto the stack, then jumps to the function.
  • The callee is then responsible for setting up its stack frame (prologue) and for cleaning it up on return (epilogue), restoring RBP/RSP and any caller-saved registers it clobbered.
  • Inside the callee, RBP references arguments and local variables; the return value is staged in RAX before the epilogue’s ret pops the saved return address and transfers control back to the caller.

Example

func(10) in register and stack progression

The slide walks the regfile/stack across one call. Before the call the first argument is staged in RDI; the call then pushes the return address onto the stack.

  • stage the argument 
    mov  rdi, 10
call func     ; push return address, then jmp func
  • register + stack before the call
    • RDI = 10, RAX / RSI / RDX = whatever the caller had
    • stack pointer at the caller’s frame top
  • register + stack after call enters func
    • RSP now points at the freshly pushed <return address> (the next instruction after call)
    • RDI still holds 10 — the callee’s prologue (push rbp; mov rbp, rsp; sub rsp, N) then sets up its own frame below the return address, as in Stack Frame.

Backlinks