How a Hypervisor Runs a VM: VM Exits

The idea: the guest runs on the real CPU until it must not

A virtual machine does not interpret the guest's code. With hardware virtualization (Intel VT-x, AMD-V) the guest's instructions run directly on the physical CPU, at full speed. The CPU only stops the guest when it does something the hypervisor must decide: that stop is a VM exit. The hypervisor handles it and enters the guest again. On Linux the hypervisor is split in two: KVM, a kernel module, and QEMU, a normal user process that emulates the devices.

The page shows one virtual CPU (vCPU) running a short guest program. Each line is a real x86-64 instruction with its real length, and each exit has its real exit reason number and KVM handler name.

VMX root and non-root mode

VT-x adds two modes on top of the usual rings. The hypervisor runs in VMX root mode; the guest runs in VMX non-root mode. The guest kernel still runs in ring 0 and the guest's programs in ring 3, so the guest OS works unchanged, but in non-root mode some instructions and events cause an exit instead of doing their normal work.

• VMLAUNCH enters a guest the first time; VMRESUME enters it again after each exit.
• A VM exit saves the guest's registers, loads the host's and jumps to the host RIP that KVM set up (vmx_vmexit).
• Ordinary instructions such as add or a load from a mapped page never exit (Demo 1: add runs natively, cpuid exits to KVM).

QEMU starts it all: it opens /dev/kvm, creates the VM and a vCPU with KVM_CREATE_VM and KVM_CREATE_VCPU, gives KVM the guest RAM as memory slots, and then each vCPU thread loops on ioctl(vcpu_fd, KVM_RUN).

The VMCS: what exits, and why it exited

Each vCPU has a VMCS (virtual-machine control structure), a block of memory the CPU reads at entry and writes at exit. It holds the guest state, the host state, the VM-execution controls (which events exit) and, after an exit, the exit reason, the exit qualification (details such as the port number), the instruction length and the guest-physical address.

exit reason	caused by	KVM handler	what KVM does
1 EXTERNAL_INTERRUPT	a host device or timer interrupt	handle_external_interrupt	runs the host's interrupt handler, then resumes
10 CPUID	cpuid (always exits)	handle_cpuid	answers from the CPUID table QEMU set, skips 2 bytes
12 HLT	hlt with HLT exiting	handle_halt	puts the vCPU thread to sleep until an interrupt is due
30 IO_INSTRUCTION	in/out to a port in the I/O bitmap	handle_io	an in-kernel device, or KVM_EXIT_IO to QEMU
48 EPT_VIOLATION	a guest-physical address with no EPT entry	handle_ept_violation	maps the host page into the EPT, re-runs the instruction

With cpuid leaf 0x40000000 the guest gets "KVMKVMKVM": this is how a Linux guest finds out it runs on KVM. A host timer interrupt exits even though the guest did nothing (Demo 4: a host interrupt, then hlt); after hlt, KVM later injects the guest's own timer interrupt through the VMCS at the next entry.

Handled in the kernel, or sent to QEMU

KVM handles what it can without leaving the kernel. A port that no in-kernel device claims, such as the 16550 serial port at 0x3F8 here, needs QEMU (Demo 2: out to the serial port goes up to QEMU):

1. The out exits with reason 30; the qualification says port 0x3F8, 1 byte, OUT.
2. handle_io() finds no device on its I/O bus, fills the shared struct kvm_run (exit_reason = KVM_EXIT_IO, port, direction, size, data) and returns from the ioctl.
3. QEMU's vCPU thread runs its serial device model, which prints the character.
4. QEMU calls ioctl(KVM_RUN) again; KVM finishes the instruction and runs VMRESUME.

An exit handled in KVM costs roughly a thousand cycles or more for the round trip. A trip to QEMU adds two switches between kernel and user space and the device model, several times more.

Two-level address translation: the EPT

The guest has its own page tables and its own CR3, and it thinks they map to physical memory. They map a guest-virtual address (GVA) to a guest-physical address (GPA). The EPT (extended page tables), which KVM owns and the VMCS points to (EPTP), maps each GPA to a host-physical address (HPA). The CPU does both walks in hardware.

KVM fills the EPT lazily. The first touch of GPA 0x3000 is an EPT violation: KVM finds the page in the memory slot QEMU registered (guest RAM is ordinary memory of the QEMU process), writes the EPT entry and resumes at the same RIP, so the instruction runs again and succeeds. The next access to the page does not exit (Demo 3: first write to a page: EPT violation). The guest changes its own page tables without any exit at all.

The price is the TLB miss: every step of the guest's 4-level walk reads a guest page-table page by its GPA, which needs its own EPT walk. A full miss can take up to 24 memory reads instead of 4. The TLB caches the final GVA → HPA, so most accesses pay nothing.

Why exits matter

Virtualization overhead is mostly the number of exits times their cost. That is why real setups avoid them:

• virtio devices batch many requests per notification, and vhost moves the device into the host kernel.
• Posted interrupts and APICv deliver interrupts to a running guest without an exit.
• Halt polling spins briefly after hlt before sleeping, because an interrupt often comes soon.
• Device passthrough (VFIO) gives the guest a real device, with an IOMMU to keep its DMA inside the guest's memory.

See also containers vs virtual machines, system calls (the same kind of controlled switch, one level down), the TLB and the page walk, virtual memory and Linux interrupts.

What the page leaves out

Several vCPUs and the IPIs between them; the in-kernel interrupt controller and timer devices; MMIO exits (an EPT misconfiguration that KVM decodes or sends to QEMU as KVM_EXIT_MMIO); the real 4-level guest and EPT tables, huge pages and the TLB tags (VPID); control-register and MSR exits; nested virtualization; AMD SVM, which has the same ideas under other names (VMCB, VMRUN, #VMEXIT); and exact cycle counts, which depend on the CPU.

References

Intel 64 and IA-32 Architectures Software Developer's Manual, Vol. 3C: VMX (Appendix C: exit reasons)

Linux kernel documentation: the KVM API (KVM_RUN, struct kvm_run, KVM_EXIT_IO)

Linux source: arch/x86/kvm/vmx/vmx.c (kvm_vmx_exit_handlers)

LWN: Using the KVM API