1.. SPDX-License-Identifier: GPL-2.0 2 3====================== 4The x86 kvm shadow mmu 5====================== 6 7The mmu (in arch/x86/kvm, files mmu.[ch] and paging_tmpl.h) is responsible 8for presenting a standard x86 mmu to the guest, while translating guest 9physical addresses to host physical addresses. 10 11The mmu code attempts to satisfy the following requirements: 12 13- correctness: 14 the guest should not be able to determine that it is running 15 on an emulated mmu except for timing (we attempt to comply 16 with the specification, not emulate the characteristics of 17 a particular implementation such as tlb size) 18- security: 19 the guest must not be able to touch host memory not assigned 20 to it 21- performance: 22 minimize the performance penalty imposed by the mmu 23- scaling: 24 need to scale to large memory and large vcpu guests 25- hardware: 26 support the full range of x86 virtualization hardware 27- integration: 28 Linux memory management code must be in control of guest memory 29 so that swapping, page migration, page merging, transparent 30 hugepages, and similar features work without change 31- dirty tracking: 32 report writes to guest memory to enable live migration 33 and framebuffer-based displays 34- footprint: 35 keep the amount of pinned kernel memory low (most memory 36 should be shrinkable) 37- reliability: 38 avoid multipage or GFP_ATOMIC allocations 39 40Acronyms 41======== 42 43==== ==================================================================== 44pfn host page frame number 45hpa host physical address 46hva host virtual address 47gfn guest frame number 48gpa guest physical address 49gva guest virtual address 50ngpa nested guest physical address 51ngva nested guest virtual address 52pte page table entry (used also to refer generically to paging structure 53 entries) 54gpte guest pte (referring to gfns) 55spte shadow pte (referring to pfns) 56tdp two dimensional paging (vendor neutral term for NPT and EPT) 57==== ==================================================================== 58 59Virtual and real hardware supported 60=================================== 61 62The mmu supports first-generation mmu hardware, which allows an atomic switch 63of the current paging mode and cr3 during guest entry, as well as 64two-dimensional paging (AMD's NPT and Intel's EPT). The emulated hardware 65it exposes is the traditional 2/3/4 level x86 mmu, with support for global 66pages, pae, pse, pse36, cr0.wp, and 1GB pages. Emulated hardware also 67able to expose NPT capable hardware on NPT capable hosts. 68 69Translation 70=========== 71 72The primary job of the mmu is to program the processor's mmu to translate 73addresses for the guest. Different translations are required at different 74times: 75 76- when guest paging is disabled, we translate guest physical addresses to 77 host physical addresses (gpa->hpa) 78- when guest paging is enabled, we translate guest virtual addresses, to 79 guest physical addresses, to host physical addresses (gva->gpa->hpa) 80- when the guest launches a guest of its own, we translate nested guest 81 virtual addresses, to nested guest physical addresses, to guest physical 82 addresses, to host physical addresses (ngva->ngpa->gpa->hpa) 83 84The primary challenge is to encode between 1 and 3 translations into hardware 85that support only 1 (traditional) and 2 (tdp) translations. When the 86number of required translations matches the hardware, the mmu operates in 87direct mode; otherwise it operates in shadow mode (see below). 88 89Memory 90====== 91 92Guest memory (gpa) is part of the user address space of the process that is 93using kvm. Userspace defines the translation between guest addresses and user 94addresses (gpa->hva); note that two gpas may alias to the same hva, but not 95vice versa. 96 97These hvas may be backed using any method available to the host: anonymous 98memory, file backed memory, and device memory. Memory might be paged by the 99host at any time. 100 101Events 102====== 103 104The mmu is driven by events, some from the guest, some from the host. 105 106Guest generated events: 107 108- writes to control registers (especially cr3) 109- invlpg/invlpga instruction execution 110- access to missing or protected translations 111 112Host generated events: 113 114- changes in the gpa->hpa translation (either through gpa->hva changes or 115 through hva->hpa changes) 116- memory pressure (the shrinker) 117 118Shadow pages 119============ 120 121The principal data structure is the shadow page, 'struct kvm_mmu_page'. A 122shadow page contains 512 sptes, which can be either leaf or nonleaf sptes. A 123shadow page may contain a mix of leaf and nonleaf sptes. 124 125A nonleaf spte allows the hardware mmu to reach the leaf pages and 126is not related to a translation directly. It points to other shadow pages. 127 128A leaf spte corresponds to either one or two translations encoded into 129one paging structure entry. These are always the lowest level of the 130translation stack, with optional higher level translations left to NPT/EPT. 131Leaf ptes point at guest pages. 132 133The following table shows translations encoded by leaf ptes, with higher-level 134translations in parentheses: 135 136 Non-nested guests:: 137 138 nonpaging: gpa->hpa 139 paging: gva->gpa->hpa 140 paging, tdp: (gva->)gpa->hpa 141 142 Nested guests:: 143 144 non-tdp: ngva->gpa->hpa (*) 145 tdp: (ngva->)ngpa->gpa->hpa 146 147 (*) the guest hypervisor will encode the ngva->gpa translation into its page 148 tables if npt is not present 149 150Shadow pages contain the following information: 151 role.level: 152 The level in the shadow paging hierarchy that this shadow page belongs to. 153 1=4k sptes, 2=2M sptes, 3=1G sptes, etc. 154 role.direct: 155 If set, leaf sptes reachable from this page are for a linear range. 156 Examples include real mode translation, large guest pages backed by small 157 host pages, and gpa->hpa translations when NPT or EPT is active. 158 The linear range starts at (gfn << PAGE_SHIFT) and its size is determined 159 by role.level (2MB for first level, 1GB for second level, 0.5TB for third 160 level, 256TB for fourth level) 161 If clear, this page corresponds to a guest page table denoted by the gfn 162 field. 163 role.quadrant: 164 When role.gpte_is_8_bytes=0, the guest uses 32-bit gptes while the host uses 64-bit 165 sptes. That means a guest page table contains more ptes than the host, 166 so multiple shadow pages are needed to shadow one guest page. 167 For first-level shadow pages, role.quadrant can be 0 or 1 and denotes the 168 first or second 512-gpte block in the guest page table. For second-level 169 page tables, each 32-bit gpte is converted to two 64-bit sptes 170 (since each first-level guest page is shadowed by two first-level 171 shadow pages) so role.quadrant takes values in the range 0..3. Each 172 quadrant maps 1GB virtual address space. 173 role.access: 174 Inherited guest access permissions from the parent ptes in the form uwx. 175 Note execute permission is positive, not negative. 176 role.invalid: 177 The page is invalid and should not be used. It is a root page that is 178 currently pinned (by a cpu hardware register pointing to it); once it is 179 unpinned it will be destroyed. 180 role.gpte_is_8_bytes: 181 Reflects the size of the guest PTE for which the page is valid, i.e. '1' 182 if 64-bit gptes are in use, '0' if 32-bit gptes are in use. 183 role.nxe: 184 Contains the value of efer.nxe for which the page is valid. 185 role.cr0_wp: 186 Contains the value of cr0.wp for which the page is valid. 187 role.smep_andnot_wp: 188 Contains the value of cr4.smep && !cr0.wp for which the page is valid 189 (pages for which this is true are different from other pages; see the 190 treatment of cr0.wp=0 below). 191 role.smap_andnot_wp: 192 Contains the value of cr4.smap && !cr0.wp for which the page is valid 193 (pages for which this is true are different from other pages; see the 194 treatment of cr0.wp=0 below). 195 role.ept_sp: 196 This is a virtual flag to denote a shadowed nested EPT page. ept_sp 197 is true if "cr0_wp && smap_andnot_wp", an otherwise invalid combination. 198 role.smm: 199 Is 1 if the page is valid in system management mode. This field 200 determines which of the kvm_memslots array was used to build this 201 shadow page; it is also used to go back from a struct kvm_mmu_page 202 to a memslot, through the kvm_memslots_for_spte_role macro and 203 __gfn_to_memslot. 204 role.ad_disabled: 205 Is 1 if the MMU instance cannot use A/D bits. EPT did not have A/D 206 bits before Haswell; shadow EPT page tables also cannot use A/D bits 207 if the L1 hypervisor does not enable them. 208 gfn: 209 Either the guest page table containing the translations shadowed by this 210 page, or the base page frame for linear translations. See role.direct. 211 spt: 212 A pageful of 64-bit sptes containing the translations for this page. 213 Accessed by both kvm and hardware. 214 The page pointed to by spt will have its page->private pointing back 215 at the shadow page structure. 216 sptes in spt point either at guest pages, or at lower-level shadow pages. 217 Specifically, if sp1 and sp2 are shadow pages, then sp1->spt[n] may point 218 at __pa(sp2->spt). sp2 will point back at sp1 through parent_pte. 219 The spt array forms a DAG structure with the shadow page as a node, and 220 guest pages as leaves. 221 gfns: 222 An array of 512 guest frame numbers, one for each present pte. Used to 223 perform a reverse map from a pte to a gfn. When role.direct is set, any 224 element of this array can be calculated from the gfn field when used, in 225 this case, the array of gfns is not allocated. See role.direct and gfn. 226 root_count: 227 A counter keeping track of how many hardware registers (guest cr3 or 228 pdptrs) are now pointing at the page. While this counter is nonzero, the 229 page cannot be destroyed. See role.invalid. 230 parent_ptes: 231 The reverse mapping for the pte/ptes pointing at this page's spt. If 232 parent_ptes bit 0 is zero, only one spte points at this page and 233 parent_ptes points at this single spte, otherwise, there exists multiple 234 sptes pointing at this page and (parent_ptes & ~0x1) points at a data 235 structure with a list of parent sptes. 236 unsync: 237 If true, then the translations in this page may not match the guest's 238 translation. This is equivalent to the state of the tlb when a pte is 239 changed but before the tlb entry is flushed. Accordingly, unsync ptes 240 are synchronized when the guest executes invlpg or flushes its tlb by 241 other means. Valid for leaf pages. 242 unsync_children: 243 How many sptes in the page point at pages that are unsync (or have 244 unsynchronized children). 245 unsync_child_bitmap: 246 A bitmap indicating which sptes in spt point (directly or indirectly) at 247 pages that may be unsynchronized. Used to quickly locate all unsychronized 248 pages reachable from a given page. 249 clear_spte_count: 250 Only present on 32-bit hosts, where a 64-bit spte cannot be written 251 atomically. The reader uses this while running out of the MMU lock 252 to detect in-progress updates and retry them until the writer has 253 finished the write. 254 write_flooding_count: 255 A guest may write to a page table many times, causing a lot of 256 emulations if the page needs to be write-protected (see "Synchronized 257 and unsynchronized pages" below). Leaf pages can be unsynchronized 258 so that they do not trigger frequent emulation, but this is not 259 possible for non-leafs. This field counts the number of emulations 260 since the last time the page table was actually used; if emulation 261 is triggered too frequently on this page, KVM will unmap the page 262 to avoid emulation in the future. 263 264Reverse map 265=========== 266 267The mmu maintains a reverse mapping whereby all ptes mapping a page can be 268reached given its gfn. This is used, for example, when swapping out a page. 269 270Synchronized and unsynchronized pages 271===================================== 272 273The guest uses two events to synchronize its tlb and page tables: tlb flushes 274and page invalidations (invlpg). 275 276A tlb flush means that we need to synchronize all sptes reachable from the 277guest's cr3. This is expensive, so we keep all guest page tables write 278protected, and synchronize sptes to gptes when a gpte is written. 279 280A special case is when a guest page table is reachable from the current 281guest cr3. In this case, the guest is obliged to issue an invlpg instruction 282before using the translation. We take advantage of that by removing write 283protection from the guest page, and allowing the guest to modify it freely. 284We synchronize modified gptes when the guest invokes invlpg. This reduces 285the amount of emulation we have to do when the guest modifies multiple gptes, 286or when the a guest page is no longer used as a page table and is used for 287random guest data. 288 289As a side effect we have to resynchronize all reachable unsynchronized shadow 290pages on a tlb flush. 291 292 293Reaction to events 294================== 295 296- guest page fault (or npt page fault, or ept violation) 297 298This is the most complicated event. The cause of a page fault can be: 299 300 - a true guest fault (the guest translation won't allow the access) (*) 301 - access to a missing translation 302 - access to a protected translation 303 - when logging dirty pages, memory is write protected 304 - synchronized shadow pages are write protected (*) 305 - access to untranslatable memory (mmio) 306 307 (*) not applicable in direct mode 308 309Handling a page fault is performed as follows: 310 311 - if the RSV bit of the error code is set, the page fault is caused by guest 312 accessing MMIO and cached MMIO information is available. 313 314 - walk shadow page table 315 - check for valid generation number in the spte (see "Fast invalidation of 316 MMIO sptes" below) 317 - cache the information to vcpu->arch.mmio_gva, vcpu->arch.mmio_access and 318 vcpu->arch.mmio_gfn, and call the emulator 319 320 - If both P bit and R/W bit of error code are set, this could possibly 321 be handled as a "fast page fault" (fixed without taking the MMU lock). See 322 the description in Documentation/virt/kvm/locking.rst. 323 324 - if needed, walk the guest page tables to determine the guest translation 325 (gva->gpa or ngpa->gpa) 326 327 - if permissions are insufficient, reflect the fault back to the guest 328 329 - determine the host page 330 331 - if this is an mmio request, there is no host page; cache the info to 332 vcpu->arch.mmio_gva, vcpu->arch.mmio_access and vcpu->arch.mmio_gfn 333 334 - walk the shadow page table to find the spte for the translation, 335 instantiating missing intermediate page tables as necessary 336 337 - If this is an mmio request, cache the mmio info to the spte and set some 338 reserved bit on the spte (see callers of kvm_mmu_set_mmio_spte_mask) 339 340 - try to unsynchronize the page 341 342 - if successful, we can let the guest continue and modify the gpte 343 344 - emulate the instruction 345 346 - if failed, unshadow the page and let the guest continue 347 348 - update any translations that were modified by the instruction 349 350invlpg handling: 351 352 - walk the shadow page hierarchy and drop affected translations 353 - try to reinstantiate the indicated translation in the hope that the 354 guest will use it in the near future 355 356Guest control register updates: 357 358- mov to cr3 359 360 - look up new shadow roots 361 - synchronize newly reachable shadow pages 362 363- mov to cr0/cr4/efer 364 365 - set up mmu context for new paging mode 366 - look up new shadow roots 367 - synchronize newly reachable shadow pages 368 369Host translation updates: 370 371 - mmu notifier called with updated hva 372 - look up affected sptes through reverse map 373 - drop (or update) translations 374 375Emulating cr0.wp 376================ 377 378If tdp is not enabled, the host must keep cr0.wp=1 so page write protection 379works for the guest kernel, not guest guest userspace. When the guest 380cr0.wp=1, this does not present a problem. However when the guest cr0.wp=0, 381we cannot map the permissions for gpte.u=1, gpte.w=0 to any spte (the 382semantics require allowing any guest kernel access plus user read access). 383 384We handle this by mapping the permissions to two possible sptes, depending 385on fault type: 386 387- kernel write fault: spte.u=0, spte.w=1 (allows full kernel access, 388 disallows user access) 389- read fault: spte.u=1, spte.w=0 (allows full read access, disallows kernel 390 write access) 391 392(user write faults generate a #PF) 393 394In the first case there are two additional complications: 395 396- if CR4.SMEP is enabled: since we've turned the page into a kernel page, 397 the kernel may now execute it. We handle this by also setting spte.nx. 398 If we get a user fetch or read fault, we'll change spte.u=1 and 399 spte.nx=gpte.nx back. For this to work, KVM forces EFER.NX to 1 when 400 shadow paging is in use. 401- if CR4.SMAP is disabled: since the page has been changed to a kernel 402 page, it can not be reused when CR4.SMAP is enabled. We set 403 CR4.SMAP && !CR0.WP into shadow page's role to avoid this case. Note, 404 here we do not care the case that CR4.SMAP is enabled since KVM will 405 directly inject #PF to guest due to failed permission check. 406 407To prevent an spte that was converted into a kernel page with cr0.wp=0 408from being written by the kernel after cr0.wp has changed to 1, we make 409the value of cr0.wp part of the page role. This means that an spte created 410with one value of cr0.wp cannot be used when cr0.wp has a different value - 411it will simply be missed by the shadow page lookup code. A similar issue 412exists when an spte created with cr0.wp=0 and cr4.smep=0 is used after 413changing cr4.smep to 1. To avoid this, the value of !cr0.wp && cr4.smep 414is also made a part of the page role. 415 416Large pages 417=========== 418 419The mmu supports all combinations of large and small guest and host pages. 420Supported page sizes include 4k, 2M, 4M, and 1G. 4M pages are treated as 421two separate 2M pages, on both guest and host, since the mmu always uses PAE 422paging. 423 424To instantiate a large spte, four constraints must be satisfied: 425 426- the spte must point to a large host page 427- the guest pte must be a large pte of at least equivalent size (if tdp is 428 enabled, there is no guest pte and this condition is satisfied) 429- if the spte will be writeable, the large page frame may not overlap any 430 write-protected pages 431- the guest page must be wholly contained by a single memory slot 432 433To check the last two conditions, the mmu maintains a ->disallow_lpage set of 434arrays for each memory slot and large page size. Every write protected page 435causes its disallow_lpage to be incremented, thus preventing instantiation of 436a large spte. The frames at the end of an unaligned memory slot have 437artificially inflated ->disallow_lpages so they can never be instantiated. 438 439Fast invalidation of MMIO sptes 440=============================== 441 442As mentioned in "Reaction to events" above, kvm will cache MMIO 443information in leaf sptes. When a new memslot is added or an existing 444memslot is changed, this information may become stale and needs to be 445invalidated. This also needs to hold the MMU lock while walking all 446shadow pages, and is made more scalable with a similar technique. 447 448MMIO sptes have a few spare bits, which are used to store a 449generation number. The global generation number is stored in 450kvm_memslots(kvm)->generation, and increased whenever guest memory info 451changes. 452 453When KVM finds an MMIO spte, it checks the generation number of the spte. 454If the generation number of the spte does not equal the global generation 455number, it will ignore the cached MMIO information and handle the page 456fault through the slow path. 457 458Since only 18 bits are used to store generation-number on mmio spte, all 459pages are zapped when there is an overflow. 460 461Unfortunately, a single memory access might access kvm_memslots(kvm) multiple 462times, the last one happening when the generation number is retrieved and 463stored into the MMIO spte. Thus, the MMIO spte might be created based on 464out-of-date information, but with an up-to-date generation number. 465 466To avoid this, the generation number is incremented again after synchronize_srcu 467returns; thus, bit 63 of kvm_memslots(kvm)->generation set to 1 only during a 468memslot update, while some SRCU readers might be using the old copy. We do not 469want to use an MMIO sptes created with an odd generation number, and we can do 470this without losing a bit in the MMIO spte. The "update in-progress" bit of the 471generation is not stored in MMIO spte, and is so is implicitly zero when the 472generation is extracted out of the spte. If KVM is unlucky and creates an MMIO 473spte while an update is in-progress, the next access to the spte will always be 474a cache miss. For example, a subsequent access during the update window will 475miss due to the in-progress flag diverging, while an access after the update 476window closes will have a higher generation number (as compared to the spte). 477 478 479Further reading 480=============== 481 482- NPT presentation from KVM Forum 2008 483 https://www.linux-kvm.org/images/c/c8/KvmForum2008%24kdf2008_21.pdf 484