1# Queue Present Wait Semaphore Management 2 3The following shorthand notations are used throughout this document: 4 5- PE: Presentation Engine 6- ANI: vkAcquireNextImageKHR 7- QS: vkQueueSubmit 8- QP: vkQueuePresentKHR 9- W: Wait 10- S: Signal 11- R: Render 12- P: Present 13- SN: Semaphore N 14- IN: Swapchain image N 15- FN: Fence N 16 17--- 18 19## Introduction 20 21Vulkan requires the application (ANGLE in this case) to acquire swapchain images and queue them for 22presentation, synchronizing GPU submissions with semaphores. A single frame looks like the 23following: 24 25 CPU: ANI ... QS ... QP 26 S:S1 W:S1 W:S2 27 S:S2 28 GPU: <------------ R -----------> 29 PE: <-------- P ------> 30 31That is, the GPU starts rendering after submission, and the presentation is started when rendering is 32finished. Note that Vulkan tries to abstract a large variety of PE architectures, some of which do 33not behave in a straight-forward manner. As such, ANGLE cannot know what the PE is exactly doing 34with the images or when the images are visible on the screen. The only signal out of the PE is 35received through the semaphore that's used in ANI. 36 37With multiple frames, the pipeline looks different based on present mode. Let's focus on 38FIFO (the arguments in this document translate to all modes) with 3 images: 39 40 CPU: QS QP QS QP QS QP QS QP 41 I1 I1 I2 I2 I3 I3 I1 I1 42 GPU: <---- R I1 ----><---- R I2 ----><---- R I3 ----><---- R I1 ----> 43 PE: <----- P I1 -----><----- P I2 -----><----- P I3 -----><----- P I1 -----> 44 45First, an issue is evident here. The CPU is submitting jobs and queuing images for presentation 46faster than the GPU can render them or the PE can view them. This can cause the length of the 47submit queue to grow indefinitely, resulting in larger and larger input lag. In FIFO mode, the PE 48present queue also grows indefinitely. 49 50To address this issue, ANGLE paces the CPU such that the length of the submit queue is kept at a 51maximum of 1 image (i.e. submission with one image is being processed, and another one is in queue): 52 53 CPU: QS QS W:F1 QS W:F2 QS 54 I1 I2 I3 I1 55 S:F1 S:F2 S:F3 S:F4 56 GPU: <---- R I1 ----><---- R I2 ----><---- R I3 ----><---- R I1 ----> 57 58> Note: Ideally, the length of the PE present queue should also be kept at a maximum of 1 (i.e. one 59> image being presented, and another in queue). However, the Vulkan WSI extension doesn't provide 60> enough control to achieve this. In heavy application, the length of the PE present queue is 61> probably 1 anyway (as the rendering time is almost as long as the frame (i.e. present time), in 62> which case pacing the submissions similarly paces the presentation). In theory, in FIFO mode, the 63> length of the PE present queue is below n+2 where n is the number of swapchain images. 64> 65> To understand why, imagine a FIFO swapchain with 1000 images and submissions that are 66> infinitesimally short. In this case, the CPU pacing is effectively a no-op (as the GPU instantly 67> finishes jobs) for the first 1002 submissions. The 1003rd submission waits for F1001 (which uses 68> I1). However, the 1001st submission will not start until the PE switches to presenting I2 (at the 69> next V-Sync). The CPU then waits for V-Sync before the 1003rd submission. The CPU waits for one 70> V-Sync for every subsequent submission, keeping the length of the queue 1002. 71> [`VK_GOOGLE_display_timing`][DisplayTimingGOOGLE] is likely a solution to this problem. 72 73Associated with each QP operation is a semaphore signaled by the preceding QS and waited on by the 74PE before the image can be presented. Currently, there's no feedback from Vulkan (See [internal 75Khronos issue][VulkanIssue1060]) regarding _when_ the PE has actually finished waiting on the 76semaphore! This means that the application cannot generally know when to destroy the corresponding 77semaphore. However, taking ANGLE's CPU pacing into account, we are able to destroy (or rather 78reuse) semaphores when they are provably unused. 79 80This document describes an approach for destroying semaphores that should work with all valid PE 81architectures, but will be described in terms of more common PE architectures (e.g. where the PE 82only backs each VkImage and VkSemaphore handle with one actual memory object, and where the PE 83cycles between the swapchain images in a straight-forward manner). 84 85The interested reader may follow the discussion in this abandoned [gerrit CL][CL1757018] for more 86background and ideas. 87 88[DisplayTimingGOOGLE]: https://www.khronos.org/registry/vulkan/specs/1.1-extensions/man/html/VK_GOOGLE_display_timing.html 89[VulkanIssue1060]: https://gitlab.khronos.org/vulkan/vulkan/issues/1060 90[CL1757018]: https://chromium-review.googlesource.com/c/angle/angle/+/1757018 91 92## Determining When a QP Semaphore is Waited On 93 94Let's combine the above diagrams with all the details: 95 96 CPU: ANI | QS | QP | ANI | QS | QP | ANI | W:F1 | QS | QP | ANI | W:F2 | QS | QP 97 I1 | I1 | I1 | I2 | I2 | I2 | I3 | | I3 | I3 | I1 | | I1 | I1 98 S:SA1 | W:SA1 | | S:SA2 | W:SA2 | | S:SA3 | | W:SA3 | | S:SA4 | | W:SA4 | 99 | S:SP1 | W:SP1 | | S:SP2 | W:SP2 | | | S:SP3 | W:SP3 | | | S:SP4 | W:SP4 100 | S:F1 | | | S:F2 | | | | S:F3 | | | | S:F4 | 101 102Let's focus only on sequences that return the same image: 103 104 CPU: ANI | W:F(X-2) | QS | QP | ... | ANI | W:F(Y-2) | QS | QP 105 I1 | | I1 | I1 | | I1 | | I1 | I1 106 S:SAX | | W:SAX | | | S:SAY | | W:SAY | 107 | | S:SPX | W:SPX | | | | S:SPY | W:SPY 108 | | S:FX | | | | | S:FY | 109 110Note that X and Y are arbitrarily distanced (including possibly being sequential). 111 112Say we are at frame Y+2. There's therefore a wait on FY. The following holds: 113 114 FY is signaled 115 => SAY is signaled 116 => The PE has handed I1 back to the application 117 => The PE has already processed the *previous* QP of I1 118 => SPX is waited on 119 120At this point, we can destroy SPX. In other words, in frame Y+2, we can destroy SPX (note that 2 is 121the number of frames the CPU pacing code uses). If frame Y+1 is not using I1, this means the 122history of present semaphores for I1 would be `{SPX, SPY}` and we can destroy the oldest semaphore 123in this list. If frame Y+1 is also using I1, we should still destroy SPX in frame Y+2, but the 124history of the present semaphores for I1 would be `{SPX, SPY, SP(Y+1)}`. 125 126In the Vulkan backend, we simplify destruction of semaphores by always keeping a history of 3 127present semaphores for each image (again, 3 is H+1 where H is the swap history size used in CPU 128pacing) and always reuse (instead of destroy) the oldest semaphore of the image that is about to be 129presented. 130 131To summarize, we use the completion of a submission using an image to prove when the semaphore used 132for the *previous* presentation of that image is no longer in use (and can be safely destroyed or 133reused). 134 135## Swapchain recreation 136 137When recreating the swapchain, all images are eventually freed and new ones are created, possibly 138with a different count and present mode. For the old swapchain, we can no longer rely on the 139completion of a future submission to know when a previous presentation's semaphore can be destroyed, 140as there won't be any more submissions using images from the old swapchain. 141 142> For example, imagine the old swapchain was created in FIFO mode, and one image is being presented 143> until the next V-Sync. Furthermore, imagine the new swapchain is created in MAILBOX mode. Since 144> the old swapchain's image will remain presented until V-Sync, the new MAILBOX swapchain can 145> perform an arbitrarily large number of (throw-away) presentations. The old swapchain (and its 146> associated present semaphores) cannot be destroyed until V-Sync; a signal that's not captured by 147> Vulkan. 148 149ANGLE resolves this issue by deferring the destruction of the old swapchain and its remaining 150present semaphores to the time when the semaphore corresponding to the first present of the new 151swapchain can be destroyed. In the example in the previous section, if SPX is the present semaphore 152of the first QP performed on the new swapchain, at frame Y+2, when we know SPX can be destroyed, we 153know that the first image of the new swapchain has already been presented. This proves that all 154previous QPs of the old swapchain have been processed. 155 156> Note: the swapchain can potentially be destroyed much earlier, but with no feedback from the 157> presentation engine, we cannot know that. This delays means that the swapchain could be recreated 158> while there are pending old swapchains to be destroyed. The destruction of both old swapchains 159> must now be deferred to when the first QP of the new swapchain has been processed. If an 160> application resizes the window constantly and at a high rate, ANGLE would keep accumulating old 161> swapchains and not free them until it stops. While a user will likely not be able to do this (as 162> the rate of window system events is lower than the framerate), this can be programmatically done 163> (as indeed done in EGL dEQP tests). Nvidia for example fails creation of a new swapchain if there 164> are already 20 allocated (on desktop, or less than ten on Quadro). If the backlog of old 165> swapchains get larger than a threshold, ANGLE calls `vkQueueWaitIdle()` and destroys the 166> swapchains. 167