1page.title=Implementing the Hardware Composer HAL 2@jd:body 3 4<!-- 5 Copyright 2016 The Android Open Source Project 6 7 Licensed under the Apache License, Version 2.0 (the "License"); 8 you may not use this file except in compliance with the License. 9 You may obtain a copy of the License at 10 11 http://www.apache.org/licenses/LICENSE-2.0 12 13 Unless required by applicable law or agreed to in writing, software 14 distributed under the License is distributed on an "AS IS" BASIS, 15 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 See the License for the specific language governing permissions and 17 limitations under the License. 18--> 19 20<div id="qv-wrapper"> 21 <div id="qv"> 22 <h2>In this document</h2> 23 <ol id="auto-toc"> 24 </ol> 25 </div> 26</div> 27 28 29<p>The Hardware Composer HAL (HWC) is used by SurfaceFlinger to composite 30surfaces to the screen. The HWC abstracts objects such as overlays and 2D 31blitters and helps offload some work that would normally be done with OpenGL.</p> 32 33<p>Android 7.0 includes a new version of HWC (HWC2) used by SurfaceFlinger to 34talk to specialized window composition hardware. SurfaceFlinger contains a 35fallback path that uses the 3D graphics processor (GPU) to perform the task of 36window composition, but this path is not ideal for a couple of reasons:</p> 37 38<ul> 39 <li>Typically, GPUs are not optimized for this use case and may use more power 40 than necessary to perform composition.</li> 41 <li>Any time SurfaceFlinger is using the GPU for composition is time that 42 applications cannot use the processor for their own rendering, so it is 43 preferable to use specialized hardware for composition instead of the GPU 44 whenever possible.</li> 45</ul> 46 47<h2 id="guidance">General guidance</h2> 48 49<p>As the physical display hardware behind the Hardware Composer abstraction 50layer can vary from device to device, it's difficult to give recommendations on 51specific features. In general, use the following guidance:</p> 52 53<ul> 54 <li>The HWC should support at least four overlays (status bar, system bar, 55 application, and wallpaper/background).</li> 56 <li>Layers can be bigger than the screen, so the HWC should be able to handle 57 layers that are larger than the display (for example, a wallpaper).</li> 58 <li>Pre-multiplied per-pixel alpha blending and per-plane alpha blending 59 should be supported at the same time.</li> 60 <li>The HWC should be able to consume the same buffers the GPU, camera, and 61 video decoder are producing, so supporting some of the following 62 properties is helpful: 63 <ul> 64 <li>RGBA packing order</li> 65 <li>YUV formats</li> 66 <li>Tiling, swizzling, and stride properties</li> 67 </ul> 68 <li>To support protected content, a hardware path for protected video playback 69 must be present.</li> 70 </ul> 71 72<p>The general recommendation is to implement a non-operational HWC first; after 73the structure is complete, implement a simple algorithm to delegate composition 74to the HWC (for example, delegate only the first three or four surfaces to the 75overlay hardware of the HWC).</p> 76 77<p>Focus on optimization, such as intelligently selecting the surfaces to send 78to the overlay hardware that maximizes the load taken off of the GPU. Another 79optimization is to detect whether the screen is updating; if it isn't, delegate 80composition to OpenGL instead of the HWC to save power. When the screen updates 81again, continue to offload composition to the HWC.</p> 82 83<p>Prepare for common use cases, such as:</p> 84 85<ul> 86 <li>Full-screen games in portrait and landscape mode</li> 87 <li>Full-screen video with closed captioning and playback control</li> 88 <li>The home screen (compositing the status bar, system bar, application 89 window, and live wallpapers)</li> 90 <li>Protected video playback</li> 91 <li>Multiple display support</li> 92</ul> 93 94<p>These use cases should address regular, predictable uses rather than edge 95cases that are rarely encountered (otherwise, optimizations will have little 96benefit). Implementations must balance two competing goals: animation smoothness 97and interaction latency.</p> 98 99 100<h2 id="interface_activities">HWC2 interface activities</h2> 101 102<p>HWC2 provides a few primitives (layer, display) to represent composition work 103and its interaction with the display hardware.</p> 104<p>A <em>layer</em> is the most important unit of composition; every layer has a 105set of properties that define how it interacts with other layers. Property 106categories include the following:</p> 107 108<ul> 109<li><strong>Positional</strong>. Defines where the layer appears on its display. 110Includes information such as the positions of a layer's edges and its <em>Z 111order</em> relative to other layers (whether it should be in front of or behind 112other layers).</li> 113<li><strong>Content</strong>. Defines how content displayed on the layer should 114be presented within the bounds defined by the positional properties. Includes 115information such as crop (to expand a portion of the content to fill the bounds 116of the layer) and transform (to show rotated or flipped content).</li> 117<li><strong>Composition</strong>. Defines how the layer should be composited 118with other layers. Includes information such as blending mode and a layer-wide 119alpha value for 120<a href="https://en.wikipedia.org/wiki/Alpha_compositing#Alpha_blending">alpha 121compositing</a>.</li> 122<li><strong>Optimization</strong>. Provides information not strictly necessary 123to correctly composite the layer, but which can be used by the HWC device to 124optimize how it performs composition. Includes information such as the visible 125region of the layer and which portion of the layer has been updated since the 126previous frame.</li> 127</ul> 128 129<p>A <em>display</em> is another important unit of composition. Every layer can 130be present only on one display. A system can have multiple displays, and 131displays can be added or removed during normal system operations. This 132addition/removal can come at the request of the HWC device (typically in 133response to an external display being plugged into or removed from the device, 134called <em>hotplugging</em>), or at the request of the client, which permits the 135creation of <em>virtual displays</em> whose contents are rendered into an 136off-screen buffer instead of to a physical display.</p> 137<p>HWC2 provides functions to determine the properties of a given display, to 138switch between different configurations (e.g., 4k or 1080p resolution) and color 139modes (e.g., native color or true sRGB), and to turn the display on, off, or 140into a low-power mode if supported.</p> 141<p>In addition to layers and displays, HWC2 also provides control over the 142hardware vertical sync (VSYNC) signal along with a callback into the client to 143notify it of when a vsync event has occurred.</p> 144 145<h3 id="func_pointers">Function pointers</h3> 146<p>In this section and in HWC2 header comments, HWC interface functions are 147referred to by lowerCamelCase names that do not actually exist in the interface 148as named fields. Instead, almost every function is loaded by requesting a 149function pointer using <code>getFunction</code> provided by 150<code>hwc2_device_t</code>. For example, the function <code>createLayer</code> 151is a function pointer of type <code>HWC2_PFN_CREATE_LAYER</code>, which is 152returned when the enumerated value <code>HWC2_FUNCTION_CREATE_LAYER</code> is 153passed into <code>getFunction</code>.</p> 154<p>For detailed documentation on functions (including functions required for 155every HWC2 implementation), refer to the 156<a href="{@docRoot}devices/halref/hwcomposer2_8h.html">HWC2 header</a>.</p> 157 158<h3 id="layer_display_handles">Layer and display handles</h3> 159<p>Layers and displays are manipulated by opaque handles.</p> 160<p>When SurfaceFlinger wants to create a new layer, it calls the 161<code>createLayer</code> function, which then returns an opaque handle of type 162<code>hwc2_layer_t</code>. From that point on, any time SurfaceFlinger wants to 163modify a property of that layer, it passes that <code>hwc2_layer_t</code> value 164into the appropriate modification function, along with any other information 165needed to make the modification. The <code>hwc2_layer_t</code> type was made 166large enough to be able to hold either a pointer or an index, and it will be 167treated as opaque by SurfaceFlinger to provide HWC implementers maximum 168flexibility.</p> 169<p>Most of the above also applies to display handles, though handles are created 170differently depending on whether they are hotplugged (where the handle is passed 171through the hotplug callback) or requested by the client as a virtual display 172(where the handle is returned from <code>createVirtualDisplay</code>).</p> 173 174<h2 id="display_comp_ops">Display composition operations</h2> 175<p>Once per hardware vsync, SurfaceFlinger wakes if it has new content to 176composite. This new content could be new image buffers from applications or just 177a change in the properties of one or more layers. When it wakes, it performs the 178following steps:</p> 179 180<ol> 181<li>Apply transactions, if present. Includes changes in the properties of layers 182specified by the window manager but not changes in the contents of layers (i.e., 183graphic buffers from applications).</li> 184<li>Latch new graphic buffers (acquire their handles from their respective 185applications), if present.</li> 186<li>If step 1 or 2 resulted in a change to the display contents, perform a new 187composition (described below).</li> 188</ol> 189 190<p>Steps 1 and 2 have some nuances (such as deferred transactions and 191presentation timestamps) that are outside the scope of this section. However, 192step 3 involves the HWC interface and is detailed below.</p> 193<p>At the beginning of the composition process, SurfaceFlinger will create and 194destroy layers or modify layer state as applicable. It will also update the 195layers with their current contents, using calls such as 196<code>setLayerBuffer</code> or <code>setLayerColor</code>. After all layers have 197been updated, it will call <code>validateDisplay</code>, which tells the device 198to examine the state of the various layers and determine how composition will 199proceed. By default, SurfaceFlinger usually attempts to configure every layer 200such that it will be composited by the device, though there may be some 201circumstances where it will mandate that it be composited by the client.</p> 202<p>After the call to <code>validateDisplay</code>, SurfaceFlinger will follow up 203with a call to <code>getChangedCompositionTypes</code> to see if the device 204wants any of the layers' composition types changed before performing the actual 205composition. SurfaceFlinger may choose to:</p> 206 207<ul> 208<li>Change some of the layer composition types and re-validate the display.</li> 209</ul> 210 211<blockquote><strong><em>OR</strong></em></blockquote> 212 213<ul> 214<li>Call <code>acceptDisplayChanges</code>, which has the same effect as 215changing the composition types as requested by the device and re-validating 216without actually having to call <code>validateDisplay</code> again.</li> 217</ul> 218 219<p>In practice, SurfaceFlinger always takes the latter path (calling 220<code>acceptDisplayChanges</code>) though this behavior may change in the 221future.</p> 222<p>At this point, the behavior differs depending on whether any of the layers 223have been marked for client composition. If any (or all) layers have been marked 224for client composition, SurfaceFlinger will now composite all of those layers 225into the client target buffer. This buffer will be provided to the device using 226the <code>setClientTarget</code> call so that it may be either displayed 227directly on the screen or further composited with layers that have not been 228marked for client composition. If no layers have been marked for client 229composition, then the client composition step is bypassed.</p> 230<p>Finally, after all of the state has been validated and client composition has 231been performed if needed, SurfaceFlinger will call <code>presentDisplay</code>. 232This is the HWC device's cue to complete the composition process and display the 233final result.</p> 234 235<h2 id="multiple_displays">Multiple displays in Android 7.0</h2> 236<p>While the HWC2 interface is quite flexible when it comes to the number of 237displays in the system, the rest of the Android framework is not yet as 238flexible. When designing a HWC2 implementation intended for use on Android 7.0, 239there are some additional restrictions not present in the HWC definition itself: 240</p> 241 242<ul> 243<li>It is assumed that there is exactly one <em>primary</em> display; that is, 244that there is one physical display that will be hotplugged immediately during 245the initialization of the device (specifically after the hotplug callback is 246registered).</li> 247<li>In addition to the primary display, exactly one <em>external</em> display 248may be hotplugged during normal operation of the device.</li> 249</ul> 250 251<p>While the SurfaceFlinger operations described above are performed per-display 252(eventual goal is to be able to composite displays independently of each other), 253they are currently performed sequentially for all active displays, even if only 254the contents of one display are updated.</p> 255<p>For example, if only the external display is updated, the sequence is:</p> 256 257<pre> 258// Update state for internal display 259// Update state for external display 260validateDisplay(<internal display>) 261validateDisplay(<external display>) 262presentDisplay(<internal display>) 263presentDisplay(<external display>) 264</pre> 265 266 267<h2 id="sync_fences">Synchronization fences</h2> 268<p>Synchronization (sync) fences are a crucial aspect of the Android graphics 269system. Fences allow CPU work to proceed independently from concurrent GPU work, 270blocking only when there is a true dependency.</p> 271<p>For example, when an application submits a buffer that is being produced on 272the GPU, it will also submit a fence object; this fence signals only when the 273GPU has finished writing into the buffer. Since the only part of the system that 274truly needs the GPU write to have finished is the display hardware (the hardware 275abstracted by the HWC HAL), the graphics pipeline is able to pass this fence 276along with the buffer through SurfaceFlinger to the HWC device. Only immediately 277before that buffer would be displayed does the device need to actually check 278that the fence has signaled.</p> 279<p>Sync fences are integrated tightly into HWC2 and organized in the following 280categories:</p> 281 282<ol> 283<li>Acquire fences are passed along with input buffers to the 284<code>setLayerBuffer</code> and <code>setClientTarget</code> calls. These 285represent a pending write into the buffer and must signal before the HWC client 286or device attempts to read from the associated buffer to perform composition. 287</li> 288<li>Release fences are retrieved after the call to <code>presentDisplay</code> 289using the <code>getReleaseFences</code> call and are passed back to the 290application along with buffers that will be replaced during the next 291composition. These represent a pending read from the buffer, and must signal 292before the application attempts to write new contents into the buffer.</li> 293<li>Retire fences are returned, one per frame, as part of the call to 294<code>presentDisplay</code> and represent when the composition of this frame 295has completed, or alternately, when the composition result of the prior frame is 296no longer needed. For physical displays, this is when the current frame appears 297on the screen and can also be interpreted as the time after which it is safe to 298write to the client target buffer again (if applicable). For virtual displays, 299this is the time when it is safe to read from the output buffer.</li> 300</ol> 301 302<h3 id="hwc2_changes">Changes in HWC2</h3> 303<p>The meaning of sync fences in HWC 2.0 has changed significantly relative to 304previous versions of the HAL.</p> 305<p>In HWC v1.x, the release and retire fences were speculative. A release fence 306for a buffer or a retire fence for the display retrieved in frame N would not 307signal any sooner than frame N + 1. In other words, the meaning of the fence 308was "the content of the buffer you provided for frame N is no longer needed." 309This is speculative because in theory SurfaceFlinger may not run again after 310frame N for an indeterminate period of time, which would leave those fences 311unsignaled for the same period.</p> 312<p>In HWC 2.0, release and retire fences are non-speculative. A release or 313retire fence retrieved in frame N will signal as soon as the content of the 314associated buffers replaces the contents of the buffers from frame N - 1, or in 315other words, the meaning of the fence is "the content of the buffer you provided 316for frame N has now replaced the previous content." This is non-speculative, 317since this fence should signal shortly after <code>presentDisplay</code> is 318called as soon as the hardware presents this frame's content.</p> 319<p>For implementation details, refer to the 320<a href="{@docRoot}devices/halref/hwcomposer2_8h.html">HWC2 header</a>.</p> 321