1Android Hardware OpenGLES emulation design overview 2=================================================== 3 4Introduction: 5------------- 6 7Hardware GLES emulation in the Android platform is implemented with a mix 8of components, which are: 9 10 - Several host "translator" libraries. They implement the EGL, GLES 1.1 and 11 GLES 2.0 ABIs defined by Khronos, and translate the corresponding function 12 calls into calls to the appropriate desktop APIs, i.e.: 13 14 - Xgl (Linux), AGL (OS X) or WGL (Windows) for EGL 15 - desktop GL 2.0 for GLES 1.1 and GLES 2.0 16 17 _________ __________ __________ 18 | | | | | | HOST 19 |TRANSLATOR |TRANSLATOR| |TRANSLATOR| HOST 20 | EGL | | GLES 1.1 | | GLES 2.0 | TRANSLATOR 21 |_________| |__________| |__________| LIBRARIES 22 | | | 23 - - - | - - - - - - - - - | - - - - - - - - - | - - - - - 24 | | | 25 ____v____ ____v_____ _____v____ HOST 26 | | | | | | SYSTEM 27 | Xgl | | GL 2.0 | | GL 2.0 | LIBRARIES 28 |_________| |__________| |__________| 29 30 31 32 - Several system libraries inside the emulated guest system that implement 33 the same EGL / GLES 1.1 and GLES 2.0 ABIs. 34 35 They collect the sequence of EGL/GLES function calls and translate then 36 into a custom wire protocol stream that is sent to the emulator program 37 through a high-speed communication channel called a "QEMU Pipe". 38 39 For now, all you need to know is that the pipe is implemented with a 40 custom kernel driver, and provides for _very_ fast bandwidth. All read() 41 and writes() from/to the pipes are essentially instantaneous from the 42 guest's point of view. 43 44 45 _________ __________ __________ 46 | | | | | | 47 |EMULATION| |EMULATION | |EMULATION | GUEST 48 | EGL | | GLES 1.1 | | GLES 2.0 | SYSTEM 49 |_________| |__________| |__________| LIBRARIES 50 | | | 51 - - - | - - - - - - - - - | - - - - - - - - - | - - - - - 52 | | | 53 ____v____________________v____________________v____ GUEST 54 | | KERNEL 55 | QEMU PIPE | 56 |___________________________________________________| 57 | 58 - - - - - - - - - - - - - -|- - - - - - - - - - - - - - - - 59 | 60 v 61 EMULATOR 62 63 - Specific code inside the emulator program that is capable of transmitting 64 the wire protocol stream to a special rendering library or process (called 65 the "renderer" here), which understands the format. 66 67 | 68 | PROTOCOL BYTE STREAM 69 _____v_____ 70 | | 71 | EMULATOR | 72 |___________| 73 | 74 | UNMODIFIED PROTOCOL BYTE STREAM 75 _____v_____ 76 | | 77 | RENDERER | 78 |___________| 79 80 81 - The renderer decodes the EGL/GLES commands from the wire 82 protocol stream, and dispatches them to the translator libraries 83 appropriately. 84 85 | 86 | PROTOCOL BYTE STREAM 87 _____v_____ 88 | | 89 | RENDERER | 90 |___________| 91 | | | 92 +-----------------+ | +-----------------+ 93 | | | 94 ____v____ ___v______ ____v_____ 95 | | | | | | HOST 96 |TRANSLATOR |TRANSLATOR| |TRANSLATOR| HOST 97 | EGL | | GLES 1.1 | | GLES 2.0 | TRANSLATOR 98 |_________| |__________| |__________| LIBRARIES 99 100 101 102 - In reality, the protocol stream flows in both directions, even though most 103 of the commands result in data going from the guest to the host. A complete 104 picture of the emulation would thus be: 105 106 107 108 109 110 _________ __________ __________ 111 | | | | | | 112 |EMULATION| |EMULATION | |EMULATION | GUEST 113 | EGL | | GLES 1.1 | | GLES 2.0 | SYSTEM 114 |_________| |__________| |__________| LIBRARIES 115 ^ ^ ^ 116 | | | 117 - - - | - - - - - - - - - | - - - - - - - - - | - - - - - 118 | | | 119 ____v____________________v____________________v____ GUEST 120 | | KERNEL 121 | QEMU PIPE | 122 |___________________________________________________| 123 ^ 124 | 125 - - - - - - - - - - - - - -|- - - - - - - - - - - - - - - - 126 | 127 | PROTOCOL BYTE STREAM 128 _____v_____ 129 | | 130 | EMULATOR | 131 |___________| 132 ^ 133 | UNMODIFIED PROTOCOL BYTE STREAM 134 _____v_____ 135 | | 136 | RENDERER | 137 |___________| 138 ^ ^ ^ 139 | | | 140 +-----------------+ | +-----------------+ 141 | | | 142 ____v____ ___v______ ____v_____ 143 | | | | | | 144 |TRANSLATOR |TRANSLATOR| |TRANSLATOR| HOST 145 | EGL | | GLES 1.1 | | GLES 2.0 | TRANSLATOR 146 |_________| |__________| |__________| LIBRARIES 147 ^ ^ ^ 148 | | | 149 - - - | - - - - - - - - - | - - - - - - - - - | - - - - - 150 | | | 151 ____v____ ____v_____ _____v____ HOST 152 | | | | | | SYSTEM 153 | Xgl | | GL 2.0 | | GL 2.0 | LIBRARIES 154 |_________| |__________| |__________| 155 156 (NOTE: 'Xgl' is for Linux only, replace 'AGL' on OS X, and 'WGL' on Windows). 157 158 159Note that, in the above graphics, only the host system libraries at the bottom 160are _not_ provided by Android. 161 162 163Design Requirements: 164-------------------- 165 166The above design comes from several important requirements that were decided 167early in the project: 168 1691 - The ability to run the renderer in a separate process from the emulator 170 itself is important. 171 172 For various practical reasons, we plan to completely separate the core QEMU 173 emulation from the UI window by using two distinct processes. As such, the 174 renderer will be implemented as a library inside the UI program, but will 175 need to receive protocol bytes from the QEMU process. 176 177 The communication channel will be either a fast Unix socket or a Win32 178 named pipe between these two. A shared memory segment with appropriate 179 synchronization primitives might also be used if performance becomes 180 an issue. 181 182 This explains why the emulator doesn't alter or even try to parse the 183 protocol byte stream. It only acts as a dumb proxy between the guest 184 system and the renderer. This also avoids adding lots of GLES-specific 185 code inside the QEMU code base which is terribly complex. 186 1872 - The ability to use vendor-specific desktop EGL/GLES libraries is 188 important. 189 190 GPU vendors like NVidia, AMD or ARM all provide host versions of the 191 EGL/GLES libraries that emulate their respectivie embedded graphics 192 chipset. 193 194 The renderer library can be configured to use these instead of the 195 translator libraries provided with this project. This can be useful to 196 more accurately emulate the behaviour of specific devices. 197 198 Moreover, these vendor libraries typically expose vendor-specific 199 extensions that are not provided by the translator libraries. We cannot 200 expose them without modifying our code, but it's important to be able 201 to do so without too much pain. 202 203 204Code organization: 205------------------ 206 207All source code for the components above is spread over multiple directories 208in the Android source trees: 209 210 - The emulator sources are under $ANDROID/external/qemu, which we'll 211 call $QEMU in the rest of this document. 212 213 - The guest libraries are under 214 $ANDROID/development/tools/emulator/opengl, which we'll call $EMUGL_GUEST 215 216 - The host renderer and translator libraries are under 217 $ANDROID/sdk/emulator/opengl, which we'll call $EMUGL_HOST 218 219 - The QEMU Pipe kernel driver is under $KERNEL/drivers/misc/qemupipe 220 221Where $ANDROID is the top of the open-source Android source tree, and 222$KERNEL is the top of the qemu-specific kernel source tree (using one 223of the android-goldfish-xxxx branches here). 224 225The emulator sources related to this projects are: 226 227 $QEMU/hw/goldfish_pipe.c -> implement QEMU pipe virtual hardware 228 $QEMU/hw/opengles.c -> implement GLES initialization 229 $QEMU/hw/hw-pipe-net.c -> implements the communication channel 230 between the QEMU Pipe and the renderer library 231 232The other sources are: 233 234 $EMUGL_GUEST/system -> system libraries 235 $EMUGL_GUEST/shared -> guest copy of shared libraries 236 $EMUGL_GUEST/tests -> various test programs 237 $EMUGL_HOST/host -> host libraries (translator + renderer) 238 $EMUGL_HOST/shared -> host copy of shared libraries 239 $EMUGL_HOST/tests -> various test programs 240 241The reason the shared libraries aren't actually shared is historical: at one 242point both guest and host code lived in the same place. That turned out to be 243impractical with the way the Android SDK is branched, and didn't support the 244requirement that a single emulator binary be able to run several generations 245of Android. 246 247 248Translator libraries: 249--------------------- 250 251There are three translator host libraries provided by this project: 252 253 libEGL_translator -> EGL 1.2 translation 254 libGLES_CM_translator -> GLES 1.1 translation 255 libGLES_V2_translator -> GLES 2.0 translation 256 257The full name of the library will depend on the host system. 258For simplicity, only the library name suffix will change (i.e. the 259'lib' prefix is not dropped on Windows), i.e.: 260 261 libEGL_translator.so -> for Linux 262 libEGL_translator.dylib -> for OS X 263 libEGL_translator.dll -> for Windows 264 265The source code for these libraries is located under the following 266path in the Android source tree: 267 268 $EMUGL_HOST/host/libs/Translator/EGL 269 $EMUGL_HOST/host/libs/Translator/GLES_CM 270 $EMUGL_HOST/host/libs/Translator/GLES_V2 271 272The translator libraries also use a common routines defined under: 273 274 $EMUGL_HOST/host/libs/Translator/GLcommon 275 276 277Wire Protocol Overiew: 278---------------------- 279 280The "wire protocol" is implemented as follows: 281 282 - EGL/GLES function calls are described through several "specification" 283 files, which describes the types, function signatures and various 284 attributes for each one of them. 285 286 - These files are read by a tool called "emugen" which generates C 287 source files and headers based on the specification. These correspond 288 to both encoding, decoding and "wrappers" (more on this later). 289 290 - System "encoder" static libraries are built using some of these generated 291 files. They contain code that can serialize EGL/GLES calls into simple 292 byte messages and send it through a generic "IOStream" object. 293 294 - Host "decoder" static libraries are also built using some of these 295 generated files. Their code retrieves byte messages from an "IOStream" 296 object, and translates them into function callbacks. 297 298IOStream abstraction: 299- - - - - - - - - - - 300 301The "IOStream" is a very simple abstract class used to send byte messages 302both in the guest and host. It is defined through a shared header under 303$EMUGL/host/include/libOpenglRender/IOStream.h 304 305Note that despite the path, this header is included by *both* host and guest 306source code. The main idea around IOStream's design is that to send a message, 307one does the following: 308 309 1/ call stream->allocBuffer(size), which returns the address of a 310 memory buffer of at least 'size' bytes. 311 312 2/ write the content of the serialized command (usually a header + some 313 payload) directly into the buffer 314 315 3/ call stream->commitBuffer() to send it. 316 317Alternatively, one can also pack several commands into a single buffer with 318stream->alloc() and stream->flush(), as in: 319 320 1/ buf1 = stream->alloc(size1) 321 2/ write first command bytes into buf1 322 3/ buf2 = stream->alloc(size2) 323 4/ write second command bytes into buf2 324 5/ stream->flush() 325 326Finally, there are also explict read/write methods like stream->readFully() 327or stream->writeFully() which can be used when you don't want an intermediate 328buffer. This is used in certain cases by the implementation, e.g. to avoid 329an intermediate memory copy when sending texture data from the guest to the 330host. 331 332The host IOStream implementations are under $EMUGL/shared/OpenglCodecCommon/, 333see in particular: 334 335 $EMUGL_HOST/shared/OpenglCodecCommon/TcpStream.cpp 336 -> using local TCP sockets 337 $EMUGL_HOST/shared/OpenglCodecCommon/UnixStream.cpp 338 -> using Unix sockets 339 $EMUGL_HOST/shared/OpenglCodecCommon/Win32PipeStream.cpp 340 -> using Win32 named pipes 341 342The guest IOStream implementation uses the TcpStream.cpp above, as well as 343an alternative QEMU-specific source: 344 345 $EMUGL_GUEST/system/OpenglSystemCommon/QemuPipeStream.cpp 346 -> uses QEMU pipe from the guest 347 348The QEMU Pipe implementation is _significantly_ faster (about 20x) due to 349several reasons: 350 351 - all succesful read() and write() operations through it are instantaneous 352 from the guest's point of view. 353 354 - all buffer/memory copies are performed directly by the emulator, and thus 355 much faster than performing the same thing inside the kernel with emulated 356 ARM instructions. 357 358 - it doesn't need to go through a kernel TCP/IP stack that will wrap the 359 data into TCP/IP/MAC packets, send them to an emulated ethernet device, 360 which is itself connected to an internal firewall implementation that 361 will unwrap the packets, re-assemble them, then send them through BSD 362 sockets to the host kernel. 363 364However, would it be necessary, you could write a guest IOStream implementation 365that uses a different transport. If you do, please look at 366$EMUGL_GUEST/system/OpenglCodecCommon/HostConnection.cpp which contains the 367code used to connect the guest to the host, on a per-thread basis. 368 369 370Source code auto-generation: 371- - - - - - - - - - - - - - 372 373The 'emugen' tool is located under $EMUGL_HOST/host/tools/emugen. There is a 374README file that explains how it works. 375 376You can also look at the following specifications files: 377 378For GLES 1.1: 379 $EMUGL_HOST/host/GLESv1_dec/gl.types 380 $EMUGL_HOST/host/GLESv1_dec/gl.in 381 $EMUGL_HOST/host/GLESv1_dec/gl.attrib 382 383For GLES 2.0: 384 $EMUGL_HOST/host/GLESv2_dec/gl2.types 385 $EMUGL_HOST/host/GLESv2_dec/gl2.in 386 $EMUGL_HOST/host/GLESv2_dec/gl2.attrib 387 388For EGL: 389 $EMUGL_HOST/host/renderControl_dec/renderControl.types 390 $EMUGL_HOST/host/renderControl_dec/renderControl.in 391 $EMUGL_HOST/host/renderControl_dec/renderControl.attrib 392 393Note that the EGL specification files are under a directory named 394"renderControl_dec" and have filenames that begin with "renderControl" 395 396This is mainly for historic reasons now, but is also related to the fact that 397this part of the wire protocol contains support functions/calls/specifications 398that are not part of the EGL specification itself, but add a few features 399required to make everything works. For example, they have calls related to 400the "gralloc" system library module used to manage graphics surfaces at a 401lower level than EGL. 402 403Generally speaking, guest encoder sources are located under directories 404named $EMUGL_GUEST/system/<name>_enc/, while the corresponding host decoder 405sources will be under $EMUGL_HOST/host/libs/<name>_dec/ 406 407However, all these sources use the same spec files located under the 408decoding directories. 409 410The encoder files are built from emugen and spec files located in $EMUGL_HOST 411and copied to the encoder directories in $EMUGL_GUEST by the gen-encoder.sh 412script. They are checked in, so that a given version of Android supports a 413specific version of the protocol, even if newer versions of the renderer (and 414future Android versions) support a newer protocol version. This step needs to 415be done manually when the protocol changes; these changes also need to be 416accompanied by changes in the renderer to handle the old version of the 417protocol. 418 419 420System libraries: 421----------------- 422 423Meta EGL/GLES system libraries, and egl.cfg: 424- - - - - - - - - - - - - - - - - - - - - - 425 426It is important to understand that the emulation-specific EGL/GLES libraries 427are not directly linked by applications at runtime. Instead, the system 428provides a set of "meta" EGL/GLES libraries that will load the appropriate 429hardware-specific libraries on first use. 430 431More specifically, the system libEGL.so contains a "loader" which will try 432to load: 433 434 - hardware-specific EGL/GLES libraries 435 - the software-based rendering libraries (called "libagl") 436 437The system libEGL.so is also capable of merging the EGL configs of both the 438hardware and software libraries transparently to the application. The system 439libGLESv1_CM.so and libGLESv2.so, work with it to ensure that the thread's 440current context will be linked to either the hardware or software libraries 441depending on the config selected. 442 443For the record, the loader's source code in under 444frameworks/base/opengl/libs/EGL/Loader.cpp. It depends on a file named 445/system/lib/egl/egl.cfg which must contain two lines that look like: 446 447 0 1 <name> 448 0 0 android 449 450The first number in each line is a display number, and must be 0 since the 451system's EGL/GLES libraries don't support anything else. 452 453The second number must be 1 to indicate hardware libraries, and 0 to indicate 454a software one. The line corresponding to the hardware library, if any, must 455always appear before the one for the software library. 456 457The third field is a name corresponding to a shared library suffix. It really 458means that the corresponding libraries will be named libEGL_<name>.so, 459libGLESv1_CM_<name>.so and libGLESv2_<name>.so. Moreover these libraries must 460be placed under /system/lib/egl/ 461 462The name "android" is reserved for the system software renderer. 463 464The egl.cfg that comes with this project uses the name "emulation" for the 465hardware libraries. This means that it provides an egl.cfg file that contains 466the following lines: 467 468 0 1 emulation 469 0 0 android 470 471See $EMUGL_GUEST/system/egl/egl.cfg and more generally the following build 472files: 473 474 $EMUGL_GUEST/system/egl/Android.mk 475 $EMUGL_GUEST/system/GLESv1/Android.mk 476 $EMUGL_GUEST/system/GLESv2/Android.mk 477 478to see how the libraries are named and placed under /system/lib/egl/ by the 479build system. 480 481 482Emulation libraries: 483- - - - - - - - - - - 484 485The emulator-specific libraries are under the following: 486 487 $EMUGL_GUEST/system/egl/ 488 $EMUGL_GUEST/system/GLESv1/ 489 $EMUGL_GUEST/system/GLESv2/ 490 491The code for GLESv1 and GLESv2 is pretty small, since it mostly link against 492the static encoding libraries. 493 494The code for EGL is a bit more complex, because it needs to deal with 495extensions dynamically. I.e. if an extension is not available on the host 496it shouldn't be exposed by the library at runtime. So the EGL code queries 497the host for the list of available extensions in order to return them to 498clients. Similarly, it must query the list of valid EGLConfigs for the 499current host system. 500 501 502"gralloc" module implementation: 503- - - - - - - - - - - - - - - - - 504 505In addition to EGL/GLES libraries, the Android system requires a 506hardware-specific library to manage graphics surfaces at a level lower than 507EGL. This library must be what is called in Android land as a "HAL module". 508 509A "HAL module" must provide interfaces defined by Android's HAL 510(Hardware Abstraction Library). These interface definitions can be found 511under $ANDROID/hardware/libhardware/include/ 512 513Of all possible HAL modules, the "gralloc" one is used by the system's 514SurfaceFlinger to allocate framebuffers and other graphics memory regions, 515as well as eventually lock/unlock/swap them when needed. 516 517The code under $EMUGL/system/gralloc/ implements the module required by the 518GLES emulation project. It's not very long, but there are a few things to 519notice here: 520 521- first, it will probe the guest system to determine if the emulator that 522 is running the virtual device really supports GPU emulation. In certain 523 circumstances this may not be possible. 524 525 If this is the case, then the module will redirect all calls to the 526 "default" gralloc module that is normally used by the system when 527 software-only rendering is enabled. 528 529 The probing happens in the function "fallback_init" which gets called 530 when the module is first opened. This initializes the 'sFallback' variable 531 to a pointer to the default gralloc module when required. 532 533- second, this module is used by SurfaceFlinger to display "software surfaces", 534 i.e. those that are backed by system memory pixel buffers, and written to 535 directly through the Skia graphics library (i.e. the non-accelerated ones). 536 537 the default module simply copies the pixel data from the surface to the 538 virtual framebuffer i/o memory, but this project's gralloc module sends it 539 to the renderer through the QEMU Pipe instead. 540 541 It turns out that this results in _faster_ rendering/frame-rates overall, 542 because memory copies inside the guest are slow, while QEMU pipe transfers 543 are done directly in the emulator. 544 545 546Host Renderer: 547-------------- 548 549The host renderer library is located under 550$EMUGL_HOST/host/libs/libOpenglRender, and it provides an interface described 551by the headers under $EMUGL_HOST/host/include/libOpenglRender/render_api.h 552(e.g. for use by the emulator). 553 554In a nutshell, the rendering library is responsible for the following: 555 556 - Providing a virtual off-screen video surface where everything will get 557 rendered at runtime. Its dimensions are fixed by the call to 558 initOpenglRender() that must happen just after the library is 559 initialized. 560 561 - Provide a way to display the virtual video surface on a host application's 562 UI. This is done by calling createOpenGLSubWindow() which takes as argument 563 the window ID or handle of a parent window, some display dimensions and 564 a rotation angle. This allows the surface to be scaled/rotated when it is 565 displayed, even if the dimensions of the video surface do not change. 566 567 - Provide a way to listen to incoming EGL/GLES commands from the guest. 568 This is done by providing a so-called "port number" to initOpenglRender(). 569 570 By default, the port number corresponds to a local TCP port number that the 571 renderer will bind to and listen. Every new connection to this port will 572 correspond to the creation of a new guest host connection, each such 573 connection corresponding to a distinct thread in the guest system. 574 575 For performance reasons, it is possible to listen to either Unix sockets 576 (on Linux and OS X), or to a Win32 named pipe (on Windows). To do so, one 577 had to call setStreamType() between library initialization 578 (i.e. initLibrary()) and construction (i.e. initOpenglRender()). 579 580 Note that in these modes, the port number is still used to differentiate 581 between several emulator instances. These details are normally handled by 582 the emulator code so you shouldn't care too much. 583 584Note that an earlier version of the interface allowed a client of the renderer 585library to provide its own IOStream implementation. However, this wasn't very 586convenient for a number of reasons. This maybe something that could be done 587again if it makes sense, but for now the performance numbers are pretty good. 588 589 590Host emulator: 591-------------- 592 593The code under $QEMU/android/opengles.c is in charge of dynamically loading 594the rendering library and initializing / constructing it properly. 595 596QEMU pipe connections to the 'opengles' service are piped through the code 597in $QEMU/android/hw-pipe-net.c. Look for the openglesPipe_init() function, 598which is in charge of creating a connection to the renderer library 599(either through a TCP socket, or a Unix pipe depending on configuration. 600support for Win32 named pipes hasn't been implemented yet in the emulator) 601whenever a guest process opens the "opengles" service through /dev/qemu_pipe. 602 603There is also some support code for the display of the GLES framebuffer 604(through the renderer library's subwindow) under $QEMU/skin/window. 605 606Note that at the moment, scaling and rotation are supported. However, 607brightness emulation (which used to modify the pixel values from the 608hardware framebuffer before displaying them) doesn't work. 609 610Another issue is that it is not possible to display anything on top of the 611GL subwindow at the moment. E.g. this will obscure the emulated trackball 612image (that is normally toggled with Ctrl-T during emulation, or enabled 613by pressing the Delete key). 614 615