The Android framework has a variety of graphics rendering APIs for 2D and 3D that interact with your HAL implementations and graphics drivers, so it is important to have a good understanding of how they work at a higher level. There are two general ways that app developers can draw things to the screen: with Canvas or OpenGL.
android.graphics.Canvas is a 2D graphics API and is the most widely used graphics API by developers. Canvas operations draw all the stock android.view.Views and custom android.view.Views in Android. Prior to Android 3.0, Canvas always used the non-hardware accelerated Skia 2D drawing library to draw.
Introduced in Android 3.0, hardware acceleration for Canvas APIs uses a new drawing library called OpenGLRenderer that translates Canvas operations to OpenGL operations so that they can execute on the GPU. Developers had to opt-in to this feature previously, but beginning in Android 4.0, hardware-accelerated Canvas is enabled by default. Consequently, a hardware GPU that supports OpenGL ES 2.0 is mandatory for Android 4.0 devices.
Additionally, the Hardware Acceleration guide explains how the hardware-accelerated drawing path works and identifies the differences in behavior from the software drawing path.
The other main way that developers render graphics is by using OpenGL ES 1.x or 2.0 to directly render to a surface. Android provides OpenGL ES interfaces in the android.opengl package that a developer can use to call into your GL implementation with the SDK or with native APIs provided in the Android NDK.
Note:A third option, Renderscript, was introduced in Android 3.0 to serve as a platform-agnostic graphics rendering API (it used OpenGL ES 2.0 under the hood), but will be deprecated starting in the Android 4.1 release.
No matter what rendering API developers use, everything is rendered onto a buffer of pixel data called a "surface." Every window that is created on the Android platform is backed by a surface. All of the visible surfaces that are rendered to are composited onto the display by the SurfaceFlinger, Android's system service that manages composition of surfaces. Of course, there are more components that are involved in graphics rendering, and the main ones are described below:
SurfaceTextureClient, ISurfaceTexture, and
SurfaceTexture (in this case, SurfaceTexture is the actual C++ class and not
the name of the overall component). These three parts facilitate the producer (SurfaceTextureClient),
binder (ISurfaceTexture), and consumer (SurfaceTexture)
components of SurfaceTexture in processes such as requesting memory from Gralloc,
sharing memory across process boundaries, synchronizing access to buffers, and pairing the appropriate consumer with the producer.
SurfaceTexture can operate in both asynchronous (producer never blocks waiting for consumer and drops frames) and
synchronous (producer waits for consumer to process textures) modes. Some examples of image
producers are the camera preview produced by the camera HAL or an OpenGL ES game. Some examples
of image consumers are SurfaceFlinger or another app that wants to display an OpenGL ES stream
such as the camera app displaying the camera viewfinder.
hardware/libhardware/include/hardware/gralloc.h Hardware composer section
for more information.
The following diagram shows how these components work together:
Figure 1. How surfaces are rendered
The following list and sections describe what you need to provide to support graphics in your product:
You must provide drivers for OpenGL ES 1.x, OpenGL ES 2.0, and EGL. Some key things to keep in mind are:
GL_OES_texture_external,
EGL_ANDROID_image_native_buffer, and EGL_ANDROID_recordable. We highly
recommend supporting EGL_ANDROID_blob_cache and EGL_KHR_fence_sync as
well.Note that the OpenGL API exposed to app developers is different from the OpenGL interface that you are implementing. Apps do not have access to the GL driver layer, and must go through the interface provided by the APIs.
Many times, hardware overlays do not support rotation, so the solution is to pre-transform the buffer before
it reaches SurfaceFlinger. A query hint in ANativeWindow was added (NATIVE_WINDOW_TRANSFORM_HINT)
that represents the most likely transform to be be applied to the buffer by SurfaceFlinger.
Your GL driver can use this hint to pre-transform the buffer before it reaches SurfaceFlinger, so when the buffer
actually reaches SurfaceFlinger, it is correctly transformed. See the ANativeWindow
interface defined in system/core/include/system/window.h for more details. The following
is some pseudo-code that implements this in the hardware composer:
ANativeWindow->query(ANativeWindow, NATIVE_WINDOW_DEFAULT_WIDTH, &w); ANativeWindow->query(ANativeWindow, NATIVE_WINDOW_DEFAULT_HEIGHT, &h); ANativeWindow->query(ANativeWindow, NATIVE_WINDOW_TRANSFORM_HINT, &hintTransform); if (hintTransform & HAL_TRANSFORM_ROT_90) swap(w, h); native_window_set_buffers_dimensions(anw, w, h); ANativeWindow->dequeueBuffer(...); // here GL driver renders content transformed by " hintTransform " int inverseTransform; inverseTransform = hintTransform; if (hintTransform & HAL_TRANSFORM_ROT_90) inverseTransform ^= HAL_TRANSFORM_ROT_180; native_window_set_buffers_transform(anw, inverseTransform); ANativeWindow->queueBuffer(...);
The graphics memory allocator is needed to allocate memory that is requested by
SurfaceTextureClient in image producers. You can find a stub implementation of the HAL at
hardware/libhardware/modules/gralloc.h
There is a gralloc usage flag GRALLOC_USAGE_PROTECTED that allows
the graphics buffer to be displayed only through a hardware protected path.
The hardware composer is used by SurfaceFlinger to composite surfaces to the screen. The hardware composer abstracts things like overlays and 2D blitters and helps offload some things that would normally be done with OpenGL.
Jellybean MR1 introduces a new version of the HAL. We recommend that you start using version 1.1 of the hardware composer HAL as it will provide support for the newest features (explicit synchronization, external displays, etc). Keep in mind that in addition to 1.1 version, there is also a 1.0 version of the HAL that we used for internal compatibility reasons and a 1.2 draft mode of the hardware composer HAL. We recommend that you implement version 1.1 until 1.2 is out of draft mode.
Because the physical display hardware behind the hardware composer abstraction layer can vary from device to device, it is difficult to define recommended features, but here is some guidance:
The general recommendation when implementing your hardware composer is to implement a no-op hardware composer first. Once you have the structure done, implement a simple algorithm to delegate composition to the hardware composer. For example, just delegate the first three or four surfaces to the overlay hardware of the hardware composer. After that focus on common use cases, such as:
After implementing the common use cases, you can focus on optimizations such as intelligently selecting the surfaces to send to the overlay hardware that maximizes the load taken off of the GPU. Another optimization is to detect whether the screen is updating. If not, delegate composition to OpenGL instead of the hardware composer to save power. When the screen updates again, contin`ue to offload composition to the hardware composer.
You can find the HAL for the hardware composer in the
hardware/libhardware/include/hardware/hwcomposer.h and hardware/libhardware/include/hardware/hwcomposer_defs.h
files. A stub implementation is available in the hardware/libhardware/modules/hwcomposer directory.
VSYNC synchronizes certain events to the refresh cycle of the display. Applications always start drawing on a VSYNC boundary and SurfaceFlinger always composites on a VSYNC boundary. This eliminates stutters and improves visual performance of graphics. The hardware composer has a function pointer
int (waitForVsync*) (int64_t *timestamp)
that points to a function you must implement for VSYNC. This function blocks until a VSYNC happens and returns the timestamp of the actual VSYNC. A client can receive a VSYNC timestamps once, at specified intervals, or continously (interval of 1). You must implement VSYNC to have no more than a 1ms lag at the maximum (1/2ms or less is recommended), and the timestamps returned must be extremely accurate.
Explicit synchronization is required in Jellybean MR1 and later and provides a mechanism for Gralloc buffers to be acquired and released in a synchronized way. Explicit synchronization allows producers and consumers of graphics buffers to signal when they are done with a buffer. This allows the Android system to asynchronously queue buffers to be read or written with the certainty that another consumer or producer does not currently need them.
This communication is facilitated with the use of synchronization fences, which are now required when requesting a buffer for consuming or producing. The synchronization framework consists of three main parts:
sync_timeline: a monotonically increasing timeline that should be implemented
for each driver instance. This basically is a counter of jobs submitted to the kernel for a particular piece of hardware.sync_pt: a single value or point on a sync_timeline. A point
has three states: active, signaled, and error. Points start in the active state and transition
to the signaled or error states. For instance, when a buffer is no longer needed by an image
consumer, this sync_point is signaled so that image producers
know that it is okay to write into the buffer again.sync_fence: a collection of sync_pts that often have different
sync_timeline parents (such as for the display controller and GPU). This allows
multiple consumers or producers to signal that
they are using a buffer and to allow this information to be communicated with one function parameter.
Fences are backed by a file descriptor and can be passed from kernel-space to user-space.
For instance, a fence can contain two sync_points that signify when two separate
image consumers are done reading a buffer. When the fence is signaled,
the image producers now know that both consumers are done consuming.To implement explicit synchronization, you need to do provide the following:
system/core/include/sync/sync.h file for more implementation details. The
system/core/libsync directory includes a library to communicate with the kernel-space set() and prepare() functions in the HAL. As a last resort,
you can pass in -1 for the file descriptor parameters if you cannot support explicit synchronization for some reason. This
is not recommended, however.EGL_ANDROID_native_fence_sync and EGL_ANDROID_wait_sync,
along with incorporating fence support into your graphics drivers.