xref: /docs/source.android.com/src/devices/graphics/arch-tv.jd

page.title=TextureView
@jd:body

<!--
    Copyright 2014 The Android Open Source Project

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-->
<div id="qv-wrapper">
  <div id="qv">
    <h2>In this document</h2>
    <ol id="auto-toc">
    </ol>
  </div>
</div>


<p>The TextureView class introduced in Android 4.0 and is the most complex of
the View objects discussed here, combining a View with a SurfaceTexture.</p>

<h2 id=render_gles>Rendering with GLES</h2>
<p>Recall that the SurfaceTexture is a "GL consumer", consuming buffers of graphics
data and making them available as textures.  TextureView wraps a SurfaceTexture,
taking over the responsibility of responding to the callbacks and acquiring new
buffers.  The arrival of new buffers causes TextureView to issue a View
invalidate request.  When asked to draw, the TextureView uses the contents of
the most recently received buffer as its data source, rendering wherever and
however the View state indicates it should.</p>

<p>You can render on a TextureView with GLES just as you would SurfaceView.  Just
pass the SurfaceTexture to the EGL window creation call.  However, doing so
exposes a potential problem.</p>

<p>In most of what we've looked at, the BufferQueues have passed buffers between
different processes.  When rendering to a TextureView with GLES, both producer
and consumer are in the same process, and they might even be handled on a single
thread.  Suppose we submit several buffers in quick succession from the UI
thread.  The EGL buffer swap call will need to dequeue a buffer from the
BufferQueue, and it will stall until one is available.  There won't be any
available until the consumer acquires one for rendering, but that also happens
on the UI thread… so we're stuck.</p>

<p>The solution is to have BufferQueue ensure there is always a buffer
available to be dequeued, so the buffer swap never stalls.  One way to guarantee
this is to have BufferQueue discard the contents of the previously-queued buffer
when a new buffer is queued, and to place restrictions on minimum buffer counts
and maximum acquired buffer counts.  (If your queue has three buffers, and all
three buffers are acquired by the consumer, then there's nothing to dequeue and
the buffer swap call must hang or fail.  So we need to prevent the consumer from
acquiring more than two buffers at once.)  Dropping buffers is usually
undesirable, so it's only enabled in specific situations, such as when the
producer and consumer are in the same process.</p>

<h2 id=surface_or_texture>SurfaceView or TextureView?</h2>
SurfaceView and TextureView fill similar roles, but have very different
implementations.  To decide which is best requires an understanding of the
trade-offs.</p>

<p>Because TextureView is a proper citizen of the View hierarchy, it behaves like
any other View, and can overlap or be overlapped by other elements.  You can
perform arbitrary transformations and retrieve the contents as a bitmap with
simple API calls.</p>

<p>The main strike against TextureView is the performance of the composition step.
With SurfaceView, the content is written to a separate layer that SurfaceFlinger
composites, ideally with an overlay.  With TextureView, the View composition is
always performed with GLES, and updates to its contents may cause other View
elements to redraw as well (e.g. if they're positioned on top of the
TextureView).  After the View rendering completes, the app UI layer must then be
composited with other layers by SurfaceFlinger, so you're effectively
compositing every visible pixel twice.  For a full-screen video player, or any
other application that is effectively just UI elements layered on top of video,
SurfaceView offers much better performance.</p>

<p>As noted earlier, DRM-protected video can be presented only on an overlay plane.
 Video players that support protected content must be implemented with
SurfaceView.</p>

<h2 id=grafika>Case Study: Grafika's Play Video (TextureView)</h2>

<p>Grafika includes a pair of video players, one implemented with TextureView, the
other with SurfaceView.  The video decoding portion, which just sends frames
from MediaCodec to a Surface, is the same for both.  The most interesting
differences between the implementations are the steps required to present the
correct aspect ratio.</p>

<p>While SurfaceView requires a custom implementation of FrameLayout, resizing
SurfaceTexture is a simple matter of configuring a transformation matrix with
<code>TextureView#setTransform()</code>.  For the former, you're sending new
window position and size values to SurfaceFlinger through WindowManager; for
the latter, you're just rendering it differently.</p>

<p>Otherwise, both implementations follow the same pattern.  Once the Surface has
been created, playback is enabled.  When "play" is hit, a video decoding thread
is started, with the Surface as the output target.  After that, the app code
doesn't have to do anything -- composition and display will either be handled by
SurfaceFlinger (for the SurfaceView) or by TextureView.</p>

<h2 id=decode>Case Study: Grafika's Double Decode</h2>

<p>This activity demonstrates manipulation of the SurfaceTexture inside a
TextureView.</p>

<p>The basic structure of this activity is a pair of TextureViews that show two
different videos playing side-by-side.  To simulate the needs of a
videoconferencing app, we want to keep the MediaCodec decoders alive when the
activity is paused and resumed for an orientation change.  The trick is that you
can't change the Surface that a MediaCodec decoder uses without fully
reconfiguring it, which is a fairly expensive operation; so we want to keep the
Surface alive.  The Surface is just a handle to the producer interface in the
SurfaceTexture's BufferQueue, and the SurfaceTexture is managed by the
TextureView;, so we also need to keep the SurfaceTexture alive.  So how do we deal
with the TextureView getting torn down?</p>

<p>It just so happens TextureView provides a <code>setSurfaceTexture()</code> call
that does exactly what we want.  We obtain references to the SurfaceTextures
from the TextureViews and save them in a static field.  When the activity is
shut down, we return "false" from the <code>onSurfaceTextureDestroyed()</code>
callback to prevent destruction of the SurfaceTexture.  When the activity is
restarted, we stuff the old SurfaceTexture into the new TextureView.  The
TextureView class takes care of creating and destroying the EGL contexts.</p>

<p>Each video decoder is driven from a separate thread.  At first glance it might
seem like we need EGL contexts local to each thread; but remember the buffers
with decoded output are actually being sent from mediaserver to our
BufferQueue consumers (the SurfaceTextures).  The TextureViews take care of the
rendering for us, and they execute on the UI thread.</p>

<p>Implementing this activity with SurfaceView would be a bit harder.  We can't
just create a pair of SurfaceViews and direct the output to them, because the
Surfaces would be destroyed during an orientation change.  Besides, that would
add two layers, and limitations on the number of available overlays strongly
motivate us to keep the number of layers to a minimum.  Instead, we'd want to
create a pair of SurfaceTextures to receive the output from the video decoders,
and then perform the rendering in the app, using GLES to render two textured
quads onto the SurfaceView's Surface.</p>