Name EXT_YUV_target Name Strings GL_EXT_YUV_target Contributors Amit Bansal Arshad Bebal Jeff Leger Jing Zhou Maurice Ribble Prabindh Sundareson Jan-Harald Fredriksen Daniel Koch Jamie Gennis Mark Callow Pat Brown Andrew Garrard Etay Meiri Weifeng Zhang Contacts Jeff Leger Status Complete. Version April 6, 2018 (version 17) Number OpenGL ES Extension #222 Dependencies Requires OpenGL ES 3.0 and ESSL 3.0. Requires EGL 1.4 and following EGL extension: EGL_KHR_image_base OES_EGL_image_external This extension is written based on the wording of the OpenGL ES 3.0 Specification and OpenGL ES Shading Language 3.0. This extension interacts with OpenGL ES 3.1. See issue 1 and 13 for more details. This extension interacts with OES_tessellation_shader & OES_geometry shader see first issue for more details. This extension borrows texture function names from the OES_EGL_image_external_essl3. This extension is compatible with OES_EGL_image_external_essl3 and can coexist but does not require it. This extension interacts with EXT_shader_framebuffer_fetch. see issue 7 for more details. Overview This extension adds support for three new YUV related items: first rendering to YUV images, second sampling from YUV images while keeping the data in YUV space, third it defines a new built in function that does conversion from RGB to YUV with controls to choose ITU-R BT.601-7, ITU-R BT.601-7 Full range (JFIF images), or ITU-R BT.709-5 standard. This new functionality is layered on top of the OES_EGL_image_external extension. To perform the YUV rendering capability in this extension an application will attach a texture to the framebuffer object as the color attachment. If the texture has a target type of TEXTURE_EXTERNAL_OES with YUV color format then the GL driver can use this framebuffer object as the render target, TEXTURE_EXTERNAL_OES target with RGB color format are not allowed with this extension. New Types & Keywords A new OpenGL GLSL extension flag is added: #extension GL_EXT_YUV_target When the above GLSL extension is specified, one new sampler type will be available for sampling the 2D texture: __samplerExternal2DY2YEXT The "__samplerExternal2DY2YEXT" is used to sample a YUV texture image and output color value without any color conversion. Whenever a YUV sample is output from the sampler, the format of the YUV will be as if YUV 4:4:4 format is output. This also means that the Y sample maps to component R, the U sample maps to component G, the V sample maps to component B, and the component A will be 1.0f. The RGB sample output will be the same as in OpenGL ES specification. Here is one example: uniform __samplerExternal2DY2YEXT u_sTexture; When the above GLSL extension is specified, one new additional type will be available to specify color space standard formula for yuv to rgb transformation. yuvCscStandardEXT The value of new type can be specified using one of three new keywords itu_601, itu_601_full_range and itu_709. For example: yuvCscStandardEXT conv_standard = itu_601; yuvCscStandardEXT conv_standard = itu_601_full_range; yuvCscStandardEXT conv_standard = itu_709; Keyword itu_601 means color conversion is required using the formula specified in the ITU-R BT.601. Similarly keyword itu_601_full_range means conversion is done using ITU BT.601 full range formula as specified in JFIF standard while keyword itu_709 specify ITU-R BT.709 standard. Note that new yuvCscStandardEXT type can't be use for any other purpose apart from specifying color standard value to newly introduced color conversion built in functions mentioned below. New Built-in function When the new GLSL extension is specified, two new built in functions will be available for rgb to yuv or yuv to rgb color space conversion. vec3 rgb_2_yuv(vec3 color, yuvCscStandardEXT conv_standard); The function rgb_2_yuv will apply rgb to yuv color conversion transformation on "color" value using the formula specified as per new type yuvCscStandardEXT variable. The first input parameter supposed to specify rgb value using x, y & z channels of a vec3 variable, correspondingly return value of this function will have transformed y, u and v value in its x, y and z channel. Precision of the input color will define the precision used for color space conversion and for output yuv color value. vec3 yuv_2_rgb (vec3 color, yuvCscStandardEXT conv_standard); The function yuv_2_rgb will apply yuv to rgb color conversion transformation on "color" value using the formula specified as per new type yuvCscStandardEXT variable. The first input parameter supposed to specify yuv value using x, y & z channels of a vec3 variable, correspondingly return value of this function will have transformed r, g and b value in its x, y and z channel. Precision of the input color will define the precision used for color space conversion and for output yuv color value. New layout qualifier A shader which produces yuv format color output must qualify the fragment shader output variable with new yuv layout qualifier as described below. layout (yuv) out vec4 color; The new yuv layout qualifier can't be combined with any other layout qualifier, can only be used with fragment shader outputs and would be available only when the new GLSL extension is specified. Additionally if the shader qualifies fragment shader output with the new yuv qualifier and write depth or multiple color output, it would cause compilation failure. New Tokens Returned in the parameter of GetActiveUniform: SAMPLER_EXTERNAL_2D_Y2Y_EXT 0x8BE7 Reuse tokens as in extension "OES_EGL_image_external." Accepted as a target in the parameter of BindTexture, EGLImageTargetTexture2DOES, and FramebufferTexture2D: TEXTURE_EXTERNAL_OES 0x8D65 Accepted as in GetIntegerv() and GetFloatv() queries: TEXTURE_BINDING_EXTERNAL_OES 0x8D67 Accepted as in GetTexParameter*() queries: REQUIRED_TEXTURE_IMAGE_UNITS_OES 0x8D68 Addition to Chapter 3 of the OpenGL ES 3.0 specification Add a new section: "3.8.18 External Textures," which follows section "3.8.17 Shader Exponent Texture Color Conversion." The wording will be the same as in extension "OES_EGL_image_external." Changes to Section 4.4.2 "Attaching Images to Framebuffer Objects" At the end of subsection "Attaching Texture Images to a Framebuffer," add the following paragraphs: from "If texture is not zero, then texture must either name an existing two dimensional texture object and textarget must be TEXTURE_2D or texture must name an existing cube map...." to "If texture is not zero, then texture must either name an existing two dimensional texture object and textarget must be TEXTURE_2D or TEXTURE_EXTERNAL_OES or texture must name an existing cube map...." from "If textarget is TEXTURE_2D, level must be greater than or equal to zero and no larger than log2 of the value of MAX_TEXTURE_SIZE. Otherwise, an INVALID_VALUE error is generated. The command..." to "If textarget is TEXTURE_2D, level must be greater than or equal to zero and no larger than log2 of the value of MAX_TEXTURE_SIZE. Otherwise, an INVALID_VALUE error is generated. If textarget is TEXTURE_EXTERNAL_OES and attachment is other than COLOR_ATTACHMENT0, an INVALID_OPERATION error is generated. The command..." Add an additional sentence at the end of above paragraph: "Since TEXTURE_EXTERNAL_OES texture target can only be specified using an EGLImage, yuv formats can be rendered by attaching TEXTURE_EXTERNAL_OES textarget via EGLImage." Changes to Section 4.2.3 "Clearing the Buffers" from "Unsigned normalized fixed-point RGBA color buffers are cleared to color values derived by clamping each component of the clear color to the range [0; 1], then converting the (possibly sRGB converted and/or dithered) color to fixed-point using equations 2.3 or 2.4, respectively. The result of clearing integer color buffers with Clear is undefined." to "Unsigned normalized fixed-point RGBA color buffers are cleared to color values derived by clamping each component of the clear color to the range [0; 1], then converting the (possibly sRGB converted and/or dithered) color to fixed-point using equations 2.3 or 2.4, respectively. When clearing YUV Color Buffers, clear color should be defined in yuv color space and so floating point r, g, and b value will be mapped to corresponding y, u and v value and alpha channel will be ignored. The result of clearing integer color buffers with Clear is undefined." Changes to Section 4.4.3 "Fine Control of Buffer Updates" from "ColorMask is used to mask the writing of R, G, B and A values to all active draw buffers. r, g, b, and a indicate whether R, G, B, or A values, respectively, are written or not (a value of TRUE means that the corresponding value is written). In the initial state, all color values are enabled for writing for all draw buffers..." to "ColorMask is used to mask the writing of R, G, B and A values to all active draw buffers. r, g, b, and a indicate whether R, G, B, or A values, respectively, are written or not (a value of TRUE means that the corresponding value is written) except when the color buffer is in YUV color space, in that case setting any one of the r, g, b, and a values as false would generate draw time INVALID_OPERATION error. In the initial state, all color values are enabled for writing for all draw buffers..." Changes to Section 4.1.7 "Blending" from "Blending applies only if the color buffer has a fixed-point format. If the color buffer has an integer format, proceed to the next operation." to "Blending applies only if the color buffer has a fixed-point format and non YUV space format color buffer. If the color buffer has an integer, proceed to the next operation. If color buffer format is YUV color space, it will generate draw time INVALID_OPERATION error." Changes to 4.2.1 "Selecting Buffers for Writing" from "If some, but not all user-defined output variables are written, the values of fragment colors corresponding to unwritten variables are similarly undefined. The order of writes..." to "If some, but not all user-defined output variables are written, the values of fragment colors corresponding to unwritten variables are similarly undefined unless any of the fragment output is qualified with yuv format qualifier. In later case compiler would restrict usage of yuv format qualifier up to a single color output. The order of writes..." Changes to section 4.4.4 ("Framebuffer Completeness") from "An internal format is color-renderable if it is one of the formats from table 3.12 noted as color-renderable or if it is unsized format RGBA or RGB. No other formats," to "An internal format is color-renderable if it is one of the formats from table 3.12 noted as color-renderable or if it is unsized format RGBA or RGB or if it is YUV color space format supported by implementation. No other formats," Changes to section 3.7 ("Keywords") of the OpenGL ES Shading Language specification - Add to the list of keywords: __samplerExternal2DY2YEXT yuvCscStandardExt itu_601 itu_601_full_range itu_709 Changes to section 4.1 ("Basic Types") of the OpenGL ES Shading Language specification - Add to the list of Transparent types: "yuvCscStandardExt a value type to specify color space standard formula for transformation between yuv and rgb, taking values of itu_601, itu_601_full_range, itu_709" - Add to the list of Floating Point Sampler Types (opaque): "__samplerExternal2DY2YEXT a handle for accessing an external 2D texture whose underneath format is YUV" - Add a paragraph at the end of the section: "The __samplerExternal2DY2YEXT and yuvCscStandardExt types are optional and must be enabled by #extension GL_EXT_YUV_target : enable No type constructor or type conversion is allowed on yuvCscStandardExt. A value of itu_601, itu_601_full_range, itu_709, or a variable typed as yuvCscStandardExt can be used as an argument to user-defined functions and the built-in functions yuv_2_rgb and rgb_2_yuv. Other than the assignment operator, any other operations on this type of values or variables will be treated as a compilation error." Changes to section 4.5.4 ("Default Precision Qualifiers") of the OpenGL ES Shading Language specification - Add to the list of "vertex language has the following predeclared globally scoped default precision statements:" "precision lowp __samplerExternal2DY2YEXT;" - Add to the list of "fragment language has the following predeclared globally scoped default precision statements:" "precision lowp __samplerExternal2DY2YEXT;" Changes to section 8.8 ("Texture Lookup Functions") of the OpenGL ES Shading Language specification - Add to the table the following sampler functions: "highp ivec2 textureSize(__samplerExternal2DY2YEXT sampler, int lod) vec4 texture(__samplerExternal2DY2YEXT sampler, vec2 P [, float bias] ) vec4 textureProj(__samplerExternal2DY2YEXT sampler, vec3 P [, float bias] ) vec4 textureProj(__samplerExternal2DY2YEXT sampler, vec4 P [, float bias]) vec4 texelFetch(__samplerExternal2DY2YEXT sampler, ivec2 P, int lod)" Add a new section 8.10 "Conversion Functions between YUV and RGB" - Add a new table containing 2 entries: vec3 rgb_2_yuv(vec3 color, yuvCscStandardEXT conv_standard); apply rgb to yuv color conversion transformation on "color" value using the formula specified by conv_standard. vec3 yuv_2_rgb(vec3 color, yuvCscStandardEXT conv_standard); apply yuv to rgb color conversion transformation on "color" value using the formula specified by conv_standard. Compatibility with OpenGL ES 3.1 and ESSL 3.10 This extension is expected to work without modification with OpenGL ES 3.1 and ESSL 3.10. Some further clarification in issue 1, 9 and 13. Issues 1. Do we want to explicitly disallow "__samplerExternal2DY2YEXT" from use in other shader apart from fragment shader e.g. vertex, compute, geometry and tessellation shaders? RESOLVED: No, modern unified shader hardware supports YUV texture sampling in all of the shader stages, we should keep this enable irrespective of shader stage. 2. What happens if the input source isn't in the expected color space for built in functions or lets say there is cross conversion between different color space conversion standards e.g. 601 RGB to 709 YUV etc or yuv layout qualified color output is actually written with in rgb color space? RESOLVED: Implementation can't determine how the input pixel has been manipulated before producing the shader outputs, so results would be undefined in this case. 3. What happens if the source formats aren't the expected format for sampler? RESOLVED: Non-YUV format will generate a draw time INVALID_OPERATION error. 4. Is the depth or stencil behaviour changed when rendering to yuv surfaces. RESOLVED: The depth and stencil buffers will work the same when rendering to RGB or YUV surfaces, however when shader output is qualified with yuv layout qualifier, writing out gl_FragDepth would cause compilation failure. 5. Should this extension include support for renderbuffers or just renderable textures? RESOLVED: Renderbuffers doesn't add any functionality so it will not be included. 6. Is BlitFramebuffer or CopyTex[Sub]Image supported with YUV renderable surfaces? RESOLVED: No. There is a lot of driver complexity in supporting and testing case. Using these calls with a YUV source or destination will cause an INVALID_OPERATION error. 7. Does EXT_shader_framebuffer_fetch work with YUV renderable surfaces? RESOLVED: Yes. The next question is the result before or after sub-sampling? To give maximum flexibility to hardware either of these options is acceptable. 8. Should we add support for BT.2020? RESOLVED: Currently No, If require BT.2020 functionality can be achieved using __samplerExternal2DY2YEXT and user defined matrix. To keep the spec simple, we just want to have only three currently. 9. Should sampling return samples converted in to linear space or raw samples? RESOLVED: This extension will return raw samples. If the YUV data is encoded in a non-linear space, as is common, the returned samples will be in the non-linear space. If the data is encoded linearly, the returned samples will be linear. 10. Are there specific requirements on how a lower resolution UV plane must be sampled during rendering? RESOLVED: To allow maximum flexibility in hardware at this time this extension does not define how this sampling must be performed. Even if we decide to define this behaviour in future, it would be more appropriate to do it via a separate EGL extension. 11. How goes gl_FragCoord work with a lower resolution UV plane? RESOLVED: It is expected to work as if that plane was rendered at full resolution. The exception to this is talked about in issue #11 where there is some flexibility in subpixel sampling. 12. Should we add YUV format support for image load store in this extension or is it valid to use new yuv format layout qualifier with image load store ? RESOLVED: No, Since that would require major changes, we should do this with a separate extension, currently using new yuv format qualifier with image load store would cause compilation failure. 13. How can we defines color standard matrix other than provided via built-in functions e.g. full range BT.709 or adding compensation for the different colour primaries of the different standards ? RESOLVED: This extension defines built in utility functions for some of the common color conversion standard, however its no way limit an app to certain standard of output data. App can define own color matrix for such purpose and use with either __samplerExternal2DY2YEXT or sampler2D. 14. What happens if shader produce a color output qualified with yuv layout qualifier and current draw FBO format isn't YUV or draw FBO format is in YUV color space but shader output isn't qualified with yuv layout qualifier. RESOLVED: In order to write YUV color output, shader color output must be qualified with yuv layout qualifier and draw FBO format needs to be in YUV color space otherwise lack of any of these would cause INVALID_OPERATION error. 15. What happens if shader produce yuv color output which is out of range as per definition of color space format of rendering surface e.g. full range output when the output color space is limited or similarly input to one of built in color conversion functions or to the new __samplerExternal2DY2YEXT is out of range. RESOLVED: If shader yuv color output or input values to one of the built in color conversion functions or to the new __samplerExternal2DY2YEXT are not within the specified range of the format, results would be undefined values. 16. How ReadPixels operate for YUV render target? RESOLVED: ReadPixels will convert YUV to RGBA under the current spec requirement to support format RGBA and type UNSIGNED_BYTE combination for the normalized fixed point surfaces depending upon format and color space standard of currently bound rendering surface. Known Quality Issues Since YUV is a lossy compressed format and UV values represent sampling at even locations, there may be minor quality issues along the primitive edges or for scissor when its X or Y set to an odd value. In case of YUV primitive rendering if geometry is animated, there may be more artifacts as edges might flicker and sparkle frame to frame. Another way to explain this corruption is that sometimes the different planes are stored at different resolutions so the different interpolation for different planes causes these artifacts. Due to nature of CbCr subsampling, there might be artifacts along the primitive edges if there are partially covered pixels. Since commonly used 4:2:0 YUV formats writes out 1 pixel for a block of 2x2 pixels, depending upon how subsampled pixel output is calculated, there may always be artifacts around the primitive edges for partially covered pixels e.g. implementation might always decide to choose CbCr subsample cosited with top left pixel 2x2 Y plane block based on this single pixel coverage. This could help to avoid artifacts in more key cases e.g. along the diagonal of quad when its drawn using triangle strip but it may cause artifacts in other cases when we have partially covered pixels around the primitive edges visible after using very high zoom factor. Dependencies on EGL_KHR_image, EGL_KHR_image_base, and EGL 1.4 If EGL 1.2 is not supported, or if neither the EGL_OES_image nor EGL_OES_image_base extensions is supported, all discussion of EGLImages should be ignored, and any calls to either EGLImageTargetTexture2DOES or FramebufferTexture2D with TEXTURE_EXTERNAL_OES textarget should generate the error INVALID_OPERATION. Dependencies on OES_EGL_image_external This extension is built on top of the OES_EGL_image_external extension. It can coexist with the OES_EGL_image_external extension. Revision History #17 (April 6, 2018, H1Gdev) - Fix typo. #16 (June 8, 2016, Jeff Leger) - Add yuvCscStandardExt itu_* to the list of keywords - Add yuvCscStandardExt as a basic type and clarify that this type can only be used in assignments or as a function argument. - Add rgb_2_yuv and yuv_2_rgb as built-in functions (Bug 15704). #15 (April 23, 2015, Jon Leech) - Fix typo EGLImageTargetTexImage2DOES -> EGLImageTargetTexture2DOES (Bug 8114). #14 (February 10, 2015, Amit Bansal) - Removed YUV_EXT and Y_EXT tokens hard requirement for ReadPixels, current plan is to propose it later as separate extension once we have more corresponding compelling use cases. - Adding interaction wording with EXT_shader_framebuffer_fetch - Removed duplicated image load store issue. - Further clarification about usage of yuv layout qualifier #13 (January 23, 2015, Amit Bansal) - added texture built in functions from OES_EGL_image_external_essl3 - Added explicit clarification in overview section about RGB external images - Interaction with geometry & tessellation shaders w.r.t. issue 1 #12 (December 12, 2014, Amit Bansal) - Added new yuv qualifiers and corresponding draw time checks requirement issues - Clarification regarding yuvCscStandardEXT type usage - Output clamping requirement issue - New YUV_EXT and Y_EXT tokens for ReadPixels - Changed blending & color mask behaviour to draw time error. - Some language clean up #11 (September 30, 2014, Amit Bansal) - Changed couple of issues to spec language - Reworded some issues after recent spec updates #10 (August 12, 2013, Maurice Ribble) - Changed from QCOM to EXT extension - Reworded issue 16 based on Mark Callow's suggestion - Updated names of SAMPLER_EXTERNAL_2D_Y2Y_EXT, __samplerExternal2DY2YEXT, and yuvCscStandardEXT #9 (August 6, 2013, Maurice Ribble) - Added issues 15-18 #8 (July 12, 2013, Maurice Ribble) - Some language cleanup - Changed some edge case errors to be more consistent - Updates based on feedback from other khronos members - Added issues 10-14 #7 (June 27, 2013, Maurice Ribble) - Some language cleanup #6 (May 16, 2013, Amit Bansal) - Correcting spacing - Modified overview to keep only one sampler and add new built-in - Defined new types for built in and removed previous layouts - Removed the separate GL functions for external attachment - Modified description of sections meant for spec - Modified issues to reflect layout and built in function update - Changed proposed to resolved for finalized issues - Move precision issue to the description of built in function #5 (March 14, 2013, Jing zhou) - Minor rewording on same of proposals for the Issues - Assign new token IDs from reserved QCOM ID block #4 (March 4, 2013, Maurice Ribble) - First round of cleanup and todos #3 (February 19, 2013, Amit Bansal) - Modified glReadPixels()issue section #2 (January 9, 2013, Amit Bansal) - Changed the OES suffix to QCOM as extension is still pending to be approved. - Removed EGLImageTargetTexture2DOES from New Procedures and Functions category - Corrected error in FramebufferRenderbuffer2DExternalQCOM name at line 167. - Modified description of FramebufferTexture2DExternalQCOM to reflect argument changes correctly - Corrected typo extension to extension - Finished the documentation of final proposal for issue 2 & 3. - Added a new Known Quality Issue section and text corresponding to it #1 - (Oct 1, 2012) Original Draft