1Name 2 3 QCOM_texture_foveated 4 5Name Strings 6 7 GL_QCOM_texture_foveated 8 9Contributors 10 11 Tate Hornbeck 12 Jonathan Wicks 13 Robert VanReenen 14 Matthew Hoffman 15 Jeff Leger 16 17Contact 18 19 Jeff Leger - jleger 'at' qti.qualcomm.com 20 21Status 22 23 Complete 24 25Version 26 27 Last Modified Date: Oct 06, 2017 28 Revision: #5 29 30Number 31 32 OpenGL ES Extension #293 33 34Dependencies 35 36 OpenGL ES 2.0 is required. This extension is written against OpenGL ES 3.2. 37 38 This interacts with QCOM_framebuffer_foveated. 39 40Overview 41 42 Foveated rendering is a technique that aims to reduce fragment processing 43 workload and bandwidth by reducing the average resolution of a render target. 44 Perceived image quality is kept high by leaving the focal point of 45 rendering at full resolution. 46 47 It exists in two major forms: 48 49 - Static foveated (lens matched) rendering: where the gaze point is 50 fixed with a large fovea region and designed to match up with the lens 51 characteristics. 52 - Eye-tracked foveated rendering: where the gaze point is continuously 53 tracked by a sensor to allow a smaller fovea region (further reducing 54 average resolution) 55 56 Traditionally foveated rendering involves breaking a render target's area 57 into smaller regions such as bins, tiles, viewports, or layers which are 58 rendered to individually. Each of these regions has the geometry projected 59 or scaled differently so that the net resolution of these layers is less 60 than the original render target's resolution. When these regions are mapped 61 back to the original render target, they create a rendered result with 62 decreased quality as pixels get further from the focal point. 63 64 Foveated rendering is currently achieved by large modifications to an 65 applications render pipelines to manually implement the required geometry 66 amplifications, blits, and projection changes. This presents a large 67 implementation cost to an application developer and is generally 68 inefficient as it can not make use of a platforms unique hardware features 69 or optimized software paths. This extension aims to address these problems 70 by exposing foveated rendering in an explicit and vendor neutral way, and by 71 providing an interface with minimal changes to how an application specifies 72 its render targets. 73 74New Tokens 75 76 Accepted as a value for <pname> for the TexParameter{if} and 77 TexParameter{if}v commands and for the <pname> parameter of 78 GetTexParameter{if}v: 79 80 TEXTURE_FOVEATED_FEATURE_BITS_QCOM 0x8BFB 81 TEXTURE_FOVEATED_MIN_PIXEL_DENSITY_QCOM 0x8BFC 82 83 Accepted as the <pname> parameter of GetTexParameter{if}v: 84 85 TEXTURE_FOVEATED_FEATURE_QUERY_QCOM 0x8BFD 86 TEXTURE_FOVEATED_NUM_FOCAL_POINTS_QUERY_QCOM 0x8BFE 87 88 Accepted as a value to <param> for the TexParameter{if} and 89 to <params> for the TexParameter{if}v commands with a <pname> of 90 TEXTURE_FOVEATED_FEATURE_BITS_QCOM; returned as possible values for 91 <params> when GetTexParameter{if}v is queried with a <pname> of 92 TEXTURE_FOVEATED_FEATURE_BITS_QCOM: 93 94 FOVEATION_ENABLE_BIT_QCOM 0x1 95 FOVEATION_SCALED_BIN_METHOD_BIT_QCOM 0x2 96 97 Returned by CheckFramebufferStatus: 98 99 FRAMEBUFFER_INCOMPLETE_FOVEATION_QCOM 0x8BFF 100 101Add new rows to Table 8.19 (Texture parameters and their values): 102 103 Name | Type | Legal Values 104 ------------------------------------------------------------ 105 TEXTURE_FOVEATED_FEATURE_BITS_QCOM | uint | 0, 106 FOVEATION_ENABLE_BIT_QCOM, 107 FOVEATION_ENABLE_BIT_QCOM | 108 FOVEATION_SCALED_BIN_METHOD_BIT_QCOM) 109 110 TEXTURE_FOVEATED_FEATURE_QUERY_QCOM | uint | 0, 111 FOVEATION_ENABLE_BIT_QCOM, 112 FOVEATION_ENABLE_BIT_QCOM | 113 FOVEATION_SCALED_BIN_METHOD_BIT_QCOM) 114 115 TEXTURE_FOVEATED_NUM_FOCAL_POINTS_QUERY_QCOM | uint | Any integer greater than 0 116 117 TEXTURE_FOVEATED_MIN_PIXEL_DENSITY_QCOM | float | Any float between 0.0 and 1.0 118 119Add new rows to Table 21.10 Textures (state per texture object) 120 121 Get value | Type | Get Command | Initial Value | Description | Sec 122 ------------------------------------------------------------------ 123 TEXTURE_FOVEATED_FEATURE_BITS_QCOM | Z+ | GetTexParameter{if}v | 0 | Foveation State | 8.19 124 125 TEXTURE_FOVEATED_FEATURE_QUERY_QCOM | Z+ | GetTexParameter{if}v | see sec 8.19 | Supported foveation state | 8.19 126 127 TEXTURE_FOVEATED_NUM_FOCAL_POINTS_QUERY_QCOM | Z+ | GetTexParameter{if}v | see sec 8.19 | Number of supported focal points per texture layer | 8.19 128 129 TEXTURE_FOVEATED_MIN_PIXEL_DENSITY_QCOM | R[0.0,1.0] | GetTexParameter{if}v | 0.0 | Minimum pixel density allowed | 8.19 130 131New Procedures and Functions 132 133 void TextureFoveationParametersQCOM(uint texture, 134 uint layer, 135 uint focalPoint, 136 float focalX, 137 float focalY, 138 float gainX, 139 float gainY, 140 float foveaArea); 141 142Additions to the end of section 8.19 of the OpenGL ES 3.2 Specification 143 144 TEXTURE_FOVEATED_FEATURE_QUERY_QCOM is a texture property that can only be 145 queried via GetTexParameter{if}v. This will return the implementation's 146 supported foveation methods. 147 148 glGetTexParameteriv(GL_TEXTURE_2D, 149 GL_TEXTURE_FOVEATED_FEATURE_QUERY_QCOM, 150 &query); 151 152 if ((query & FOVEATION_ENABLE_BIT_QCOM == FOVEATION_ENABLE_BIT_QCOM) && 153 (query & FOVEATION_SCALED_BIN_METHOD_BIT_QCOM == 154 FOVEATION_SCALED_BIN_METHOD_BIT_QCOM)) 155 { 156 // Implementation supports scaled bin method of foveation 157 } 158 159 TEXTURE_FOVEATED_NUM_FOCAL_POINTS_QUERY_QCOM is a texture property that 160 can only be queried GetTexParameter{if}v. This will return the number of 161 focal points per texture layer that the implementation supports. This must 162 be greater than 0. 163 164 TEXTURE_FOVEATED_MIN_PIXEL_DENSITY_QCOM defines the minimum pixel density 165 that can be used for scaled bin foveated rendering. 166 167 TEXTURE_FOVEATED_FEATURE_BITS_QCOM can be used to enable foveation for a 168 texture render target. 169 170 An explanation of each of the features is below: 171 172 FOVEATION_ENABLE_BIT_QCOM: Is used to enable foveated rendering, if 173 this bit is not specified foveated rendering will not be used. 174 175 FOVEATION_SCALED_BIN_METHOD_BIT_QCOM: Requests that the implementation 176 perform foveated rendering by dividing the texture render target into a 177 grid of subregions. Each subregions will be greater than or equal to one pixel 178 and less than or equal to the full size of the texture. Then rendering 179 the geometry to each of these regions with a different projection or scale. 180 Then, finally upscaling the subregion to the native texture resolution. 181 Quality in the scaled bin method is defined as a minimum pixel density 182 which is the ratio of the resolution rendered compared to the native 183 texture. 184 185 glTexParameteri(GL_TEXTURE_2D, 186 GL_TEXTURE_FOVEATED_FEATURE_BITS_QCOM, 187 GL_FOVEATION_ENABLE_BIT_QCOM | 188 GL_FOVEATION_SCALED_BIN_METHOD_BIT_QCOM); 189 190 This will set a texture as foveated so all subsequent rendering to 191 this texture will be foveated as long as the FOVEATION_ENABLE_BIT_QCOM 192 is set. Foveation is a texture property that is only applicable for 193 rendering operations, it does not affect traditional texture functions 194 like TexImage2D or TexSubImage2D. 195 196 The command 197 198 void TextureFoveationParametersQCOM(uint texture, 199 uint layer, 200 uint focalPoint, 201 float focalX, 202 float focalY, 203 float gainX, 204 float gainY, 205 float foveaArea); 206 207 is used to control the falloff of the foveated rendering of 'focalPoint' 208 for layer 'layer' in the texture object 'texture'. Multiple focal points 209 per layer are provided to enable foveated rendering when different regions 210 of a texture represent different views, such as with double wide 211 rendering. TEXTURE_FOVEATED_NUM_FOCAL_POINTS_QUERY_QCOM focal points are 212 supported per texture so values of 0 to 213 TEXTURE_FOVEATED_NUM_FOCAL_POINTS_QUERY_QCOM minus one are valid for the 214 'focalPoint' input and specify which focal point's data to update for the 215 layer. 216 217 'focalX' and 'focalY' is used to specify the x and y coordinate 218 of the focal point of the foveated texture in normalized device 219 coordinates. 'gainX' and 'gainY' are used to control how quickly the 220 quality falls off as you get further away from the focal point in each 221 axis. The larger these values are the faster the quality degrades. 222 'foveaArea' is used to control the minimum size of the fovea region, the 223 area before the quality starts to fall off. These parameters should be 224 modified to match the lens characteristics. 225 226 For the scaled bin method, these parameters define the minimum pixel 227 density allowed for a given focal point at the location (px,py) on a 228 texture layer in NDC as: 229 230 min_pixel_density=0.; 231 for(int i=0;i<focalPointsPerLayer;++i){ 232 focal_point_density = 1./max((focalX[i]-px)^2*gainX[i]^2+ 233 (focalY[i]-py)^2*gainY[i]^2-foveaArea[i],1.); 234 min_pixel_density=max(min_pixel_density,focal_point_density); 235 min_pixel_desnsity=max(min_pixel_desnsity, 236 TEXTURE_FOVEATED_MIN_PIXEL_DENSITY_QCOM); 237 } 238 239 While this function is continuous, it is worth noting that an 240 implementation is allowed to decimate to a fixed number of supported 241 quality levels, and it is allowed to group pixels into larger regions of 242 constant quality level, as long as the implementation at least provides 243 the quality level given in the above equation. 244 245 Future supported foveation methods could have different definitions of 246 quality. 247 248 The default values for each of the focal points in a layer is: 249 250 focalX = focalY = 0.0; 251 gainX = gainY = 0.0; 252 foveaArea = 0.0; 253 254 Which requires the entire render target to be rendered at full quality. 255 256 By specifying these constraints an application can fully constrain its 257 render quality while leaving the implementation enough flexibility to 258 render efficiently. 259 260Additions to Chapter 9.4 (Framebuffer Completeness) of the OpenGL ES 3.2 Specification 261 262 More than one color attachment is foveated. 263 264 { FRAMEBUFFER_INCOMPLETE_FOVEATION_QCOM } 265 266 Depth or stencil attachments are foveated textures. 267 268 { FRAMEBUFFER_INCOMPLETE_FOVEATION_QCOM } 269 270 The framebuffer has been configured for foveation via QCOM_framebuffer_foveated 271 and any color attachment is a foveated texture. 272 273 { FRAMEBUFFER_INCOMPLETE_FOVEATION_QCOM } 274 275Errors 276 277 INVALID_VALUE is generated by TextureFoveationParametersQCOM if 'texture' 278 is not a valid texture object. 279 280 INVALID_OPERATION is generated by TextureFoveationParametersQCOM if 281 'texture' has not been set as foveated. i.e. 'texture's parameter 282 TEXTURE_FOVEATED_FEATURE_BITS_QCOM does not contain 283 FOVEATION_ENABLE_BIT_QCOM. 284 285 INVALID_VALUE is generated by TextureFoveationParametersQCOM if 286 'focalPoint' is larger than TEXTURE_FOVEATED_NUM_FOCAL_POINTS_QUERY_QCOM 287 minus one. 288 289 INVALID_ENUM is generated by TexParameter{if} or TexParamter{if}v 290 if <pname> is TEXTURE_FOVEATED_FEATURE_QUERY_QCOM or 291 TEXTURE_FOVEATED_NUM_FOCAL_POINTS_QUERY_QCOM. 292 293 INVALID_ENUM is generated by TexParameter{if} or TexParamter{if}v 294 if <pname> is TEXTURE_FOVEATED_FEATURE_BITS_QCOM and <param> has 295 other bits set besides the legal values listed in table 8.19. 296 297 INVALID_OPERATION is generated by TexParameter{if} or TexParamter{if}v 298 if <pname> is TEXTURE_FOVEATED_FEATURE_BITS_QCOM and <param> does not have 299 FOVEATION_ENABLE_BIT_QCOM bit set, but the texture's parameter 300 TEXTURE_FOVEATED_FEATURE_BITS_QCOM already contains FOVEATION_ENABLE_BIT_QCOM. 301 i.e. Once foveation has been enabled for a texture, it cannot be disabled. 302 303 INVALID_OPERATION is generated by TexParameter{if} or TexParamter{if}v 304 if <pname> is TEXTURE_FOVEATED_MIN_PIXEL_DENSITY_QCOM and <param> is a 305 float less than 0.0 or greater than 1.0. 306 307 INVALID_OPERATION is generated by TexParameter{if} or TexParamter{if}v if 308 <pname> is GL_TEXTURE_FOVEATED_FEATURE_BITS_QCOM and <param> contains 309 FOVEATION_ENABLE_BIT_QCOM, but the query of TEXTURE_FOVEATED_FEATURE_QUERY_QCOM 310 of <target> does not contain FOVEATION_ENABLE_BIT_QCOM. 311 312Issues 313 314 1. Are texture arrays supported? 315 316 Texture arrays are supported to enable stereoscopic foveated 317 rendering which is a main use case of this extension. When a texture 318 array is used as a foveated render target, each slice has its own set 319 of foveation parameters. 320 321 2. How is foveation performed? 322 323 How foveated rendering is performed to the texture render target is implementation 324 defined. However, if 'FOVEATION_SCALED_BIN_METHOD_BIT_QCOM' is set the 325 implementation must perform foveation by dividing the render target into a 326 grid of subregions. Then rendering the geometry to each of these regions 327 with a different projection or scale. And finally upscaling the subregion 328 to the native full resolution render target. 329 330 When that bit is not set the implementation can use any algorithm it 331 wants for foveated rendering as long as it meets the application 332 requested features. 333 334 3. How are MRTs handled? 335 336 Only one color attachment may be foveated, all other color attachments 337 will inherit the foveated color attachment's foveation state. 338 339 4. Effect on screenspace shader built-ins 340 341 When using the scaled bin method, a number of screenspace built-ins 342 can produce unexpected results. 343 344 gl_FragCoord will be scaled to match the relative location in a 345 foveated texture. This means the absolute value of gl_FragCoord 346 will not be correct in lower resolution areas, but the value relative 347 to the full resolution will be consistent. 348 349 interpolateAtOffset, gl_PointSize, gl_SamplePosition, dFdx, dFdy, 350 glLineWidth will have no corrective scaling applied and thus could 351 have unexpected results. 352 353 5. How is depth/stencil handled? 354 355 Foveation cannot be enabled for depth or stencil texture attachments. However, 356 they will inherit foveation from a foveated color attachment attached to the 357 same framebuffer. In this case the depth and/or stencil attachments should 358 be discarded or invalidated after rendering, as the upscaled depth contents 359 are unlikely to be useful and may cause undesired rendering artifacts when used. 360 361 6. Does foveation have any effect on BlitFramebuffer. 362 363 No, there is no option to foveate a BlitFramebuffer blit. You can BlitFramebuffer 364 from a fbo with a foveated color attachment. 365 366 7. Rendering to a foveated texture multiple times per flush 367 368 The application must be careful to fully clear, or discard/invalidate, any 369 foveated attachments before rendering. Failure to do so would cause 370 unresolves of foveated content which may be undesirable (e.g. cases where 371 the foveation parameters or focal point has changed between resolves). 372 To prevent this the implementation may disable foveation for any rendering 373 to a foveated attachment that requires unresolves. Texture state related 374 to foveation, like TEXTURE_FOVEATED_FEATURE_BITS_QCOM and 375 TEXTURE_FOVEATED_MIN_PIXEL_DENSITY_QCOM will not be affected. 376 377 8. Interactions with QCOM_framebuffer_foveated 378 379 It is illegal to mix usage of these extensions. If a framebuffer has been 380 configured for foveation via QCOM_framebuffer_foveated, no attached textures 381 can be configured for foveation via QCOM_texture_foveated. The framebuffer 382 will be incomplete for this situation. 383 384 9. Implementation forced down non-tiled path 385 386 Certain feature sets may force an implementation to perform non tiled rendering. 387 The implemenation may implicitly disable foveation for these cases. Some 388 potential examples include tessellation, geometry shaders, or compute. 389 390Examples: 391 392 (1) Setup a foveated texture 393 394 // Allocate a texture 395 GLuint foveatedTexture; 396 glGenTextures(1, &foveatedTexture); 397 glBindTexture(GL_TEXTURE_2D, foveatedTexture); 398 ... 399 // Set texture as foveated 400 glTexParameteri(GL_TEXTURE_2D, 401 GL_TEXTURE_FOVEATED_FEATURE_BITS_QCOM, 402 GL_FOVEATION_ENABLE_BIT_QCOM | GL_FOVEATION_SCALED_BIN_METHOD_BIT_QCOM); 403 ... 404 // Rendering to foveatedTexture 405 ... 406 // Set foveation parameters on texture 407 glTextureFoveationParametersQCOM(foveatedTexture, 0, 0, focalX, focalY, gainX, gainY, 0.0f); 408 glFlush(); 409 410 (2) Setting parameters for a multiview stereo texture array 411 412 float focalX1 = 0.0f; // Gaze of left eye 413 float focalY1 = 0.0f; // Gaze of left eye 414 float focalX2 = 0.0f; // Gaze of right eye 415 float focalY2 = 0.0f; // Gaze of right eye 416 float gainX = 4.0f; // Weak foveation 417 float gainY = 4.0f; // Weak foveation 418 419 glTextureFoveationParametersQCOM(foveatedTextureArray, 420 0, 421 0, 422 focalX1, 423 focalY1, 424 gainX, 425 gainY, 426 2.0f); 427 glTextureFoveationParametersQCOM(foveatedTextureArray, 428 1, 429 0, 430 focalX2, 431 focalY2, 432 gainX, 433 gainY, 434 2.0f); 435 436 (3) Setting parameters for a double wide stereo texture 437 438 float focalX1 = -0.5f; // Gaze of left eye 439 float focalY1 = 0.0f; // Gaze of left eye 440 float focalX2 = 0.5f; // Gaze of right eye 441 float focalY2 = 0.0f; // Gaze of right eye 442 float gainX = 8.0f; // Strong foveation 443 float gainY = 8.0f; // Strong foveation 444 445 glTextureFoveationParametersQCOM(foveatedTexture, 446 0, 447 0, 448 focalX1, 449 focalY1, 450 gainX * 2.0f, 451 gainY, 452 8.0f); 453 glTextureFoveationParametersQCOM(foveatedTexture, 454 0, 455 1, 456 focalX2, 457 focalY2, 458 gainX * 2.0f, 459 gainY, 460 8.0f); 461 462Revision History 463 464 Rev. Date Author Changes 465 ---- -------- -------- ---------------------------------------------- 466 1 08/22/17 tateh Initial spec 467 2 09/20/17 tateh Clarified screenspace shader issues. Added 468 way to query number of supported focal points. 469 3 09/25/17 tateh Add max scale factor texture parameter. 470 4 10/3/17 tateh Add Table 21.10 modifications and issue #9 471 5 10/6/17 tateh Changed max scale factor to min pixel density 472 6 01/8/17 tateh Removed depth/stencil discard and invalidate 473 language. 474