1Name 2 3 QCOM_framebuffer_foveated 4 5Name Strings 6 7 GL_QCOM_framebuffer_foveated 8 9Contributors 10 11 Skyler Saleh 12 Maurice Ribble 13 Tate Hornbeck 14 Jonathan Wicks 15 Robert VanReenen 16 17Contact 18 19 Jeff Leger - jleger 'at' qti.qualcomm.com 20 21Status 22 23 Complete 24 25Version 26 27 Last Modified Date: May 10, 2017 28 Revision: #11 29 30Number 31 32 OpenGL ES Extension #273 33 34Dependencies 35 36 OpenGL ES 2.0 is required. This extension is written against OpenGL ES 3.2. 37 38Overview 39 40 Foveated rendering is a technique that aims to reduce fragment processing 41 workload and bandwidth by reducing the average resolution of a framebuffer. 42 Perceived image quality is kept high by leaving the focal point of 43 rendering at full resolution. 44 45 It exists in two major forms: 46 47 - Static foveated(lens matched) rendering: where the gaze point is 48 fixed with a large fovea region and designed to match up with the lens 49 characteristics. 50 - Eye-tracked foveated rendering: where the gaze point is continuously 51 tracked by a sensor to allow a smaller fovea region (further reducing 52 average resolution) 53 54 Traditionally foveated rendering involves breaking a framebuffer's area 55 into smaller regions such as bins, tiles, viewports, or layers which are 56 rendered to individually. Each of these regions has the geometry projected 57 or scaled differently so that the net resolution of these layers is less 58 than the original framebuffer's resolution. When these regions are mapped 59 back to the original framebuffer, they create a rendered result with 60 decreased quality as pixels get further from the focal point. 61 62 Foveated rendering is currently achieved by large modifications to an 63 applications render pipelines to manually implement the required geometry 64 amplifications, blits, and projection changes. This presents a large 65 implementation cost to an application developer and is generally 66 inefficient as it can not make use of a platforms unique hardware features 67 or optimized software paths. This extension aims to address these problems 68 by exposing foveated rendering in an explicit and vendor neutral way, and by 69 providing an interface with minimal changes to how an application specifies 70 its framebuffer. 71 72New Tokens 73 74 Allowed in the config input in FramebufferFoveationConfigQCOM: 75 76 FOVEATION_ENABLE_BIT_QCOM 0x1 77 FOVEATION_SCALED_BIN_METHOD_BIT_QCOM 0x2 78 79New Procedures and Functions 80 81 void FramebufferFoveationConfigQCOM(uint fbo, 82 uint numLayers, 83 uint focalPointsPerLayer, 84 uint requestedFeatures, 85 uint *providedFeatures); 86 87 void FramebufferFoveationParametersQCOM(uint fbo, 88 uint layer, 89 uint focalPoint, 90 float focalX, 91 float focalY, 92 float gainX, 93 float gainY, 94 float foveaArea); 95 96Additions to Chapter 9 of the OpenGL ES 3.2 Specification 97 98 The command 99 100 void FramebufferFoveationConfigQCOM( uint fbo, uint numLayers, 101 uint focalPointsPerLayer, uint requestedFeatures, uint *providedFeatures); 102 103 is used to configure foveated rendering for the framebuffer object 'fbo' 104 and to instruct the implementation to allocate any additional 105 intermediate resources needed for foveated rendering. 106 107 In order to enable foveation, this call must be issued prior to any 108 operation which causes data to be written to a framebuffer attachment. 109 Once this call is made for a framebuffer object, the fbo will remain a 110 "foveated fbo". The following scenarios are unsupported conditions: 111 112 1. Rendering to a non foveated fbo, then calling 113 FramebufferFoveationConfigQCOM results in current framebuffer content 114 becoming undefined. 115 116 2. Rendering to a foveated fbo then switching attachments results in an 117 error. 118 119 Each layer of a foveated framebuffer, the max of which is specified by 120 'numLayers', can have multiple focal points as controlled by 121 'focalPointsPerLayer'. This enables applications that make use of double 122 wide rendering to utilize foveated rendering and also allows more complex 123 falloff characteristics to be modeled with multiple overlapping focal 124 points. There are limitations to the number of focal points that an 125 implementation can support with different enabled features or framebuffer 126 formats. When an implementation can not support having as many focal points 127 per layer as was specified in this function, it will fail with the error 128 INVALID_VALUE. It is recommended that an application utilize as 129 few focal points per layer as possible. 130 131 The 'requestedFeatures' bitfield is used to specify which features an 132 application would like to use. 133 134 An explanation of each of the features is below: 135 136 FOVEATION_ENABLE_BIT_QCOM: Is used to enable foveated rendering, if 137 this bit is not specified foveated rendering will not be used. 138 139 FOVEATION_SCALED_BIN_METHOD_BIT_QCOM: Requests that the implementation 140 perform foveated rendering by dividing the framebuffer into a grid of 141 subregions. Each subregions will be greater than or equal to one pixel 142 and less than or equal to the full framebuffer. Then rendering the geometry 143 to each of these regions with a different projection or scale. Then, finally 144 upscaling the subregion to the native full resolution framebuffer. 145 Quality in the scaled bin method is defined as a minimum pixel density 146 which is the ratio of the resolution rendered compared to the native 147 framebuffer. 148 149 In the future it is expected that more features will be added, without 150 breaking backwards compatibility. 151 152 'providedFeatures' is a pointer to a uint that will be set to a new bitfield 153 that tells the application which features the implementation provided for the 154 current foveation configuration, in the same format as used in the 'requested 155 Features' bitfield. This may include more or fewer features than the application 156 requested. The FOVEATION_ENABLE_BIT_QCOM will not be included if the 157 implementation has to fallback to non-foveated rendering. 158 159 If an application tries to make use of a feature that is not included in the 160 'providedFeatures' bitfield, the results of the operation are implementation 161 defined, but should not yield application termination. 162 163 If this command is called with 'requestedFeatures' equal to zero, then the value 164 of 'providedFeatures' will have FOVEATION_ENABLE_BIT_QCOM unset, and the other 165 bits will be set or unset to indicate which foveation features are supported 166 by the implementation. 167 168 The command 169 170 void FramebufferFoveationParametersQCOM(uint fbo,uint layer, uint focalPoint, 171 float focalX, float focalY, float gainX, float gainY, float foveaArea); 172 173 is used to control the falloff of the foveated rendering of 'focalPoint' 174 for layer 'layer' in the framebuffer object 'fbo'. Multiple focal points 175 per layer are provided to enable foveated rendering when different regions 176 of a framebuffer represent different views, such as with double wide 177 rendering. Values of 0 to the framebuffers focalPointsPerLayer-1 are valid 178 for the 'focalPoint' input and specify which focal point's data to update 179 for the layer. 180 181 'focalX' and 'focalY' is used to specify the x and y coordinate 182 of the focal point of the foveated framebuffer in normalized device 183 coordinates. 'gainX' and 'gainY' are used to control how quickly the 184 quality falls off as you get further away from the focal point in each 185 axis. The larger these values are the faster the quality degrades. 186 'foveaArea' is used to control the minimum size of the fovea region, the 187 area before the quality starts to fall off. These parameters should be 188 modified to match the lens characteristics. 189 190 For the scaled bin method, these parameters define the minimum pixel 191 density allowed for a given focal point at the location (px,py) on a 192 framebuffer layer in NDC as: 193 194 min_pixel_density=0.; 195 for(int i=0;i<focalPointsPerLayer;++i){ 196 focal_point_density = 1./max((focalX[i]-px)^2*gainX[i]^2+ 197 (focalY[i]-py)^2*gainY[i]^2-foveaArea[i],1.); 198 min_pixel_density=max(min_pixel_density,focal_point_density); 199 } 200 201 While this function is continuous, it is worth noting that an 202 implementation is allowed to decimate to a fixed number of supported 203 quality levels, and it is allowed to group pixels into larger regions of 204 constant quality level, as long as the implementation at least provides 205 the quality level given in the above equation. 206 207 Future supported foveation methods could have different definitions of 208 quality. 209 210 The default values for each of the focal points in a layer is: 211 212 focalX=focalY=0; 213 gainX=gainY=0; 214 foveaArea=0; 215 216 Which requires the entire framebuffer to be rendered at full quality. 217 218 By specifying these constraints an application can fully constrain its 219 render quality while leaving the implementation enough flexibility to 220 render efficiently. 221 222Errors 223 224 OUT_OF_MEMORY is generated by FramebufferFoveationConfigQCOM if an 225 implementation runs out of memory when trying to reserve the needed 226 additional resources for the foveated framebuffer. 227 228 INVALID_VALUE is generated by FramebufferFoveationConfigQCOM if 'fbo' is 229 not a valid framebuffer. 230 231 INVALID_VALUE is generated by FramebufferFoveationConfigQCOM if 'numLayers' 232 is greater than GL_MAX_ARRAY_TEXTURE_LAYERS - 1. 233 234 INVALID_VALUE is generated by FramebufferFoveationConfigQCOM if 235 'numFocalPoints' is greater than implementation can support. 236 237 INVALID_OPERATION is generated by FramebufferFoveationConfigQCOM if it is 238 called for a fbo that has already been cofigured for foveated rendering. 239 240 INVALID_VALUE is generated by FramebufferFoveationParametersQCOM if 'fbo' 241 is not a valid framebuffer. 242 243 INVALID_OPERATION is generated by FramebufferFoveationParametersQCOM if 244 'fbo' has not been configured for foveated rendering. 245 246 INVALID_VALUE is generated by FramebufferFoveationParametersQCOM if 247 'layer' is greater than or equal to the numLayers that the fbo was 248 previously configured for in FramebufferFoveationConfigQCOM. 249 250 INVALID_VALUE is generated by FramebufferFoveationParametersQCOM if 251 'numFocalPoints' is greater than implementation can support. 252 253 INVALID_OPERATION is generated by any API call which causes a framebuffer 254 attachment to be written to if the framebuffer attachments have changed for 255 a foveated fbo. 256 257 INVALID_OPERATION is generated if a rendering command is issued and the 258 current bound program uses tessellation or geometry shaders. 259 260Issues 261 262 1. Are layered framebuffers supported? 263 264 Layered framebuffers are supported to enable stereoscopic foveated 265 rendering which is a main use case of this extension. When a layered 266 framebuffer is used each layer has its own set of foveation parameters. 267 268 2. How is foveation performed? 269 270 How foveated rendering is performed to the framebuffer is implementation 271 defined. However, if 'FOVEATION_SCALED_BIN_METHOD_BIT_QCOM' is set the 272 implementation must perform foveation by dividing the framebuffer into a 273 grid of subregions. Then rendering the geometry to each of these regions 274 with a different projection or scale. And finally upscaling the subregion 275 to the native full resolution framebuffer. 276 277 When that bit is not set the implementation can use any algorithm it 278 wants for foveated rendering as long as it meets the application 279 requested features. 280 281 3. How are MRTs handled? 282 283 Every framebuffer attachment uses the same quality in a given region. 284 285 4. How does gl_FragCoord work? 286 287 When using the scaled bin method, gl_FragCoord will be scaled to match 288 the relative location in a foveated framebuffer. This means the absolute 289 value of gl_FragCoord will not be correct in lower resolution areas, 290 but the value relative to the full resolution will be consistent. 291 292 5. How is depth handled? 293 294 Depth surfaces can be used during foveated rendering, but the contents 295 of the depth buffer will not be resolved out. The implementation can 296 do an implicit discard of the depth buffer. The reasoning here is that 297 the upsample of depth during resolve would produce irrelevant results. 298 299 6. How does unresolving from a foveated framebuffer work? 300 301 Loading from a foveated framebuffer is undefined, so the app must be 302 sure not to trigger mid frame flushes. In the dynamic foveation case 303 the focal point can move constantly. If a region A of a frame 0 was 304 rendered at a lower quality because the focal point was far away, 305 then the focal point moved to cover region A during frame 1, the 306 unresolve could not reconstruct the full quality region A. The 307 app must be careful to fully clear the surface and remove mid frame 308 flushes to prevent unresolves. 309 310Examples: 311 312 (1) Initialize a foveated framebuffer 313 314 // Allocate and initialize a regular framebuffer and attachments 315 GLuint fbo = createFramebufferAndAttachments(); 316 GLuint providedFeatures; 317 glFramebufferFoveationConfigQCOM(fbo,1,1, GL_FOVEATION_ENABLE_BIT_QCOM, &providedFeatures); 318 if(!(providedFeatures & GL_FOVEATION_ENABLE_BIT_QCOM)) { 319 // Failed to enable foveation 320 } 321 322 (2) Setup static foveated rendering 323 324 // Insert code from #1 325 GLfloat focalX=0.f, focalY=0.f; // Setup focal point at the center of screen 326 GLfloat gainX=4.f, gainY=4.f; // Increase these for stronger foveation 327 glFramebufferFoveationParametersQCOM(fbo, 0, 0, focalX, focalY, gainX, gainY, 0.f); 328 329 (3) Change eye position for eye tracked foveated rendering 330 331 // Code called whenever the eye position changes 332 // It is best to position this call both before rendering anything to 333 // a fbo and right before Flush or changing FBO since some 334 // some implementations can apply this state late by patching command 335 // buffers. 336 glFramebufferFoveationParametersQCOM(fbo, 0, 0, focalX, focalY, gainX, gainY, 0.f); 337 338 (4) Setting parameters for a multiview stereo framebuffer 339 340 //focalPointsPerLayer should be 1 341 float focalX1=0.f,focalY1=0.f; // Gaze of left eye 342 float focalX2=0.f,focalY2=0.f; // Gaze of right eye 343 float gain_x=10.f,gain_y=10.f; // Strong foveation 344 glFramebufferFoveationParametersQCOM(fbo, 0, 0, focalX1, focalY1,gainX, gainY, 0.f); 345 glFramebufferFoveationParametersQCOM(fbo, 1, 0, focalX2, focalY2,gainX, gainY, 0.f); 346 347 (5) Setting parameters for a double wide stereo framebuffer 348 349 //focalPointsPerLayer should be 2 350 float focalX1=0.f,focalY1=0.f; // Gaze of left eye 351 float focalX2=0.f,focalY2=0.f; // Gaze of right eye 352 float gainX=10.f,gainY=10.f; 353 glFramebufferFoveationParametersQCOM(fbo, 0, 0, focalX1*0.5f-0.5f, focalY1, gainX*2.f ,gainY, 0.f); 354 glFramebufferFoveationParametersQCOM(fbo, 0, 1, focalX2*0.5f+0.5f, focalY2, gainX*2.f ,gainY, 0.f); 355 356 357Revision History 358 359 Rev. Date Author Changes 360 ---- -------- -------- ---------------------------------------------- 361 1 05/19/16 ssaleh Initial draft. 362 2 05/27/16 ssaleh Made the extension much more explicit. 363 3 07/08/16 ssaleh Further refinements. 364 4 08/11/16 ssaleh Specified bitfield values 365 5 08/19/16 ssaleh Added support for double wide rendering 366 6 08/24/16 mribble Name changes and cleanup 367 7 08/24/16 ssaleh Add examples 368 8 10/14/16 tateh Clarify gl_FragCoord 369 9 01/04/17 tateh Update entry points and cleanup 370 10 02/17/17 jleger Convert from EXT to QCOM extension. 371 11 05/10/17 tateh Minor cleanup 372