1Caps Negotiation 2================ 3 4 5Definitions 6----------- 7 8- GstCaps is a structure that holds a mimetype and a set of properties. The properties are 9 either fixed or non-fixed, they contain ranges or lists. 10 11- filter caps: caps set by the application to constrain a link to a specific GstCaps. 12 13- allowed caps: caps calculated when connecting elements, these are the types that are 14 possible on this connection. If no types are possible, the link is refused. 15 16- pad template caps: caps put on padtemplates to describe the possible media types of 17 the pad. 18 19- pad caps: the caps of the pad as it is currently negotiated. 20 21 22General overview 23---------------- 24 25Caps negotiation is the process of two pads agreeing on a common fixed GstCaps structure 26that describes the media type that will be exchanged between those pads. 27 28We define that caps negotiation only happens in the PLAYING state, when there is actual 29dataflow and elements thus know what kind of data they are dealing with, or when 30linking elements. 31 32Caps negotiation is first performed when linking elements. If the elements can agree 33on a media type at link time then negotiation will not have to be performed at runtime. 34Usually this is not possible, so the pads remain unnegotiated until at runtime. 35 36Caps negotiation is initiated by an element that wants to send data on a non-negotiated 37pad, thus the source pad. The core only provides a policy and convenience methods to aid 38the element in performing the negotiation. The core does not keep track of what pads 39are negotiated or not, this is a task of the pads. 40 41Caps negotiation is normally started by the source element right before it sends out 42data (case 1). 43Caps negotiation can also be redone by a sink element that wants to receive another 44format from a downstream element (case 2). 45 46There are two functions handling the negotiation, the link function and the do_link 47function. The link function is always called first and must eventually call do_link on 48the peer pad. The link function must figure out a compatible format for the connection 49and then call the do_link function on the peer pad. The do_link function can only 50accept or refuse the provided caps. 51 52For autopluggers it is important to know when the pad is ready to start the negotiation. 53It is also important to know when the negotiation failed and it must be possible to 54restart the negotiation with another element. This functionality will be provided 55with signals. 56 57Pad functions 58------------- 59 60! 61! const GstCaps* gst_pad_iterate_caps (GstPad *pad, gint position); 62! 63Returns the nth possible caps that describes the media type that can flow over 64this pad. This function should not return static caps but caps that are 65dependent of the media type and the peer connections of the element. 66 67This function can be called at any time so that an autoplugger can know the 68exact types of the pads at any time. 69 70! 71! gboolean gst_pad_accept_caps (GstPad *pad, GstCaps *caps); 72! 73Checks that a given caps is acceptable for this pad. Returns FALSE if the 74pad cannot handle the caps. 75 76! 77! gboolean gst_pad_link (GstPad *pad, GstPad *peer, GstCaps *caps); 78! 79Tells the pad to start negotiation with the given filtercaps. The 80caps need not be fixed and serves as a filter for performing the negotiation 81with the peerpad. 82 83! 84! gboolean gst_pad_do_link (GstPad *pad, GstPad *peer, GstCaps *caps); 85! 86Configures the pad to accept data with the given caps. The caps must be fixed. 87 88! 89! const GstCaps* gst_pad_get_negotiated_caps (GstPad *pad); 90! 91Get the negotiated caps of a pad or NULL if the pad is not negotiated. 92 93 94Linking Elements 95---------------- 96 97When linking elements with the gst_pad_link function, the core will call 98the link function on the srcpad. If that pad returns GST_PAD_LINK_OK, 99the link is established, else the link fails. 100 101Since the link function of the pad will call the do_link function of 102the peerpad, the caps will be set on both pads. 103 104It is not required to decide on a caps at link time, a plugin can choose to 105dynamically renegotiate at runtime. 106 107When a link fails, the core emits a signal link_failed, which the application 108can catch to try to establish a new link with another element, for example. 109 110! 111! def gst_pad_link_filtered (pad1, pad2, filtercaps): 112! 113! ... get srcpad and sinkpad ... 114! srcpad = (pad1 == GST_PAD_SRC ? pad1 : pad2); 115! sinkpad = (pad1 == GST_PAD_SINK ? pad1 : pad2); 116! 117! ... more checks to see if the pads are of different types and 118! ... that they live in the same scheduler etc... 119! 120! res = gst_pad_link (srcpad, sinkpad, filtercaps) 121! if (res == GST_PAD_LINK_OK) 122! ... perform other setup ... 123! else 124! res = signal_emit (srcpad, "link_failed") 125! 126! return res 127! 128 129Pad link is just a convenience function that passes a NULL filtercaps: 130! 131! def gst_pad_link (pad1, pad2): 132! gst_pad_link_filtered (pad1, pad2, NULL) 133! 134 135 136Dynamic renegotiation 137--------------------- 138 139Dynamic renegotiation happens at runtime when the element knows the exact media 140type it is handling. 141 142The element that wants to redecide on the data type of a connection just calls 143gst_pad_relink on one of it's pads. The core will call unlink and link on the 144pads again as if this were a new connection. Since the iterate_caps function 145returns other values while linking, the new link will renegotiate to a new 146format or the link will fail. 147 148Prototype of the relink function: 149! 150! def gst_pad_relink_filtered (pad, filtercaps): 151! gst_pad_unlink (pad, peerpad) 152! gst_pad_link_filtered (pad, peerpad, filtercaps) 153! 154 155Again the relink function is a convenience function for not having to pass 156filtercaps. 157! 158! def gst_pad_relink (pad): 159! gst_pad_relink_filtered (pad, NULL) 160! 161 162The core, however, should optimize the relink function so that it does a minimal 163amount of work, like not informing the scheduler about the unlink/link call in 164case of success. 165 166Error recovery 167-------------- 168 169When a pad is ready to negotiate and it has no peer pad, it fires the "need-link" 170signal. Autopluggers can use this signal to start plugging elements to the pad. 171 172When a link fails because of a renegotiation, the "link-failed" signal is fired 173so that autopluggers can try other elements. 174 175 176Default implementations 177----------------------- 178 179! 180! gst_pad_src_link_func (pad, peerpad, filtercaps) 181! { 182! srcpad->negotiated_caps = NULL; 183! gboolean possible = FALSE; 184! 185! for (i=0; peercaps = gst_pad_iterate_caps (peerpad, i); i++) 186! { 187! for (j=0; srccaps = gst_pad_iterate_caps (srcpad, j); j++) 188! { 189! test = gst_caps_intersect (srccaps, peercaps, filtercaps); 190! if (test == GST_CAPS_EMPTY) 191! continue; 192! 193! /* non empty caps, something is possible */ 194! possible = TRUE; 195! 196! if (!gst_caps_is_fixed (caps)) 197! continue; 198! 199! if (gst_pad_accepts_caps (peerpad, test)) 200! { 201! if (gst_pad_do_link (peerpad, srcpad, test)) 202! { 203! srcpad->negotiated_caps = test; 204! goto done; 205! } 206! else 207! { 208! /* weird, it accepted but did not want to link */ 209! } 210! } 211! else 212! { 213! /* caps are not accepted by peer, try next */ 214! } 215! } 216! } 217! done: 218! if (!possible) 219! return GST_PAD_LINK_FAILED; 220! else 221! return GST_PAD_LINK_OK; 222! } 223! 224 225gst_pad_iterate_caps returns caps in the following order: 226 227 1) preferred caps (if any) 228 2) negotiated caps (if any) 229 3) profile caps (if any, filtered against caps of current media type) 230 4) padtemplate caps (filtered against caps of current media type) 231 2321 = a caps describing the media type that would result in optimal 233 processing of the current media type 2342 = a caps that it is currently handling 2353 = a caps describing the user configured default values 2364 = a caps describing all other possibilities that this pad can 237 produce or consume given the current media type if any. 238 239 240generic flow 1: 241 242 Linkage 243 244 src sink 245 | | 246 | <--- link(sink, A) | 247check if | | 248A is ok | | 249or filter | | 250against | | 251allowed | | 252src caps | | 253 | iterate_caps | 254if A not +-------------------------> | 255fixed | | 256 | <-------------------------+ 257filter | | 258against A | | 259to get A' | | 260 | can_accept(A') | 261if A' fix +-------------------------> | 262 | | check if A' is ok 263 | yes | 264 | <-------------------------+ 265 | | 266if A' not | | 267fixed, | | 268A'=null | | 269 | do_link(src, A') | 270 +-------------------------> | 271 | ok | store src, A' 272 | <-------------------------+ 273store | | 274sink, A' | | 275 276 277 278Unlink 279 280 src sink 281 | | 282 | <--- unlink() | 283 | | 284 | unlink() | 285 +-------------------------> | 286unref | | unref src, format 287sink, fmt | | 288 289 290 Dynamic negotiation of B 291 292 src sink 293 | can_accept(B) | 294 +-------------------------> | 295 | | check if B is ok 296 | yes | 297 | <-------------------------+ 298 | | 299 | do_link(B) | 300 +-------------------------> | 301 | ok | store B 302 | <-------------------------+ 303store | | 304B | | 305 306 307 308TODO 309----- 310 311Write test objects to simulate different behaviour. 312 313 314tests (in python): 315 316 vts -> colorspace -> ximagsink 317 318 relink with different colorspaces while running 319 320 321 vts -> videoscale -> ximagesink 322 323 relink to different sizes while running, scaling vts or ximagesink. 324 325 sinesrc -> audiorate -> audiosink 326 327 where audiosink only has one possible fixed caps. 328 329