1Filter design 2============= 3 4This document explains guidelines that should be observed (or ignored with 5good reason) when writing filters for libavfilter. 6 7In this document, the word “frame” indicates either a video frame or a group 8of audio samples, as stored in an AVFrame structure. 9 10 11Format negotiation 12================== 13 14 The query_formats method should set, for each input and each output links, 15 the list of supported formats. 16 17 For video links, that means pixel format. For audio links, that means 18 channel layout, sample format (the sample packing is implied by the sample 19 format) and sample rate. 20 21 The lists are not just lists, they are references to shared objects. When 22 the negotiation mechanism computes the intersection of the formats 23 supported at each end of a link, all references to both lists are replaced 24 with a reference to the intersection. And when a single format is 25 eventually chosen for a link amongst the remaining list, again, all 26 references to the list are updated. 27 28 That means that if a filter requires that its input and output have the 29 same format amongst a supported list, all it has to do is use a reference 30 to the same list of formats. 31 32 query_formats can leave some formats unset and return AVERROR(EAGAIN) to 33 cause the negotiation mechanism to try again later. That can be used by 34 filters with complex requirements to use the format negotiated on one link 35 to set the formats supported on another. 36 37 38Frame references ownership and permissions 39========================================== 40 41 Principle 42 --------- 43 44 Audio and video data are voluminous; the frame and frame reference 45 mechanism is intended to avoid, as much as possible, expensive copies of 46 that data while still allowing the filters to produce correct results. 47 48 The data is stored in buffers represented by AVFrame structures. 49 Several references can point to the same frame buffer; the buffer is 50 automatically deallocated once all corresponding references have been 51 destroyed. 52 53 The characteristics of the data (resolution, sample rate, etc.) are 54 stored in the reference; different references for the same buffer can 55 show different characteristics. In particular, a video reference can 56 point to only a part of a video buffer. 57 58 A reference is usually obtained as input to the filter_frame method or 59 requested using the ff_get_video_buffer or ff_get_audio_buffer 60 functions. A new reference on an existing buffer can be created with 61 av_frame_ref(). A reference is destroyed using 62 the av_frame_free() function. 63 64 Reference ownership 65 ------------------- 66 67 At any time, a reference “belongs” to a particular piece of code, 68 usually a filter. With a few caveats that will be explained below, only 69 that piece of code is allowed to access it. It is also responsible for 70 destroying it, although this is sometimes done automatically (see the 71 section on link reference fields). 72 73 Here are the (fairly obvious) rules for reference ownership: 74 75 * A reference received by the filter_frame method belongs to the 76 corresponding filter. 77 78 * A reference passed to ff_filter_frame is given away and must no longer 79 be used. 80 81 * A reference created with av_frame_ref() belongs to the code that 82 created it. 83 84 * A reference obtained with ff_get_video_buffer or ff_get_audio_buffer 85 belongs to the code that requested it. 86 87 * A reference given as return value by the get_video_buffer or 88 get_audio_buffer method is given away and must no longer be used. 89 90 Link reference fields 91 --------------------- 92 93 The AVFilterLink structure has a few AVFrame fields. 94 95 partial_buf is used by libavfilter internally and must not be accessed 96 by filters. 97 98 fifo contains frames queued in the filter's input. They belong to the 99 framework until they are taken by the filter. 100 101 Reference permissions 102 --------------------- 103 104 Since the same frame data can be shared by several frames, modifying may 105 have unintended consequences. A frame is considered writable if only one 106 reference to it exists. The code owning that reference it then allowed 107 to modify the data. 108 109 A filter can check if a frame is writable by using the 110 av_frame_is_writable() function. 111 112 A filter can ensure that a frame is writable at some point of the code 113 by using the ff_inlink_make_frame_writable() function. It will duplicate 114 the frame if needed. 115 116 A filter can ensure that the frame passed to the filter_frame() callback 117 is writable by setting the needs_writable flag on the corresponding 118 input pad. It does not apply to the activate() callback. 119 120 121Frame scheduling 122================ 123 124 The purpose of these rules is to ensure that frames flow in the filter 125 graph without getting stuck and accumulating somewhere. 126 127 Simple filters that output one frame for each input frame should not have 128 to worry about it. 129 130 There are two design for filters: one using the filter_frame() and 131 request_frame() callbacks and the other using the activate() callback. 132 133 The design using filter_frame() and request_frame() is legacy, but it is 134 suitable for filters that have a single input and process one frame at a 135 time. New filters with several inputs, that treat several frames at a time 136 or that require a special treatment at EOF should probably use the design 137 using activate(). 138 139 activate 140 -------- 141 142 This method is called when something must be done in a filter; the 143 definition of that "something" depends on the semantic of the filter. 144 145 The callback must examine the status of the filter's links and proceed 146 accordingly. 147 148 The status of output links is stored in the frame_wanted_out, status_in 149 and status_out fields and tested by the ff_outlink_frame_wanted() 150 function. If this function returns true, then the processing requires a 151 frame on this link and the filter is expected to make efforts in that 152 direction. 153 154 The status of input links is stored by the status_in, fifo and 155 status_out fields; they must not be accessed directly. The fifo field 156 contains the frames that are queued in the input for processing by the 157 filter. The status_in and status_out fields contains the queued status 158 (EOF or error) of the link; status_in is a status change that must be 159 taken into account after all frames in fifo have been processed; 160 status_out is the status that have been taken into account, it is final 161 when it is not 0. 162 163 The typical task of an activate callback is to first check the backward 164 status of output links, and if relevant forward it to the corresponding 165 input. Then, if relevant, for each input link: test the availability of 166 frames in fifo and process them; if no frame is available, test and 167 acknowledge a change of status using ff_inlink_acknowledge_status(); and 168 forward the result (frame or status change) to the corresponding input. 169 If nothing is possible, test the status of outputs and forward it to the 170 corresponding input(s). If still not possible, return FFERROR_NOT_READY. 171 172 If the filters stores internally one or a few frame for some input, it 173 can consider them to be part of the FIFO and delay acknowledging a 174 status change accordingly. 175 176 Example code: 177 178 ret = ff_outlink_get_status(outlink); 179 if (ret) { 180 ff_inlink_set_status(inlink, ret); 181 return 0; 182 } 183 if (priv->next_frame) { 184 /* use it */ 185 return 0; 186 } 187 ret = ff_inlink_consume_frame(inlink, &frame); 188 if (ret < 0) 189 return ret; 190 if (ret) { 191 /* use it */ 192 return 0; 193 } 194 ret = ff_inlink_acknowledge_status(inlink, &status, &pts); 195 if (ret) { 196 /* flush */ 197 ff_outlink_set_status(outlink, status, pts); 198 return 0; 199 } 200 if (ff_outlink_frame_wanted(outlink)) { 201 ff_inlink_request_frame(inlink); 202 return 0; 203 } 204 return FFERROR_NOT_READY; 205 206 The exact code depends on how similar the /* use it */ blocks are and 207 how related they are to the /* flush */ block, and needs to apply these 208 operations to the correct inlink or outlink if there are several. 209 210 Macros are available to factor that when no extra processing is needed: 211 212 FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); 213 FF_FILTER_FORWARD_STATUS_ALL(outlink, filter); 214 FF_FILTER_FORWARD_STATUS(inlink, outlink); 215 FF_FILTER_FORWARD_STATUS_ALL(inlink, filter); 216 FF_FILTER_FORWARD_WANTED(outlink, inlink); 217 218 filter_frame 219 ------------ 220 221 For filters that do not use the activate() callback, this method is 222 called when a frame is pushed to the filter's input. It can be called at 223 any time except in a reentrant way. 224 225 If the input frame is enough to produce output, then the filter should 226 push the output frames on the output link immediately. 227 228 As an exception to the previous rule, if the input frame is enough to 229 produce several output frames, then the filter needs output only at 230 least one per link. The additional frames can be left buffered in the 231 filter; these buffered frames must be flushed immediately if a new input 232 produces new output. 233 234 (Example: frame rate-doubling filter: filter_frame must (1) flush the 235 second copy of the previous frame, if it is still there, (2) push the 236 first copy of the incoming frame, (3) keep the second copy for later.) 237 238 If the input frame is not enough to produce output, the filter must not 239 call request_frame to get more. It must just process the frame or queue 240 it. The task of requesting more frames is left to the filter's 241 request_frame method or the application. 242 243 If a filter has several inputs, the filter must be ready for frames 244 arriving randomly on any input. Therefore, any filter with several inputs 245 will most likely require some kind of queuing mechanism. It is perfectly 246 acceptable to have a limited queue and to drop frames when the inputs 247 are too unbalanced. 248 249 request_frame 250 ------------- 251 252 For filters that do not use the activate() callback, this method is 253 called when a frame is wanted on an output. 254 255 For a source, it should directly call filter_frame on the corresponding 256 output. 257 258 For a filter, if there are queued frames already ready, one of these 259 frames should be pushed. If not, the filter should request a frame on 260 one of its inputs, repeatedly until at least one frame has been pushed. 261 262 Return values: 263 if request_frame could produce a frame, or at least make progress 264 towards producing a frame, it should return 0; 265 if it could not for temporary reasons, it should return AVERROR(EAGAIN); 266 if it could not because there are no more frames, it should return 267 AVERROR_EOF. 268 269 The typical implementation of request_frame for a filter with several 270 inputs will look like that: 271 272 if (frames_queued) { 273 push_one_frame(); 274 return 0; 275 } 276 input = input_where_a_frame_is_most_needed(); 277 ret = ff_request_frame(input); 278 if (ret == AVERROR_EOF) { 279 process_eof_on_input(); 280 } else if (ret < 0) { 281 return ret; 282 } 283 return 0; 284 285 Note that, except for filters that can have queued frames and sources, 286 request_frame does not push frames: it requests them to its input, and 287 as a reaction, the filter_frame method possibly will be called and do 288 the work. 289