1\input texinfo @c -*- texinfo -*- 2@documentencoding UTF-8 3 4@settitle FFmpeg Automated Testing Environment 5@titlepage 6@center @titlefont{FFmpeg Automated Testing Environment} 7@end titlepage 8 9@node Top 10@top 11 12@contents 13 14@chapter Introduction 15 16FATE is an extended regression suite on the client-side and a means 17for results aggregation and presentation on the server-side. 18 19The first part of this document explains how you can use FATE from 20your FFmpeg source directory to test your ffmpeg binary. The second 21part describes how you can run FATE to submit the results to FFmpeg's 22FATE server. 23 24In any way you can have a look at the publicly viewable FATE results 25by visiting this website: 26 27@url{http://fate.ffmpeg.org/} 28 29This is especially recommended for all people contributing source 30code to FFmpeg, as it can be seen if some test on some platform broke 31with their recent contribution. This usually happens on the platforms 32the developers could not test on. 33 34The second part of this document describes how you can run FATE to 35submit your results to FFmpeg's FATE server. If you want to submit your 36results be sure to check that your combination of CPU, OS and compiler 37is not already listed on the above mentioned website. 38 39In the third part you can find a comprehensive listing of FATE makefile 40targets and variables. 41 42 43@chapter Using FATE from your FFmpeg source directory 44 45If you want to run FATE on your machine you need to have the samples 46in place. You can get the samples via the build target fate-rsync. 47Use this command from the top-level source directory: 48 49@example 50make fate-rsync SAMPLES=fate-suite/ 51make fate SAMPLES=fate-suite/ 52@end example 53 54The above commands set the samples location by passing a makefile 55variable via command line. It is also possible to set the samples 56location at source configuration time by invoking configure with 57@option{--samples=<path to the samples directory>}. Afterwards you can 58invoke the makefile targets without setting the @var{SAMPLES} makefile 59variable. This is illustrated by the following commands: 60 61@example 62./configure --samples=fate-suite/ 63make fate-rsync 64make fate 65@end example 66 67Yet another way to tell FATE about the location of the sample 68directory is by making sure the environment variable FATE_SAMPLES 69contains the path to your samples directory. This can be achieved 70by e.g. putting that variable in your shell profile or by setting 71it in your interactive session. 72 73@example 74FATE_SAMPLES=fate-suite/ make fate 75@end example 76 77@float NOTE 78Do not put a '~' character in the samples path to indicate a home 79directory. Because of shell nuances, this will cause FATE to fail. 80@end float 81 82To get the complete list of tests, run the command: 83@example 84make fate-list 85@end example 86 87You can specify a subset of tests to run by specifying the 88corresponding elements from the list with the @code{fate-} prefix, 89e.g. as in: 90@example 91make fate-ffprobe_compact fate-ffprobe_xml 92@end example 93 94This makes it easier to run a few tests in case of failure without 95running the complete test suite. 96 97To use a custom wrapper to run the test, pass @option{--target-exec} to 98@command{configure} or set the @var{TARGET_EXEC} Make variable. 99 100 101@chapter Submitting the results to the FFmpeg result aggregation server 102 103To submit your results to the server you should run fate through the 104shell script @file{tests/fate.sh} from the FFmpeg sources. This script needs 105to be invoked with a configuration file as its first argument. 106 107@example 108tests/fate.sh /path/to/fate_config 109@end example 110 111A configuration file template with comments describing the individual 112configuration variables can be found at @file{doc/fate_config.sh.template}. 113 114@ifhtml 115The mentioned configuration template is also available here: 116@verbatiminclude fate_config.sh.template 117@end ifhtml 118 119Create a configuration that suits your needs, based on the configuration 120template. The @env{slot} configuration variable can be any string that is not 121yet used, but it is suggested that you name it adhering to the following 122pattern @samp{@var{arch}-@var{os}-@var{compiler}-@var{compiler version}}. The 123configuration file itself will be sourced in a shell script, therefore all 124shell features may be used. This enables you to setup the environment as you 125need it for your build. 126 127For your first test runs the @env{fate_recv} variable should be empty or 128commented out. This will run everything as normal except that it will omit 129the submission of the results to the server. The following files should be 130present in $workdir as specified in the configuration file: 131 132@itemize 133 @item configure.log 134 @item compile.log 135 @item test.log 136 @item report 137 @item version 138@end itemize 139 140When you have everything working properly you can create an SSH key pair 141and send the public key to the FATE server administrator who can be contacted 142at the email address @email{fate-admin@@ffmpeg.org}. 143 144Configure your SSH client to use public key authentication with that key 145when connecting to the FATE server. Also do not forget to check the identity 146of the server and to accept its host key. This can usually be achieved by 147running your SSH client manually and killing it after you accepted the key. 148The FATE server's fingerprint is: 149 150@table @samp 151@item RSA 152 d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51 153@item ECDSA 154 76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86 155@end table 156 157If you have problems connecting to the FATE server, it may help to try out 158the @command{ssh} command with one or more @option{-v} options. You should 159get detailed output concerning your SSH configuration and the authentication 160process. 161 162The only thing left is to automate the execution of the fate.sh script and 163the synchronisation of the samples directory. 164 165@chapter Uploading new samples to the fate suite 166 167If you need a sample uploaded send a mail to samples-request. 168 169This is for developers who have an account on the fate suite server. 170If you upload new samples, please make sure they are as small as possible, 171space on each client, network bandwidth and so on benefit from smaller test cases. 172Also keep in mind older checkouts use existing sample files, that means in 173practice generally do not replace, remove or overwrite files as it likely would 174break older checkouts or releases. 175Also all needed samples for a commit should be uploaded, ideally 24 176hours, before the push. 177If you need an account for frequently uploading samples or you wish to help 178others by doing that send a mail to ffmpeg-devel. 179 180@example 181#First update your local samples copy: 182rsync -vauL --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X fate-suite.ffmpeg.org:/home/samples/fate-suite/ ~/fate-suite 183 184#Then do a dry run checking what would be uploaded: 185rsync -vanL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite 186 187#Upload the files: 188rsync -vaL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite 189@end example 190 191 192@chapter FATE makefile targets and variables 193 194@section Makefile targets 195 196@table @option 197@item fate-rsync 198Download/synchronize sample files to the configured samples directory. 199 200@item fate-list 201Will list all fate/regression test targets. 202 203@item fate 204Run the FATE test suite (requires the fate-suite dataset). 205@end table 206 207@section Makefile variables 208 209@table @env 210@item V 211Verbosity level, can be set to 0, 1 or 2. 212 @itemize 213 @item 0: show just the test arguments 214 @item 1: show just the command used in the test 215 @item 2: show everything 216 @end itemize 217 218@item SAMPLES 219Specify or override the path to the FATE samples at make time, it has a 220meaning only while running the regression tests. 221 222@item THREADS 223Specify how many threads to use while running regression tests, it is 224quite useful to detect thread-related regressions. 225 226@item THREAD_TYPE 227Specify which threading strategy test, either @samp{slice} or @samp{frame}, 228by default @samp{slice+frame} 229 230@item CPUFLAGS 231Specify CPU flags. 232 233@item TARGET_EXEC 234Specify or override the wrapper used to run the tests. 235The @env{TARGET_EXEC} option provides a way to run FATE wrapped in 236@command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets 237through @command{ssh}. 238 239@item GEN 240Set to @samp{1} to generate the missing or mismatched references. 241 242@item HWACCEL 243Specify which hardware acceleration to use while running regression tests, 244by default @samp{none} is used. 245 246@item KEEP 247Set to @samp{1} to keep temp files generated by fate test(s) when test is successful. 248Default is @samp{0}, which removes these files. Files are always kept when a test 249fails. 250 251@end table 252 253@section Examples 254 255@example 256make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate 257@end example 258