1\input texinfo @c -*- texinfo -*- 2@documentencoding UTF-8 3 4@settitle FFmpeg Automated Testing Environment 5@titlepage 6@center @titlefont{FFmpeg Automated Testing Environment} 7@end titlepage 8 9@node Top 10@top 11 12@contents 13 14@chapter Introduction 15 16FATE is an extended regression suite on the client-side and a means 17for results aggregation and presentation on the server-side. 18 19The first part of this document explains how you can use FATE from 20your FFmpeg source directory to test your ffmpeg binary. The second 21part describes how you can run FATE to submit the results to FFmpeg's 22FATE server. 23 24In any way you can have a look at the publicly viewable FATE results 25by visiting this website: 26 27@url{http://fate.ffmpeg.org/} 28 29This is especially recommended for all people contributing source 30code to FFmpeg, as it can be seen if some test on some platform broke 31with their recent contribution. This usually happens on the platforms 32the developers could not test on. 33 34The second part of this document describes how you can run FATE to 35submit your results to FFmpeg's FATE server. If you want to submit your 36results be sure to check that your combination of CPU, OS and compiler 37is not already listed on the above mentioned website. 38 39In the third part you can find a comprehensive listing of FATE makefile 40targets and variables. 41 42 43@chapter Using FATE from your FFmpeg source directory 44 45If you want to run FATE on your machine you need to have the samples 46in place. You can get the samples via the build target fate-rsync. 47Use this command from the top-level source directory: 48 49@example 50make fate-rsync SAMPLES=fate-suite/ 51make fate SAMPLES=fate-suite/ 52@end example 53 54The above commands set the samples location by passing a makefile 55variable via command line. It is also possible to set the samples 56location at source configuration time by invoking configure with 57@option{--samples=<path to the samples directory>}. Afterwards you can 58invoke the makefile targets without setting the @var{SAMPLES} makefile 59variable. This is illustrated by the following commands: 60 61@example 62./configure --samples=fate-suite/ 63make fate-rsync 64make fate 65@end example 66 67Yet another way to tell FATE about the location of the sample 68directory is by making sure the environment variable FATE_SAMPLES 69contains the path to your samples directory. This can be achieved 70by e.g. putting that variable in your shell profile or by setting 71it in your interactive session. 72 73@example 74FATE_SAMPLES=fate-suite/ make fate 75@end example 76 77@float NOTE 78Do not put a '~' character in the samples path to indicate a home 79directory. Because of shell nuances, this will cause FATE to fail. 80@end float 81 82To use a custom wrapper to run the test, pass @option{--target-exec} to 83@command{configure} or set the @var{TARGET_EXEC} Make variable. 84 85 86@chapter Submitting the results to the FFmpeg result aggregation server 87 88To submit your results to the server you should run fate through the 89shell script @file{tests/fate.sh} from the FFmpeg sources. This script needs 90to be invoked with a configuration file as its first argument. 91 92@example 93tests/fate.sh /path/to/fate_config 94@end example 95 96A configuration file template with comments describing the individual 97configuration variables can be found at @file{doc/fate_config.sh.template}. 98 99@ifhtml 100The mentioned configuration template is also available here: 101@verbatiminclude fate_config.sh.template 102@end ifhtml 103 104Create a configuration that suits your needs, based on the configuration 105template. The @env{slot} configuration variable can be any string that is not 106yet used, but it is suggested that you name it adhering to the following 107pattern @samp{@var{arch}-@var{os}-@var{compiler}-@var{compiler version}}. The 108configuration file itself will be sourced in a shell script, therefore all 109shell features may be used. This enables you to setup the environment as you 110need it for your build. 111 112For your first test runs the @env{fate_recv} variable should be empty or 113commented out. This will run everything as normal except that it will omit 114the submission of the results to the server. The following files should be 115present in $workdir as specified in the configuration file: 116 117@itemize 118 @item configure.log 119 @item compile.log 120 @item test.log 121 @item report 122 @item version 123@end itemize 124 125When you have everything working properly you can create an SSH key pair 126and send the public key to the FATE server administrator who can be contacted 127at the email address @email{fate-admin@@ffmpeg.org}. 128 129Configure your SSH client to use public key authentication with that key 130when connecting to the FATE server. Also do not forget to check the identity 131of the server and to accept its host key. This can usually be achieved by 132running your SSH client manually and killing it after you accepted the key. 133The FATE server's fingerprint is: 134 135@table @samp 136@item RSA 137 d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51 138@item ECDSA 139 76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86 140@end table 141 142If you have problems connecting to the FATE server, it may help to try out 143the @command{ssh} command with one or more @option{-v} options. You should 144get detailed output concerning your SSH configuration and the authentication 145process. 146 147The only thing left is to automate the execution of the fate.sh script and 148the synchronisation of the samples directory. 149 150@chapter Uploading new samples to the fate suite 151 152If you need a sample uploaded send a mail to samples-request. 153 154This is for developers who have an account on the fate suite server. 155If you upload new samples, please make sure they are as small as possible, 156space on each client, network bandwidth and so on benefit from smaller test cases. 157Also keep in mind older checkouts use existing sample files, that means in 158practice generally do not replace, remove or overwrite files as it likely would 159break older checkouts or releases. 160Also all needed samples for a commit should be uploaded, ideally 24 161hours, before the push. 162If you need an account for frequently uploading samples or you wish to help 163others by doing that send a mail to ffmpeg-devel. 164 165@example 166#First update your local samples copy: 167rsync -vauL --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X fate-suite.ffmpeg.org:/home/samples/fate-suite/ ~/fate-suite 168 169#Then do a dry run checking what would be uploaded: 170rsync -vanL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite 171 172#Upload the files: 173rsync -vaL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite 174@end example 175 176 177@chapter FATE makefile targets and variables 178 179@section Makefile targets 180 181@table @option 182@item fate-rsync 183Download/synchronize sample files to the configured samples directory. 184 185@item fate-list 186Will list all fate/regression test targets. 187 188@item fate 189Run the FATE test suite (requires the fate-suite dataset). 190@end table 191 192@section Makefile variables 193 194@table @env 195@item V 196Verbosity level, can be set to 0, 1 or 2. 197 @itemize 198 @item 0: show just the test arguments 199 @item 1: show just the command used in the test 200 @item 2: show everything 201 @end itemize 202 203@item SAMPLES 204Specify or override the path to the FATE samples at make time, it has a 205meaning only while running the regression tests. 206 207@item THREADS 208Specify how many threads to use while running regression tests, it is 209quite useful to detect thread-related regressions. 210 211@item THREAD_TYPE 212Specify which threading strategy test, either @samp{slice} or @samp{frame}, 213by default @samp{slice+frame} 214 215@item CPUFLAGS 216Specify CPU flags. 217 218@item TARGET_EXEC 219Specify or override the wrapper used to run the tests. 220The @env{TARGET_EXEC} option provides a way to run FATE wrapped in 221@command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets 222through @command{ssh}. 223 224@item GEN 225Set to @samp{1} to generate the missing or mismatched references. 226 227@item HWACCEL 228Specify which hardware acceleration to use while running regression tests, 229by default @samp{none} is used. 230 231@item KEEP 232Set to @samp{1} to keep temp files generated by fate test(s) when test is successful. 233Default is @samp{0}, which removes these files. Files are always kept when a test 234fails. 235 236@end table 237 238@section Examples 239 240@example 241make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate 242@end example 243