1INTRO 2----- 3This directory (nightly/) contains a simple, automatic build-and-test 4system for Valgrind, intended to be run nightly by cron or a similar 5program. 6 7 8BASIC OPERATIONS 9---------------- 10When run, the system checks out two trees: the SVN trunk from 24 hours ago 11and the SVN trunk from now. ("24 hours ago" and "now" are determined when 12the script starts running, so if any commits happen while the tests are 13running they will not be tested.) 14 15If the two trees are different (i.e. there have been commits in the past 24 16hours, either to the trunk or a branch) it builds ("make"), installs ("make 17install") and runs the regression tests ("make regtest") in both, and 18compares the results. Note that the "make install" isn't necessary in order 19to run the tests because the regression tests use the code built (with 20"make") within the tree, but it's worth doing because it tests that "make 21install" isn't totally broken. After checking both trees, it emails a 22summary of the results to a recipient. All this typically takes something 23like 30 minutes. 24 25If the two trees are identical, the tests are not run and no results are 26emailed. This avoids spamming people with uninteresting results emails when 27no commits have happened recently. 28 29 30SETTING UP 31---------- 32To set up nightly testing for a machine, do the following. 33 34(1) Check out just this directory from the repository, eg: 35 36 svn co svn://svn.valgrind.org/valgrind/trunk/nightly $DIR 37 38 where $DIR is the name of the directory you want it to be in. 39 40 Note that this doesn't check out the whole Valgrind tree, just the 41 directory containing the nightly testing stuff. This is possible 42 because the testing script doesn't check the code in the tree it belongs 43 to; rather it checks out new trees (within $DIR) and tests them 44 independently. 45 46(2) Choose a tag that identifies the test results. This is usually the 47 machine name. We'll call it $TAG in what follows. 48 49(3) Create a configuration file $DIR/conf/$TAG.conf. It is sourced by the 50 'nightly' script, and can define any or all of the following environment 51 variables. (In most cases, only ABT_DETAILS is needed.) 52 53 - ABT_DETAILS: describes the machine in more detail, eg. the OS. The 54 default is empty, but you should define it. An example: 55 56 export ABT_DETAILS="Ubuntu 9.04, Intel x86-64" 57 58 You could also use some invocation of 'uname' or something similar 59 to generate this string. Eg. on Ubuntu Linux this works nicely: 60 61 export ABT_DETAILS="`cat /etc/issue.net`, `uname -m`" 62 63 And on Mac OS X this works nicely: 64 65 export ABT_DETAILS=`uname -mrs` 66 67 The advantage of doing it like this is that if you update the OS on 68 the test machine you won't have to update ABT_DETAILS manually. 69 70 - ABT_CONFIGURE_OPTIONS: gives extra configure options. The default is 71 empty. 72 73 - ABT_EVAL: if provided, it must be the name of a shell script that 74 executes the shell command $1 with arguments $2 .. ${$#}. Allows to 75 compile and run the Valgrind regression tests on another system than 76 the system the 'nightly' script runs on. It is assumed that the remote 77 system shares the local filesystem tree through e.g. NFS. It is the 78 responsibility of the shell script to set the remote working directory 79 such that it matches the local current directory ($PWD). 80 81 - ABT_RUN_REGTEST: if provided, it must be the name of an argumentless 82 shell function (also specified in the $TAG.conf file) that will be used 83 to run the tests. If not specified, the usual "make regtest" will be 84 used. 85 86 - ABT_JOBS: allows parallel builds -- it's passed as the argument to 87 "make -j" when building Valgrind and the tests. The default is 1. 88 89 - ABT_PERF: unset or set to "" mean 'do not run perf tests' (default value) 90 set to "--vg=../valgrind-new" (run perf tests for new tree) 91 set to "--vg=../valgrind-new --vg=../valgrind-old" 92 (run perf tests for "new" and for "24 hours ago", 93 to compare the performances between the 2 trees). 94 95 - ABT_PERF_TOOLS: --tools=.... option of perf/vg_perf. 96 (default value: all non experimental tools) 97 98 - ABT_PERF_REPS: --reps=... option of perf/vg_perf 99 (default value: --reps=3) 100 101 Note that the appropriate syntax to use in this file will depend on the 102 shell from which the $DIR/bin/nightly script is run (which in turn may 103 depend on what shell is used by cron or any similar program). 104 105(4) Create a mailer script $DIR/conf/$TAG.sendmail. It must be executable. 106 It's used to send email results to the desired recipient (e.g. 107 valgrind-developers@lists.sourceforge.net) It must handle three command 108 line arguments. 109 110 - The first argument is the email subject line. It contains 111 $ABT_DETAILS plus some other stuff. 112 113 - The second argument is the name of the file containing the email's 114 body (which shows the tests that failed, and the differences between now 115 and 24 hours ago). 116 117 - The third is the name of the file containing all the diffs from 118 failing tests. Depending on the test results you get, you could 119 inline this file into the email body, or attach it, or compress and 120 attach it, or even omit it. The right choice depends on how many 121 failures you typically get -- if you get few failures, inlining the 122 results make them easier to read; if you get many failures, 123 compressing might be a good idea to minimise the size of the emails. 124 125 The best way to do this depends on how mail is set up on your machine. 126 You might be able to use /usr/bin/mail, or you might need something more 127 elaborate like using Mutt to send mail via an external account. 128 129 At first, you should probably just send emails to yourself for testing 130 purposes. After it's working, then sending it to others might be 131 appropriate. 132 133(5) To run the tests, execute: 134 135 $DIR/bin/nightly $DIR $TAG 136 137 You probably want to put this command into a cron file or equivalent 138 so it is run regularly (preferably every night). Actually, it's 139 probably better to put that command inside a script, and run the script 140 from cron, rather than running $DIR/bin/nightly directly. That way you 141 can put any other configuration stuff that's necessary inside the 142 script (e.g. make sure that programs used by the mailer script are in 143 your PATH). 144 145 146OUTPUT FILES 147------------ 148If the tests are run, the following files are produced: 149 150- $DIR/old.verbose and $DIR/new.verbose contain full output of the whole 151 process for each of the two trees. 152 153- $DIR/old.short and $DIR/new.short contain summary output of the process 154 for each of the two trees. The diff between these two files goes in 155 $DIR/diff.short. 156 157- $DIR/final contains the overall summary, constructed from $DIR/old.short, 158 $DIR/new.short, $DIR/diff.short and some other bits and pieces. (The name 159 of this file is what's passed as the second argument to 160 $DIR/conf/$TAG.sendmail.) 161 162- $DIR/diffs holds the diffs from all the failing tests in the newer tree, 163 concatenated together; the diff from each failure is truncated at 100 164 lines to minimise possible size blow-outs. (The name of this file is 165 what's passed as the third argument to $DIR/conf/$TAG.sendmail.) 166 167- $DIR/sendmail.log contains the output (stdout and stderr) from 168 $DIR/conf/$TAG.sendmail goes in $DIR/sendmail.log. 169 170- $DIR/valgrind-old/ and $DIR/valgrind-new/ contain the tested trees (and 171 $DIR/valgrind-old/Inst/ and $DIR/valgrind-new/Inst/ contain the installed 172 code). 173 174If the tests aren't run, the following file is produced: 175 176- $DIR/unchanged.log is created only if no tests were run because the two 177 trees were identical. It will contain a short explanatory message. 178 179Each time the tests are run, all files from previous runs are deleted. 180 181 182TROUBLESHOOTING 183--------------- 184If something goes wrong, looking at the output files can be useful. For 185example, if no email was sent but you expected one, check sendmail.log to 186see if the mailer script had a problem. Or check if unchanged.log exists. 187 188Occasionally the SVN server isn't available when the tests runs, for either 189or both trees. When this happens the email will be sent but it won't be 190very informative. Usually it's just a temporary server problem and it'll 191run fine the next time without you having to do anything. 192 193Note that the test suite is imperfect: 194- There are very few machines where all tests pass; that's why the old/new 195 diff is produced. Some of the tests may not be as portable as intended. 196- Some tests are non-deterministic, and so may pass one day and fail the 197 next. 198 199Improving the test suite to avoid these problems is a long-term goal but it 200isn't easy. 201 202 203MAINTENANCE 204----------- 205The scripts in the nightly/ directory occasionally get updated. If that 206happens, you can just "svn update" within $DIR to get the updated versions, 207which will then be used the next time the tests run. (It's possible that 208the scripts will be changed in a way that requires changes to the files in 209$DIR/conf/, but we try to avoid this.) 210 211If you want such updates to happen automatically, you could write a script 212that does all the steps in SETTING UP above, and instead run that script 213from cron. 214 215 216