1============================= 2How To Validate a New Release 3============================= 4 5.. contents:: 6 :local: 7 :depth: 1 8 9Introduction 10============ 11 12This document contains information about testing the release candidates that 13will ultimately be the next LLVM release. For more information on how to 14manage the actual release, please refer to :doc:`HowToReleaseLLVM`. 15 16Overview of the Release Process 17------------------------------- 18 19Once the release process starts, the Release Manager will ask for volunteers, 20and it'll be the role of each volunteer to: 21 22* Test and benchmark the previous release 23 24* Test and benchmark each release candidate, comparing to the previous release 25 and candidates 26 27* Identify, reduce and report every regression found during tests and benchmarks 28 29* Make sure the critical bugs get fixed and merged to the next release candidate 30 31Not all bugs or regressions are show-stoppers and it's a bit of a grey area what 32should be fixed before the next candidate and what can wait until the next 33release. 34 35It'll depend on: 36 37* The severity of the bug, how many people it affects and if it's a regression 38 or a known bug. Known bugs are "unsupported features" and some bugs can be 39 disabled if they have been implemented recently. 40 41* The stage in the release. Less critical bugs should be considered to be 42 fixed between RC1 and RC2, but not so much at the end of it. 43 44* If it's a correctness or a performance regression. Performance regression 45 tends to be taken more lightly than correctness. 46 47.. _scripts: 48 49Scripts 50======= 51 52The scripts are in the ``utils/release`` directory. 53 54test-release.sh 55--------------- 56 57This script will check-out, configure and compile LLVM+Clang (+ most add-ons, 58like ``compiler-rt``, ``libcxx``, ``libomp`` and ``clang-extra-tools``) in 59three stages, and will test the final stage. 60It'll have installed the final binaries on the Phase3/Releasei(+Asserts) 61directory, and that's the one you should use for the test-suite and other 62external tests. 63 64To run the script on a specific release candidate run:: 65 66 ./test-release.sh \ 67 -release 3.3 \ 68 -rc 1 \ 69 -no-64bit \ 70 -test-asserts \ 71 -no-compare-files 72 73Each system will require different options. For instance, x86_64 will 74obviously not need ``-no-64bit`` while 32-bit systems will, or the script will 75fail. 76 77The important flags to get right are: 78 79* On the pre-release, you should change ``-rc 1`` to ``-final``. On RC2, 80 change it to ``-rc 2`` and so on. 81 82* On non-release testing, you can use ``-final`` in conjunction with 83 ``-no-checkout``, but you'll have to create the ``final`` directory by hand 84 and link the correct source dir to ``final/llvm.src``. 85 86* For release candidates, you need ``-test-asserts``, or it won't create a 87 "Release+Asserts" directory, which is needed for release testing and 88 benchmarking. This will take twice as long. 89 90* On the final candidate you just need Release builds, and that's the binary 91 directory you'll have to pack. 92 93This script builds three phases of Clang+LLVM twice each (Release and 94Release+Asserts), so use screen or nohup to avoid headaches, since it'll take 95a long time. 96 97Use the ``--help`` option to see all the options and chose it according to 98your needs. 99 100 101findRegressions-nightly.py 102-------------------------- 103 104TODO 105 106.. _test-suite: 107 108Test Suite 109========== 110 111.. contents:: 112 :local: 113 114Follow the `LNT Quick Start Guide 115<http://llvm.org/docs/lnt/quickstart.html>`__ link on how to set-up the 116test-suite 117 118The binary location you'll have to use for testing is inside the 119``rcN/Phase3/Release+Asserts/llvmCore-REL-RC.install``. 120Link that directory to an easier location and run the test-suite. 121 122An example on the run command line, assuming you created a link from the correct 123install directory to ``~/devel/llvm/install``:: 124 125 ./sandbox/bin/python sandbox/bin/lnt runtest \ 126 nt \ 127 -j4 \ 128 --sandbox sandbox \ 129 --test-suite ~/devel/llvm/test/test-suite \ 130 --cc ~/devel/llvm/install/bin/clang \ 131 --cxx ~/devel/llvm/install/bin/clang++ 132 133It should have no new regressions, compared to the previous release or release 134candidate. You don't need to fix all the bugs in the test-suite, since they're 135not necessarily meant to pass on all architectures all the time. This is 136due to the nature of the result checking, which relies on direct comparison, 137and most of the time, the failures are related to bad output checking, rather 138than bad code generation. 139 140If the errors are in LLVM itself, please report every single regression found 141as blocker, and all the other bugs as important, but not necessarily blocking 142the release to proceed. They can be set as "known failures" and to be 143fix on a future date. 144 145.. _pre-release-process: 146 147Pre-Release Process 148=================== 149 150.. contents:: 151 :local: 152 153When the release process is announced on the mailing list, you should prepare 154for the testing, by applying the same testing you'll do on the release 155candidates, on the previous release. 156 157You should: 158 159* Download the previous release sources from 160 http://llvm.org/releases/download.html. 161 162* Run the test-release.sh script on ``final`` mode (change ``-rc 1`` to 163 ``-final``). 164 165* Once all three stages are done, it'll test the final stage. 166 167* Using the ``Phase3/Release+Asserts/llvmCore-MAJ.MIN-final.install`` base, 168 run the test-suite. 169 170If the final phase's ``make check-all`` failed, it's a good idea to also test 171the intermediate stages by going on the obj directory and running 172``make check-all`` to find if there's at least one stage that passes (helps 173when reducing the error for bug report purposes). 174 175.. _release-process: 176 177Release Process 178=============== 179 180.. contents:: 181 :local: 182 183When the Release Manager sends you the release candidate, download all sources, 184unzip on the same directory (there will be sym-links from the appropriate places 185to them), and run the release test as above. 186 187You should: 188 189* Download the current candidate sources from where the release manager points 190 you (ex. http://llvm.org/pre-releases/3.3/rc1/). 191 192* Repeat the steps above with ``-rc 1``, ``-rc 2`` etc modes and run the 193 test-suite the same way. 194 195* Compare the results, report all errors on Bugzilla and publish the binary blob 196 where the release manager can grab it. 197 198Once the release manages announces that the latest candidate is the good one, 199you have to pack the ``Release`` (no Asserts) install directory on ``Phase3`` 200and that will be the official binary. 201 202* Rename (or link) ``clang+llvm-REL-ARCH-ENV`` to the .install directory 203 204* Tar that into the same name with ``.tar.gz`` extensioan from outside the 205 directory 206 207* Make it available for the release manager to download 208 209.. _bug-reporting: 210 211Bug Reporting Process 212===================== 213 214.. contents:: 215 :local: 216 217If you found regressions or failures when comparing a release candidate with the 218previous release, follow the rules below: 219 220* Critical bugs on compilation should be fixed as soon as possible, possibly 221 before releasing the binary blobs. 222 223* Check-all tests should be fixed before the next release candidate, but can 224 wait until the test-suite run is finished. 225 226* Bugs in the test suite or unimportant check-all tests can be fixed in between 227 release candidates. 228 229* New features or recent big changes, when close to the release, should have 230 done in a way that it's easy to disable. If they misbehave, prefer disabling 231 them than releasing an unstable (but untested) binary package. 232