1Continuous Integration 2====================== 3 4GitLab CI 5--------- 6 7GitLab provides a convenient framework for running commands in response to Git pushes. 8We use it to test merge requests (MRs) before merging them (pre-merge testing), 9as well as post-merge testing, for everything that hits ``main`` 10(this is necessary because we still allow commits to be pushed outside of MRs, 11and even then the MR CI runs in the forked repository, which might have been 12modified and thus is unreliable). 13 14The CI runs a number of tests, from trivial build-testing to complex GPU rendering: 15 16- Build testing for a number of build systems, configurations and platforms 17- Sanity checks (``meson test``) 18- Some drivers (softpipe, llvmpipe, freedreno and panfrost) are also tested 19 using `VK-GL-CTS <https://github.com/KhronosGroup/VK-GL-CTS>`__ 20- Replay of application traces 21 22A typical run takes between 20 and 30 minutes, although it can go up very quickly 23if the GitLab runners are overwhelmed, which happens sometimes. When it does happen, 24not much can be done besides waiting it out, or cancel it. 25 26Due to limited resources, we currently do not run the CI automatically 27on every push; instead, we only run it automatically once the MR has 28been assigned to ``Marge``, our merge bot. 29 30If you're interested in the details, the main configuration file is ``.gitlab-ci.yml``, 31and it references a number of other files in ``.gitlab-ci/``. 32 33If the GitLab CI doesn't seem to be running on your fork (or MRs, as they run 34in the context of your fork), you should check the "Settings" of your fork. 35Under "CI / CD" → "General pipelines", make sure "Custom CI config path" is 36empty (or set to the default ``.gitlab-ci.yml``), and that the 37"Public pipelines" box is checked. 38 39If you're having issues with the GitLab CI, your best bet is to ask 40about it on ``#freedesktop`` on OFTC and tag `Daniel Stone 41<https://gitlab.freedesktop.org/daniels>`__ (``daniels`` on IRC) or 42`Emma Anholt <https://gitlab.freedesktop.org/anholt>`__ (``anholt`` on 43IRC). 44 45The three GitLab CI systems currently integrated are: 46 47 48.. toctree:: 49 :maxdepth: 1 50 51 bare-metal 52 LAVA 53 docker 54 55Application traces replay 56------------------------- 57 58The CI replays application traces with various drivers in two different jobs. The first 59job replays traces listed in ``src/<driver>/ci/traces-<driver>.yml`` files and if any 60of those traces fail the pipeline fails as well. The second job replays traces listed in 61``src/<driver>/ci/restricted-traces-<driver>.yml`` and it is allowed to fail. This second 62job is only created when the pipeline is triggered by `marge-bot` or any other user that 63has been granted access to these traces. 64 65A traces YAML file also includes a ``download-url`` pointing to a MinIO 66instance where to download the traces from. While the first job should always work with 67publicly accessible traces, the second job could point to an url with restricted access. 68 69Restricted traces are those that have been made available to Mesa developers without a 70license to redistribute at will, and thus should not be exposed to the public. Failing to 71access that URL would not prevent the pipeline to pass, therefore forks made by 72contributors without permissions to download non-redistributable traces can be merged 73without friction. 74 75As an aside, only maintainers of such non-redistributable traces are responsible for 76ensuring that replays are successful, since other contributors would not be able to 77download and test them by themselves. 78 79Those Mesa contributors that believe they could have permission to access such 80non-redistributable traces can request permission to Daniel Stone <daniels@collabora.com>. 81 82gitlab.freedesktop.org accounts that are to be granted access to these traces will be 83added to the OPA policy for the MinIO repository as per 84https://gitlab.freedesktop.org/freedesktop/helm-gitlab-config/-/commit/a3cd632743019f68ac8a829267deb262d9670958 . 85 86So the jobs are created in personal repositories, the name of the user's account needs 87to be added to the rules attribute of the Gitlab CI job that accesses the restricted 88accounts. 89 90Intel CI 91-------- 92 93The Intel CI is not yet integrated into the GitLab CI. 94For now, special access must be manually given (file a issue in 95`the Intel CI configuration repo <https://gitlab.freedesktop.org/Mesa_CI/mesa_jenkins>`__ 96if you think you or Mesa would benefit from you having access to the Intel CI). 97Results can be seen on `mesa-ci.01.org <https://mesa-ci.01.org>`__ 98if you are *not* an Intel employee, but if you are you 99can access a better interface on 100`mesa-ci-results.jf.intel.com <http://mesa-ci-results.jf.intel.com>`__. 101 102The Intel CI runs a much larger array of tests, on a number of generations 103of Intel hardware and on multiple platforms (X11, Wayland, DRM & Android), 104with the purpose of detecting regressions. 105Tests include 106`Crucible <https://gitlab.freedesktop.org/mesa/crucible>`__, 107`VK-GL-CTS <https://github.com/KhronosGroup/VK-GL-CTS>`__, 108`dEQP <https://android.googlesource.com/platform/external/deqp>`__, 109`Piglit <https://gitlab.freedesktop.org/mesa/piglit>`__, 110`Skia <https://skia.googlesource.com/skia>`__, 111`VkRunner <https://github.com/Igalia/vkrunner>`__, 112`WebGL <https://github.com/KhronosGroup/WebGL>`__, 113and a few other tools. 114A typical run takes between 30 minutes and an hour. 115 116If you're having issues with the Intel CI, your best bet is to ask about 117it on ``#dri-devel`` on OFTC and tag `Nico Cortes 118<https://gitlab.freedesktop.org/ngcortes>`__ (``ngcortes`` on IRC). 119 120.. _CI-farm-expectations: 121 122CI farm expectations 123-------------------- 124 125To make sure that testing of one vendor's drivers doesn't block 126unrelated work by other vendors, we require that a given driver's test 127farm produces a spurious failure no more than once a week. If every 128driver had CI and failed once a week, we would be seeing someone's 129code getting blocked on a spurious failure daily, which is an 130unacceptable cost to the project. 131 132Additionally, the test farm needs to be able to provide a short enough 133turnaround time that we can get our MRs through marge-bot without the 134pipeline backing up. As a result, we require that the test farm be 135able to handle a whole pipeline's worth of jobs in less than 15 minutes 136(to compare, the build stage is about 10 minutes). 137 138If a test farm is short the HW to provide these guarantees, consider dropping 139tests to reduce runtime. dEQP job logs print the slowest tests at the end of 140the run, and piglit logs the runtime of tests in the results.json.bz2 in the 141artifacts. Or, you can add the following to your job to only run some fraction 142(in this case, 1/10th) of the deqp tests. 143 144.. code-block:: yaml 145 146 variables: 147 DEQP_FRACTION: 10 148 149to just run 1/10th of the test list. 150 151If a HW CI farm goes offline (network dies and all CI pipelines end up 152stalled) or its runners are consistently spuriously failing (disk 153full?), and the maintainer is not immediately available to fix the 154issue, please push through an MR disabling that farm's jobs by adding 155'.' to the front of the jobs names until the maintainer can bring 156things back up. If this happens, the farm maintainer should provide a 157report to mesa-dev@lists.freedesktop.org after the fact explaining 158what happened and what the mitigation plan is for that failure next 159time. 160 161Personal runners 162---------------- 163 164Mesa's CI is currently run primarily on packet.net's m1xlarge nodes 165(2.2Ghz Sandy Bridge), with each job getting 8 cores allocated. You 166can speed up your personal CI builds (and marge-bot merges) by using a 167faster personal machine as a runner. You can find the gitlab-runner 168package in Debian, or use GitLab's own builds. 169 170To do so, follow `GitLab's instructions 171<https://docs.gitlab.com/ce/ci/runners/#create-a-specific-runner>`__ to 172register your personal GitLab runner in your Mesa fork. Then, tell 173Mesa how many jobs it should serve (``concurrent=``) and how many 174cores those jobs should use (``FDO_CI_CONCURRENT=``) by editing these 175lines in ``/etc/gitlab-runner/config.toml``, for example:: 176 177 concurrent = 2 178 179 [[runners]] 180 environment = ["FDO_CI_CONCURRENT=16"] 181 182 183Docker caching 184-------------- 185 186The CI system uses Docker images extensively to cache 187infrequently-updated build content like the CTS. The `freedesktop.org 188CI templates 189<https://gitlab.freedesktop.org/freedesktop/ci-templates/>`_ help us 190manage the building of the images to reduce how frequently rebuilds 191happen, and trim down the images (stripping out manpages, cleaning the 192apt cache, and other such common pitfalls of building Docker images). 193 194When running a container job, the templates will look for an existing 195build of that image in the container registry under 196``MESA_IMAGE_TAG``. If it's found it will be reused, and if 197not, the associated `.gitlab-ci/containers/<jobname>.sh`` will be run 198to build it. So, when developing any change to container build 199scripts, you need to update the associated ``MESA_IMAGE_TAG`` to 200a new unique string. We recommend using the current date plus some 201string related to your branch (so that if you rebase on someone else's 202container update from the same day, you will get a Git conflict 203instead of silently reusing their container) 204 205When developing a given change to your Docker image, you would have to 206bump the tag on each ``git commit --amend`` to your development 207branch, which can get tedious. Instead, you can navigate to the 208`container registry 209<https://gitlab.freedesktop.org/mesa/mesa/container_registry>`_ for 210your repository and delete the tag to force a rebuild. When your code 211is eventually merged to main, a full image rebuild will occur again 212(forks inherit images from the main repo, but MRs don't propagate 213images from the fork into the main repo's registry). 214 215Building locally using CI docker images 216--------------------------------------- 217 218It can be frustrating to debug build failures on an environment you 219don't personally have. If you're experiencing this with the CI 220builds, you can use Docker to use their build environment locally. Go 221to your job log, and at the top you'll see a line like:: 222 223 Pulling docker image registry.freedesktop.org/anholt/mesa/debian/android_build:2020-09-11 224 225We'll use a volume mount to make our current Mesa tree be what the 226Docker container uses, so they'll share everything (their build will 227go in _build, according to ``meson-build.sh``). We're going to be 228using the image non-interactively so we use ``run --rm $IMAGE 229command`` instead of ``run -it $IMAGE bash`` (which you may also find 230useful for debug). Extract your build setup variables from 231.gitlab-ci.yml and run the CI meson build script: 232 233.. code-block:: console 234 235 IMAGE=registry.freedesktop.org/anholt/mesa/debian/android_build:2020-09-11 236 sudo docker pull $IMAGE 237 sudo docker run --rm -v `pwd`:/mesa -w /mesa $IMAGE env PKG_CONFIG_PATH=/usr/local/lib/aarch64-linux-android/pkgconfig/:/android-ndk-r21d/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/pkgconfig/ GALLIUM_DRIVERS=freedreno UNWIND=disabled EXTRA_OPTION="-D android-stub=true -D llvm=disabled" DRI_LOADERS="-D glx=disabled -D gbm=disabled -D egl=enabled -D platforms=android" CROSS=aarch64-linux-android ./.gitlab-ci/meson-build.sh 238 239All you have left over from the build is its output, and a _build 240directory. You can hack on mesa and iterate testing the build with: 241 242.. code-block:: console 243 244 sudo docker run --rm -v `pwd`:/mesa $IMAGE ninja -C /mesa/_build 245 246 247Conformance Tests 248----------------- 249 250Some conformance tests require a special treatment to be maintained on Gitlab CI. 251This section lists their documentation pages. 252 253.. toctree:: 254 :maxdepth: 1 255 256 skqp 257 258 259Updating Gitlab CI Linux Kernel 260------------------------------- 261 262Gitlab CI usually runs a bleeding-edge kernel. The following documentation has 263instructions on how to uprev Linux Kernel in the Gitlab Ci ecosystem. 264 265.. toctree:: 266 :maxdepth: 1 267 268 kernel 269