1Bare-metal CI 2============= 3 4The bare-metal scripts run on a system with gitlab-runner and Docker, 5connected to potentially multiple bare-metal boards that run tests of 6Mesa. Currently only "fastboot" and "ChromeOS Servo" devices are 7supported. 8 9In comparison with LAVA, this doesn't involve maintaining a separate 10web service with its own job scheduler and replicating jobs between the 11two. It also places more of the board support in Git, instead of 12web service configuration. On the other hand, the serial interactions 13and bootloader support are more primitive. 14 15Requirements (fastboot) 16----------------------- 17 18This testing requires power control of the DUTs by the gitlab-runner 19machine, since this is what we use to reset the system and get back to 20a pristine state at the start of testing. 21 22We require access to the console output from the gitlab-runner system, 23since that is how we get the final results back from the tests. You 24should probably have the console on a serial connection, so that you 25can see bootloader progress. 26 27The boards need to be able to have a kernel/initramfs supplied by the 28gitlab-runner system, since the initramfs is what contains the Mesa 29testing payload. 30 31The boards should have networking, so that we can extract the dEQP .xml 32results to artifacts on GitLab. 33 34Requirements (servo) 35-------------------- 36 37For servo-connected boards, we can use the EC connection for power 38control to reboot the board. However, loading a kernel is not as easy 39as fastboot, so we assume your bootloader can do TFTP, and that your 40gitlab-runner mounts the runner's tftp directory specific to the board 41at /tftp in the container. 42 43Since we're going the TFTP route, we also use NFS root. This avoids 44packing the rootfs and sending it to the board as a ramdisk, which 45means we can support larger rootfses (for piglit testing), at the cost 46of needing more storage on the runner. 47 48Telling the board about where its TFTP and NFS should come from is 49done using dnsmasq on the runner host. For example, this snippet in 50the dnsmasq.conf.d in the google farm, with the gitlab-runner host we 51call "servo":: 52 53 dhcp-host=1c:69:7a:0d:a3:d3,10.42.0.10,set:servo 54 55 # Fixed dhcp addresses for my sanity, and setting a tag for 56 # specializing other DHCP options 57 dhcp-host=a0:ce:c8:c8:d9:5d,10.42.0.11,set:cheza1 58 dhcp-host=a0:ce:c8:c8:d8:81,10.42.0.12,set:cheza2 59 60 # Specify the next server, watch out for the double ',,'. The 61 # filename didn't seem to get picked up by the bootloader, so we use 62 # tftp-unique-root and mount directories like 63 # /srv/tftp/10.42.0.11/jwerner/cheza as /tftp in the job containers. 64 tftp-unique-root 65 dhcp-boot=tag:cheza1,cheza1/vmlinuz,,10.42.0.10 66 dhcp-boot=tag:cheza2,cheza2/vmlinuz,,10.42.0.10 67 68 dhcp-option=tag:cheza1,option:root-path,/srv/nfs/cheza1 69 dhcp-option=tag:cheza2,option:root-path,/srv/nfs/cheza2 70 71Setup 72----- 73 74Each board will be registered in freedesktop.org GitLab. You'll want 75something like this to register a fastboot board: 76 77.. code-block:: console 78 79 sudo gitlab-runner register \ 80 --url https://gitlab.freedesktop.org \ 81 --registration-token $1 \ 82 --name MY_BOARD_NAME \ 83 --tag-list MY_BOARD_TAG \ 84 --executor docker \ 85 --docker-image "alpine:latest" \ 86 --docker-volumes "/dev:/dev" \ 87 --docker-network-mode "host" \ 88 --docker-privileged \ 89 --non-interactive 90 91For a servo board, you'll need to also volume mount the board's NFS 92root dir at /nfs and TFTP kernel directory at /tftp. 93 94The registration token has to come from a freedesktop.org GitLab admin 95going to https://gitlab.freedesktop.org/admin/runners 96 97The name scheme for Google's lab is google-freedreno-boardname-n, and 98our tag is something like google-freedreno-db410c. The tag is what 99identifies a board type so that board-specific jobs can be dispatched 100into that pool. 101 102We need privileged mode and the /dev bind mount in order to get at the 103serial console and fastboot USB devices (--device arguments don't 104apply to devices that show up after container start, which is the case 105with fastboot, and the servo serial devices are actually links to 106/dev/pts). We use host network mode so that we can spin up a nginx 107server to collect XML results for fastboot. 108 109Once you've added your boards, you're going to need to add a little 110more customization in ``/etc/gitlab-runner/config.toml``. First, add 111``concurrent = <number of boards>`` at the top ("we should have up to 112this many jobs running managed by this gitlab-runner"). Then for each 113board's runner, set ``limit = 1`` ("only 1 job served by this board at a 114time"). Finally, add the board-specific environment variables 115required by your bare-metal script, something like:: 116 117 [[runners]] 118 name = "google-freedreno-db410c-1" 119 environment = ["BM_SERIAL=/dev/ttyDB410c8", "BM_POWERUP=google-power-up.sh 8", "BM_FASTBOOT_SERIAL=15e9e390", "FDO_CI_CONCURRENT=4"] 120 121The ``FDO_CI_CONCURRENT`` variable should be set to the number of CPU threads on 122the board, which is used for auto-tuning of job parallelism. 123 124Once you've updated your runners' configs, restart with ``sudo service 125gitlab-runner restart`` 126 127Caching downloads 128----------------- 129 130To improve the runtime for downloading traces during traces job runs, you will 131want a pass-through HTTP cache. On your runner box, install nginx: 132 133.. code-block:: console 134 135 sudo apt install nginx libnginx-mod-http-lua 136 137Add the server setup files: 138 139.. literalinclude: fdo-cache: 140 :name: /etc/nginx/sites-available/fdo-cache 141 142.. literalinclude: uri-caching.conf: 143 :name: /etc/nginx/sites-available/snippets/uri-caching.conf 144 145Edit the listener addresses in fdo-cache to suit the ethernet interface that 146your devices are on. 147 148Enable the site and restart nginx: 149 150.. code-block:: console 151 152 sudo ln -s /etc/nginx/sites-available/fdo-cache /etc/nginx/sites-enabled/fdo-cache 153 sudo service nginx restart 154 155 # First download will hit the internet 156 wget http://localhost/cache/?uri=https://minio-packet.freedesktop.org/mesa-tracie-public/itoral-gl-terrain-demo/demo.trace 157 # Second download should be cached. 158 wget http://localhost/cache/?uri=https://minio-packet.freedesktop.org/mesa-tracie-public/itoral-gl-terrain-demo/demo.trace 159 160Now, set ``download-url`` in your ``traces-*.yml`` entry to something like 161``http://10.42.0.1:8888/cache/?uri=https://minio-packet.freedesktop.org/mesa-tracie-public`` 162and you should have cached downloads for traces. Add it to 163``FDO_HTTP_CACHE_URI=`` in your ``config.toml`` runner environment lines and you 164can use it for cached artifact downloads instead of going all the way to 165freedesktop.org on each job. 166