Lines Matching +full:build +full:- +full:and +full:- +full:run +full:- +full:tests
12 Guides for getting to know the package build system we use.
17 ## Autotest and ebuild workflow
22 The rest of this document is going to use some terms and only explain them
27 At a high level, tests are organized in test cases, each test case being either
28 server or client, with one main .py file named the same as the test case, and
30 test, autotest expects tests to be placed in a monolithic file structure
33 - `/client/tests/`
34 - `/client/site_tests/`
35 - `/server/tests/`
36 - `/server/site_tests/`
41 directory, and separate projects that can be built with a Makefile inside the
49 - Having confidential (publicly inaccessible) tests or generally per-test ACLs
51 - Storing test cases along with the project they wrap around, because the test
52 requires binaries built as a by-product of the project’s own build system.
53 (e.g. chrome or tpm tests)
55 Furthermore, it has been desired to generally build everything that is not
56 strongly ordered in parallel, significantly decreasing build times. That,
57 however, requires proper dependency tree declaration and being able to specify
64 depend on build parameters.
66 
69 unlike what upstream does, the tests aren’t run directly from the source
70 repository, rather from a staging read-only install location. This leads to
73 - Source may live in an arbitrary location or can be generated on the fly.
75 (cros-workon may be utilised, introducing a fairly standard Chromium OS
77 - The staging location (`/build/${board}/usr/local/autotest/`) may not be
80 - Propagating source changes requires an emerge step.
85 …tion is available at http://www.chromium.org/chromium-os/how-tos-and-troubleshooting/portage-build…
87 An **autotest ebuild** is an ebuild that produces test cases and installs them into
90 - Obtain the source - This is generally (but not necessarily) provided by
91 ‘cros-workon’ eclass. It could also work with the more standard tarball
93 - Prepare test cases - This includes, but is not limited to preprocessing any
97 - Call autotest to "build" all sources and subsequently install them - This
102 number of variables specified and works by itself otherwise. Most variables
103 describe the locations and listings of work that needs to be done:
105 - Location variables define the paths to directories containing the test
108 - `AUTOTEST_{CLIENT,SERVER}_{TESTS,SITE_TESTS}`
109 - `AUTOTEST_{DEPS,PROFILERS,CONFIG}`
114 - List variables (`AUTOTEST_*_LIST`) define the list of deps, profilers,
116 - IUSE test list specification TESTS=, is a USE_EXPANDed specification of
117 tests managed by the given ebuild. By virtue of being an IUSE variable, all
119 unlike with list variables which are a given and the ebuild has to be
123 always have to hold true, however, and in case of autotest, many ebuilds check
125 means that they have to be careful to not install the same files and split the
130 with client site_tests A and B, you can have either:
132 - one ebuild with IUSE_TESTS="+tests_A +tests_B"
133 - two different ebuilds, one with IUSE_TESTS="+tests_A", the other with
137 to ensure the overlapping tests are never enabled at the same time, or file
141 ## Building tests
145 - `run_*()` - This is is the main part that performs all testing and is
147 - `setup()` - This function, present in the test case’s main .py file is
155 nothing to build).
161 which happens on the **host and target alike**. This creates a problem with
164 in the host chroot and on the target board to properly support the test. Any
165 binaries would need to be compiled using the host compiler and either ensured
167 cross-compiled again and dynamically chosen while running on target.
171 tests are being run from the intermediate read-only location.
173 Given the above, building is as easy as **emerge**-ing the autotest ebuild that
176 $ emerge-${board} ${test_ebuild}
179 *Currently, tests are organized within these notable ebuilds*: (see
180 [FAQ](#Q1_What-autotest-ebuilds-are-out-there_) full list):
182 - chromeos-base/autotest-tests - The main ebuild handling most of autotest.git
183 repository and its client and server tests.
184 - chromeos-base/autotest-tests-* - Various ebuilds that build other parts of
186 - chromeos-base/chromeos-chrome - chrome tests; the tests that are part of
189 ### Building tests selectively argument
192 subset, or generally a different set of the tests provided by a given ebuild is
194 [USE_EXPANDed](http://devmanual.gentoo.org/general-concepts/use-flags/index.html)
195 flag called TESTS.
197 All USE flags (and therefore tests) have a default state, either enabled (+) or
198 disabled (-), specified directly in the ebuild, that can be manually overridden
201 - Non-Incremental - Simply override the default selection by an entirely new
203 test and don’t want to waste time building the others.
205 $ TESTS="test1 test2" emerge-${board} ${ebuild}
207 - Incremental - All USE_EXPAND flags are also accessible as USE flags, with
208 the appropriate prefix, and can be used incrementally to selectively
209 enable/disable tests in addition to the defaults. This can be useful if you
210 aim to enable a test that is disabled by default and want to test locally.
212 $ USE="test_to_be_enabled -test_to_be_disabled" emerge-${board} \
215 For operations across all tests, following incremental USE wildcard is
216 supported by portage: "tests_*" to select all tests at once (or - to
217 de-select).
219 **NOTE**: Both Incremental and Non-Incremental methods can be set/overriden by
224 ## Running tests
226 **NOTE**: In order to run tests on your device, it needs to have a
227 [test-enabled image](#W4_Create-and-run-a-test-enabled-image-on-your-device).
229 When running tests, fundamentally, you want to either:
231 - Run sets of tests manually - Use case: Developing test cases
233 Take your local test sources, modify them, and then attempt to run them on a
235 that the machine is imaged to a test image, and the image contains all the
236 dependencies needed to support your tests.
238 - Verify a given image - Use case: Developing the projects subject to testing
240 Take an image, re-image the target device and run a test suite on it. This
241 requires either use of build-time autotest artifacts or their reproduction
244 ### Running tests on a machine argument
246 Autotests are run with a tool called
247 …s://chromium.googlesource.com/chromiumos/third_party/autotest/+/refs/heads/main/docs/test-that.md).
249 ### Running tests in a VM - cros_run_test argument
251 VM tests are conveniently wrapped into a script `cros_run_test` that sets up
252 the VM using a given image and then calls `test_that`. This is run by builders
255 If you want to run your tests in a VM (see
256 [here](https://chromium.googlesource.com/chromiumos/docs/+/main/cros_vm.md#Run-an-autotest-in-the-V…
258 - `cros_run_test` starts up a VM and runs autotests using the port
259 - specified (defaults to 9222). As an example:
261 $ cros_run_test --autotest=suite:smoke \
262 --image-path=<my_image_to_start or don't set to use most recent build> \
263 --board=amd64-generic
265 - The emulator command line redirects localhost port 9222 to the emulated
268 - You can then run tests on the correct ssh port with something like
270 $ test_that --board=x86-generic localhost:9222 'f:.*platform_BootPerf/control'
272 - To set the sudo password run set_shared_user_password. Then within the
273 emulator you can press Ctrl-Alt-T to get a terminal, and sudo using this
276 $ ssh -p 9222 root@localhost
278 - Warning: After
280 'betty' is the only board regularly run through pre-CQ and CQ VMTest and so
281 is the most likely to work at ToT. 'betty' is based on 'amd64-generic',
282 so 'amd64-generic' is likely to also work for most (non-ARC) tests.
288 [autotest-results-logs](https://www.chromium.org/chromium-os/testing/test-code-labs/autotest-client…
294 `test_that` run, you will see a summary of pass/failure status, along with
301 22:44:32 INFO | GOOD ---- Autotest.install timestamp=1263509072 localtime=Jan 14 22:44:32
303 22:44:36 INFO | START ---- ---- timestamp=1263509075 localtime=Jan 14 14:44:35
305 22:44:36 INFO | Bundling /usr/local/autotest/client/tests/sleeptest into test-sleeptest.tar.bz2
308 22:44:42 INFO | END GOOD ---- ---- timestamp=1263509082 localtime=Jan 14 14:44:42
316 Finished running tests. Results can be found in /tmp/test_that_results_j8GoWH or /tmp/test_that_lat…
319 This directory will contain a directory per test run. Each directory contains
320 the logs pertaining to that test run.
324 ${TEST}/debug/client.DEBUG - the most detailed output from running the
325 client-side test
327 ### Running tests automatically, Suites
329 Suites provide a mechanism to group tests together in test groups. They also
330 serve as hooks for automated runs of tests verifying various builds. Most
331 importantly, that is the BVT (board verification tests) and Smoke (a subset of
332 BVT that can run in a VM).
334 Please refer to the [suites documentation](https://www.chromium.org/chromium-os/testing/test-suites…
336 ## Writing and developing tests
340 For understanding and writing the actual python code for autotest, please refer
341 … the [Developer FAQ](http://www.chromium.org/chromium-os/testing/autotest-developer-faq#TOC-Writin…
349 an ebuild. It is generally possible to run tests without an ebuild using
355 The choice of ebuild depends on the location of its sources. Structuring tests
359 - Categorisation - Grouping similar tests together, possibly with deps they
361 - Parallelisation - Multiple independent ebuilds can build entirely in
363 - Dependency tracking - Larger bundles of tests depend on more system
368 blockers for parallelism, ie. tests depending on chrome or similar packages,
369 and as such, using any of the current ebuilds should be sufficient. (see FAQ
374 ebuilds have. Failing to do so will simply make ebuilds ignore your tests
376 enabled by default, and should be the default, unless you want to keep the test
381 **chromeos-base/autotest-all** package, which is a meta-ebuild depending on all
382 autotest ebuild packages that can be built. autotest-all is used by the build
383 system to automatically build all tests that we have and therefore keep them
388 Autotest uses deps to provide a de-facto dependencies into the ecosystem. A dep
391 `setup()` function in its own `setup()` function and will be able to access the
396 automatically detect these and make them an inter-package dependencies on the
398 [provided](#Ebuild-setup_autotest-eclass) by the same ebuild that builds test
400 the dep ebuild and the test ebuild that uses it. An **autotest-deponly**
401 eclass exists to provide solution for ebuilds that build only deps and no
402 tests. A number of deponly ebuilds already exist.
406 - chrome_test - Intending to use any of the test binaries produced by chrome.
407 - pyauto_dep - Using pyauto for your code.
416 the .py file containing the test code, and the class of the Autotest test.
419 control file is \<component\>_\<TestName\>, like the directory and .py
423 control.\<testcase\>. These tests' NAMEs are then
428 ### W1. Develop and iterate on a test
433 $ export TESTS=”<the test cases to iterate on>”
439 $ cros_workon --board=${board} start ${EBUILD}
442 3. Make modifications (on first run, you may want to just do 3,4 to verify
447 4. Build test (TESTS= is not necessary if you exported it before)
449 $ emerge-$board $EBUILD
451 5. Run test to make sure it works before you touch it
453 $ test_that <machine IP> ${TESTS}
458 $ cros_workon --board=${board} stop ${EBUILD}
459 $ unset TESTS
461 ### W2. Creating a test - steps and checklist
467 2. Find the appropriate ebuild package and start working on it:
469 $ cros_workon --board=${board} start <package>
474 $ TESTS=<test> emerge-$board <package>
480 6. Iterate on 4,5 and modify source until happy with the initial version.
485 $ cros_workon --board=${board} stop <package>
489 Removing a test from one ebuild and adding to another in the same revision
490 causes portage file collisions unless counter-measures are taken. Generally,
494 1. We have ebuild **foo-0.0.1-r100** with **test** and would like to split
495 that test off into ebuild **bar-0.0.1-r1**.
497 - both ebuilds are using cros-workon (because it’s likely the case).
498 - foo is used globally (eg. autotest-all depends on it), rather than just
500 2. Remove **test** from foo-{0.0.1-r100,9999}; uprev foo-0.0.1-r100 (to -r101)
501 3. Create bar-9999 (making a copy of foo and replacing IUSE_TESTS may be a good
503 4. Verify package dependencies of test. Make bar-9999 only depend on what is
504 needed for test, remove the dependencies from foo-9999, unless they are
505 needed by tests that remained.
506 5. Add a blocker. Since bar installs files owned by foo-0.0.1-r100 and earlier,
509 RDEPEND="!<=foo-0.0.1-r100"
512 chromeos-base/autotest-all-0.0.1
516 7. Change the dependency of foo in chromeos-base/autotest-all-0.0.1 to be
519 RDEPEND=">foo-0.0.1-r100"
521 8. Uprev (move) autotest-all-0.0.1-rX symlink by one.
524 ### W4. Create and run a test-enabled image on your device
526 1. Choose which board you want to build for (we'll refer to this as ${BOARD},
527 which is for example "x86-generic").
528 2. Set up a proper portage build chroot setup. Go through the normal process
531 $ ./build_packages --board=${BOARD}
533 3. Build test image.
535 $ ./build_image --board=${BOARD} test
540 by entering the shell on the machine and forcing an auto update to your
545 - If you choose to use a USB boot, you first put the image on USB and run
548 $ ./image_to_usb.sh --to /dev/sdX --board=${BOARD} \
549 --image_name=chromiumos_test_image.bin
551 - Alternatively, if you happen to already have a machine running an image
552 modified for test and you know its IP address (${REMOTE_IP}), you can
553 avoid using a USB key and reimage it with a freshly built image by
556 $ ./image_to_live.sh --remote=${REMOTE_IP} \
557 --image=`./get_latest_image.sh \
558 --board=${BOARD}`/chromiumos_test_image.bin
562 print out the new version updated to, and shut down your dev server.
570 foo, you can run:
573 $ for dir in $(portageq-${board} envvar PORTDIR_OVERLAY); do
574 find . -name '*.ebuild' | xargs grep "inherit.*autotest" | grep "9999" | \
575 cut -f1 -d: | \
576 sed -e 's/.*\/\([^/]*\)\/\([^/]*\)\/.*\.ebuild/\1\/\2/'
582 ### Q2: I see a test of the name ‘greattests_TestsEverything’ in build output/logs/whatever! How do…
584 All ebuilds have lists of tests exported as **USE_EXPANDed** lists called
585 **TESTS**. An
591 $ equery-$board hasuse $use_search
593 * [I-O] [ ] some_ebuild_package_name:0
598 **Alternatively**, you can run a pretended emerge (emerge -p) of all autotest
599 ebuilds and scan the output.
601 $ emerge -p ${all_ebuilds_from_Q1} |grep -C 10 “${use_search}”
607 question out (and it shouldn’t be hard). However, all present autotest ebuilds
608 (at the time of this writing) are also ‘cros-workon’, and for those, this
612 $ ebuild $(equery-$board which $ebuild_search) info
617 ### Q4: I have an ebuild, what tests does it build?
619 You can run a pretended emerge on the ebuild and observe the ‘TESTS=’
623 $ emerge-$board -pv ${ebuild_name}
627 [ebuild R ] foo-foo_version to /build/$board/ USE="autox hardened tpmtools
628 xset -buildcheck -opengles" TESTS="enabled_test1 enabled_test2 ... enabled_testN
629 -disabled_test1 ...disabled_testN" 0 kB [1]
632 Alternately, you can use equery, which will list tests with the USE_EXPAND
635 $ equery-$board uses ${ebuild_name}
636 [ Legend : U - final flag setting for installation]
637 [ : I - package is installed with flag ]
639 * Found these USE flags for chromeos-base/autotest-tests-9999:
646 - - opengles : <unknown>
648 - - tests_disabled_test : <unknown>
653 You should ‘workon’ and always cros_workon start all ebuilds that have files
655 is installed in `/build/$board/usr/local/autotest/` and would like know which
659 $ equery-$board belongs /build/${board}/usr/local/autotest/client/site_tests/foo_bar/foo_bar.py
661 chromeos-base/autotest-tests-9999 (<filename>)
664 DON’T forget to do equery-$board. Just equery will also work, only never
670 chromeos-base/autotest. Individual test case will each belong to a particular
675 ### Q6: I created a test, added it into ebuild, emerged it, and I’m getting access denied failures.…
678 probably trying to write into the read-only intermediate location. See
679 [explanation](#Building-tests).