1VTS Kselftest Workflow 2====================== 3 4A Guide to Developing Kselftest for VTS/Android 5 6What is Kselftest? 7------------------ 8 9Kselftest is a collection of tests included within the Linux kernel repository 10at tools/testing/selftests. The tests cover a broad range of areas but total 11coverage (as of 4.14 anyway) is very spotty. Many of the tests do not apply to 12Android platforms. 13 14As of mid 2018 there are 23 tests in kselftest which run on ARM in VTS. 15Almost all of them get executed in both 32-bit and 64-bit mode. There are many 16tests which are not enabled because they have dependencies which are not met on 17Android platforms. 18 19How is Kselftest Run in VTS? 20---------------------------- 21 22The Kselftest source is located at external/linux-kselftest in the Android 23tree. It is a set of files (tools/testing/selftests/) copied into its own 24repository from the Linux kernel repository. This is not an exact mirror of 25upstream; there are outstanding changes to kselftest for it to work on Android 26which have not yet been pushed upstream (contained in android/patches). In 27addition to the kselftest source there is also the VTS wrapper around it. This 28is located at test/vts-testcase/kernel/linux_kselftest. Some noteworthy 29directories/files: 30 31* `external/linux-kselftest/Android.bp`: Lists build rules for the kselftest modules built in Soong. 32* `external/linux-kselftest/android/`: Contains Android-specific files, aside from Android.[bp, mk] at top level. 33* `external/linux-kselftest/android/Android.kselftest.mk`: Lists build rules for the kselftest modules built under make. 34* `external/linux-kselftest/android/kselftest_test_list.mk`: Lists all tests that will get pulled into VTS - VTS depends on this list. 35* `external/linux-kselftest/android/README`: Lists details on why particular tests are not enabled in Android. 36* `external/linux-kselftest/tools/testing/selftests`: Kselftest sources. 37* `test/vts-testcase/kernel/linux_kselftest/testcases/`: tests to run in VTS kselftest suites are 38specified in each of the test XML file, e.g, vts_linux_kselftest_arm_32.xml contains tests for 32 bit ARM platform. 39 40Running Kselftest through atest 41------------------------------- 42You can run Kselftest tests with atest, which handles all the setup and build steps. 43 44To run all 32 bit Kselftest tests: 45* `atest vts_linux_kselftest_arm` 46 47To run all 64 bit Kselftest tests: 48* `atest vts_linux_kselftest_arm_64` 49 50To run a single test: 51* `atest vts_linux_kselftest_arm_64:timers_valid-adjtimex_arm_64` 52 53Running Kselftest through VTS 54----------------------------- 55To run VTS kselftest through VTS, it must first be built. VTS is not device specific, 56you need not compile it specifically for the device you wish to run it on, assuming 57it is the same architecture. 58* `. build/envsetup.sh` 59* `lunch` 60* `make -j vts` 61 62Then open vts-tradefed and run the VTS tests (64 bit ARM platform as an example): 63* `vts-tradefed` 64* `vts-tf > run vts-kernel -m vts_linux_kselftest_arm_64` 65 66If you regularly work with multiple devices it may be useful to specify the 67specific device you wish to run VTS on via the serial number: 68* `vts-tf > run vts-kernel -m vts_linux_kselftest_arm_64 -s 000123456789` 69 70Or a specific test: 71* `vts-tf > run vts-kernel -m vts_linux_kselftest_arm_64 -t net_socket` 72 73Running Kselftest Directly 74-------------------------- 75 76Running kselftest tests within VTS can be quite cumbersome, especially if you 77are iterating a lot trying to debug something. Build and run kselftest tests 78faster by doing 79 80* `external/linux-kselftest$ mma` 81* `external/linux-kselftest$ adb sync data` 82 83The test cases will be located at `/data/nativetest{64,}/linux-kselftest/`. 84 85Build just the kselftest test you care about with `make -j kselftest_<testname>` (listed in `external/linux-kselftest/android/kselftest_test_list.mk`). 86 87What Outstanding Issues Exist? 88------------------------------ 89 90The hotlist for kselftest bugs is 91[kselftest-todo](https://buganizer.corp.google.com/hotlists/745928). 92 93When you begin working on a kselftest bug please assign the bug to yourself so 94that others know it is being worked on. 95 96Testing x86_64 97-------------- 98 99It is not advisable to run kselftest tests directly on your host unless you are fully 100aware of what the tests will do and are okay with it. These tests may 101destabilize your box or cause data loss. If you need to run tests on an x86 102platform and are unsure if they are safe you should run them in emulation, in a 103virtualized environment, or on a dedicated development x86 platform. 104 105To run Kselftest tests for x86 platform, you can do: 106* `atest vts_linux_kselftest_x86` 107* `atest vts_linux_kselftest_x86_64` 108 109Sending Fixes Upstream 110---------------------- 111 112Kselftest is part of the upstream Linux kernel and as such you can use the 113get_maintainers.pl script there to see who to send patches to. Here’s an 114example using a commit already upstream: 115 116``` 117smuckle@smuckle:~/repos/linux$ git show 352909b49ba | scripts/get_maintainer.pl 118Shuah Khan <shuah@kernel.org> (maintainer:KERNEL SELFTEST FRAMEWORK,commit_signer:2/6=33%,authored:2/6=33%,added_lines:2/6=33%,removed_lines:3/6=50%) 119Thomas Gleixner <tglx@linutronix.de> (commit_signer:2/6=33%) 120Greg Kroah-Hartman <gregkh@linuxfoundation.org> (commit_signer:1/6=17%,authored:1/6=17%,added_lines:1/6=17%) 121"Radim Krčmář" <rkrcmar@redhat.com> (commit_signer:1/6=17%) 122Ingo Molnar <mingo@kernel.org> (commit_signer:1/6=17%,commit_signer:1/1=100%) 123Andy Lutomirski <luto@kernel.org> (authored:2/6=33%,added_lines:2/6=33%,removed_lines:2/6=33%,authored:1/1=100%,added_lines:500/500=100%) 124"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> (authored:1/6=17%,added_lines:1/6=17%,removed_lines:1/6=17%) 125linux-kernel@vger.kernel.org (open list) 126linux-kselftest@vger.kernel.org (open list:KERNEL SELFTEST FRAMEWORK) 127``` 128 129In summary patches should be sent to linux-kselftest@vger.kernel.org and 130linux-kernel@vger.kernel.org, Shuah Khan (kselftest maintainer), and anyone 131else who has touched the code in question. Standard Linux kernel coding style 132and patch rules apply. They should be checkpatch (scripts/checkpatch.pl in the 133kernel repository) clean and sent in plain text in canonical patch format. One 134easy way to do this is by using git format-patch and git send-email. 135 136Merging Fixes 137------------- 138 139When possible please merge fixes upstream first. Then cherrypick the change 140onto aosp/master in external/linux-kselftest. 141 142If your change cannot go upstream first for whatever reason, please commit a 143patch for it into external/linux-kselftest/android/patches. This allows easier 144tracking of the delta with upstream and streamlines upgrades to new kselftest 145releases. 146 147Updating Kselftest 148------------------ 149 150To merge in a new upstream version of kselftest: 1511. Do a git merge of the upstream tag with the "ours" policy, dropping all upstream changes. Do not commit yet. 1522. Delete tools/testing/selftests and replace it with a copy from the upstream kernel at the same tag as used above. 1533. Apply the patches in android/patches/, resolving conflicts as required. 1544. Test on all supported kernel versions, ensuring that any tests currently enabled in VTS do not generate new failures. 1555. Commit the merge. 156 157If the runtime of kselftest changes significantly be sure to update the runtime-hint and test-timeout parameters to VTS in 158test config files under 159`test/vts-testcase/kernel/linux_kselftest/testcases/`. 160 161Notes on Individual Tests 162------------------------- 163 164### bpf/ 165 166The bpf tests depend on the bpf and elf libraries, and several kernel 167config options that are not required by Android. 168 169### cpufreq/ 170 171Cpufreq is not required by Android. 172 173### cpu-hotplug/ 174 175Cpu hotplug is not required by Android. 176 177### filesystems/dnotify_test.c 178 179This test has an endless loop in it. It is not currently run as part of the 180kselftest suite. 181 182### firmware/ 183 184These tests depend on a kernel module enabled by CONFIG_TEST_FIRMWARE, which is 185not required by Android. 186 187### ftrace/ 188 189Ftrace is a debug feature and not required by Android. 190 191### gpio/ 192 193The gpio test requires a test kernel module and gpiolib, neither of which are 194required by Android. 195 196### ipc/ 197 198The ipc test requires CONFIG_EXPERT and CONFIG_CHECKPOINT_RESTORE be enabled, 199neither of which are required by Android. 200 201### kvm/ 202 203KVM is not required by Android. 204 205### lib/ 206 207The lib/ tests rely on kernel modules enabled by CONFIG_TEST_PRINTF, 208CONFIG_TEST_BITMAP, and CONFIG_PRIME_NUMBERS. None of these are required by 209Android. 210 211### locking/ 212 213The wait-wound mutex test depends on a kernel module enabled by 214CONFIG_WW_MUTEX_SELFTEST which is not required by Android. 215 216### media_tests/ 217 218The media_tests depend on /dev/mediaX and /dev/videoX files which may not be 219present. They also require manual inspection of dmesg and are not part of 220the normal kselftest suite. 221 222### membarrier/ 223 224The membarrier test depends on CONFIG_EXPERT and CONFIG_MEMBARRIER being 225enabled, neither of which are required by Android. 226 227### memfd/ 228 229These tests depend on FUSE which is not present in Android. 230 231### memory-hotplug/ 232 233Memory hotplug is not required by Android. 234 235### mount/ 236 237The mount test depends on CONFIG_USER_NS which is not required by Android. 238 239### mqueue/ 240 241These tests depend on -lrt and -lpopt which are not present in Android. 242 243### net/ 244 245The test_bpf.sh test requires a kernel module enabled by CONFIG_TEST_BPF. 246The netdevice.sh test operates on an ethernet interface. 247The psock_fanout test depends on CONFIG_BPF_SYSCALL which is not required on 248Android. 249 250### netfilter/ 251 252These tests depend on CONFIG_NET_NS and CONFIG_NF_TABLES_INET, neither of which 253are required by Android. 254 255### nsfs/ 256 257These tests depend on CONFIG_USER_NS and CONFIG_PID_NS, neither of which are 258required by Android. 259 260### ntb/ 261 262NTB (non-transparent bridge) is not required by Android. 263 264### pstore/ 265 266Pstore is recommended, not required, for Android. 267 268### ptp/ 269 270PTP support is not required by Android. 271 272### rseq/ 273 274The rseq system call, enabled by CONFIG_RSEQ, is not required by Android. 275 276### sigaltstack/ 277 278This is a test for functionality that is not available in bionic (swapcontext()). 279 280### static_keys/ 281 282This test depends on a kernel module enabled by CONFIG_TEST_STATIC_KEYS. This 283kernel module is not required by Android. 284 285### sync/ 286 287These tests verify the legacy sync API which is deprecated on Android. 288 289### sysctl 290 291This test depends on a kernel module enabled by CONFIG_TEST_SYSCTL. This kernel 292module is not required by Android. 293 294### timers/ 295 296The adjtick test adjusts the system time and takes 2 minutes. 297 298The change_skew test takes too long to run as part of VTS. 299 300The clocksource_switch test takes too long to run as part of VTS and also tries 301all clocksources. 302 303The freq_step test relies on clocks having a greater precision than what may be 304available. 305 306The leap-a-day test repeatedly modifies the system date in order to test leap 307second behavior. This may disrupt the target environment. 308 309The leapcrash test adjusts the system date and takes several minutes to run. 310 311The mqueue_lat test relies on mq_* calls which are not present on Android. 312 313The rtctest_setdate test modifies the RTC's time and date which may disrupt the 314target environment. 315 316The set-2038 test repeatedly sets the time in order to test edge cases. This 317may disrupt the target enviornment. The test also takes a long time. 318 319The set-tz test sets the timezone which may disrupt the target environment. 320 321The skew_consistency test takes too long to run as part of VTS. 322 323### uevent/ 324 325This test depends on CONFIG_USER_NS which is not required by Android. 326 327### user/ 328 329This test depends on a kernel module enabled by CONFIG_TEST_USER_COPY. This 330kernel module is not required by Android. 331 332### vm/ 333 334The hugepage-mmap, hugepage-shm, compaction, and map_hugetlb tests rely on 335hugetlbfs/hugetlb page support which is not required by Android. 336 337The userfaultfd test relies on the userfaultfd syscall which is not required by 338Android. 339 340### watchdog/ 341 342The watchdog test depends on CONFIG_WATCHDOG which is not required by Android. 343 344### zram/ 345 346The zram test depends on CONFIG_ZSMALLOC and CONFIG_ZRAM which are not required 347by Android. 348