1C Test Case Tutorial 2==================== 3 4This is a step-by-step tutorial on writing a simple C LTP test, where topics 5of the LTP and Linux kernel testing will be introduced gradually using a 6concrete example. Most sections will include exercises, some trivial and 7others not so much. If you find an exercise is leading you off at too much of 8a tangent, just leave it for later and move on. 9 10LTP tests can be written in C or Shell script. This tutorial is only for tests 11written in C using the new LTP test API. Note that while we go into some 12detail on using Git, this is not intended as a canonical or complete guide 13for Git. 14 150. Assumptions & Feedback 16------------------------- 17 18We assume the reader is familiar with C, Git and common Unix/Linux/GNU tools 19and has some general knowledge of Operating Systems. Experienced Linux 20developers may find it too verbose while people new to system level Linux 21development may find it overwhelming. 22 23Comments and feedback are welcome, please direct them to the mailing list (see 24+README+). 25 261. Getting Started 27------------------ 28 29Git-clone the main LTP repository as described in the +README+ and change 30directory to the checked-out Git repository. We recommend installing the LTP 31and running one of the tests mentioned in the Quick guide (in the +README+) to 32ensure you are starting from a good state. 33 34We also recommended cloning the Linux kernel repository for reference, this 35guide will refer to files and directories within the mainline kernel 4.12. 36 37[source,shell] 38------------------------------------------------------------------------------ 39$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 40------------------------------------------------------------------------------ 41 42There are a number of other repositories which are useful for reference as 43well, including the GNU C library +glibc+ and the alternative C library 44+musl+. Some system calls are partially or even entirely implemented in user 45land as part of the standard C library. So in these cases, the C library is an 46important reference. +glibc+ is the most common C library for Linux, however 47+musl+ is generally easier to understand. 48 49How system calls are implemented varies from one architecture to another and 50across kernel and C library versions. To find out whether a system call is 51actually accessing the kernel (whether it is actually a system call) on any 52given machine you can use the +strace+ utility. This intercepts system calls 53made by an executable and prints them. We will use this later in the tutorial. 54 552. Choose a System Call to test 56------------------------------- 57 58We will use the +statx()+ system call, to provide a concrete example of a 59test. At the time of writing there is no test for this call which was 60introduced in Linux kernel version 4.11. 61 62Linux system call specific tests are primarily contained in 63+testcases/kernel/syscalls+, but you should also +git grep+ the entire LTP 64repository to check for any existing usages of a system call. 65 66One way to find a system call which is not currently tested by the LTP is to 67look at +include/linux/syscalls.h+ in the kernel tree. 68 69Something the LTP excels at is ensuring bug-fixes are back ported to 70maintenance releases, so targeting a specific regression is another 71option. 72 732.1. Find an untested System call 74~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 75 76Try to find an untested system call which has a manual page (i.e. +man 77syscall+ produces a result). It is a good idea to Git-clone the latest kernel 78man-pages repository. 79 80[source,shell] 81------------------------------------------------------------------------------ 82$ git clone git://git.kernel.org/pub/scm/docs/man-pages/man-pages.git 83------------------------------------------------------------------------------ 84 85At the time of writing the difference between the latest man-pages release and 86the +HEAD+ of the repository (usually the latest commit) is well over 100 87commits. This represents about 9 weeks of changes. If you are using a stable 88Linux distribution, your man-pages package may well be years old. So as with 89the kernel, it is best to have the Git repository as a reference. 90 91You could also find a system call with untested parameters or use whatever it 92is you are planning to use the LTP for. 93 943. Create the test skeleton 95--------------------------- 96 97I shall call my test +statx01.c+, by the time you read this that file name 98will probably be taken, so increment the number in the file name as 99appropriate or replace +statx+ with the system call chosen in exercise 2.1. 100 101[source,shell] 102------------------------------------------------------------------------------ 103$ mkdir testcases/kernel/syscalls/statx 104$ cd testcases/kernel/syscalls/statx 105$ echo statx >> .gitignore 106------------------------------------------------------------------------------ 107 108Next open +statx01.c+ and add the following boilerplate. Make sure to change 109the copy right notice to your name/company, correct the test name and minimum 110kernel version if necessary. I will explain what the code does below. 111 112[source,c] 113------------------------------------------------------------------------------ 114/* 115 * Copyright (c) 2017 Instruction Ignorer <"can't"@be.bothered.com> 116 * 117 * This program is free software: you can redistribute it and/or modify 118 * it under the terms of the GNU General Public License as published by 119 * the Free Software Foundation, either version 2 of the License, or 120 * (at your option) any later version. 121 * 122 * This program is distributed in the hope that it will be useful, 123 * but WITHOUT ANY WARRANTY; without even the implied warranty of 124 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 125 * GNU General Public License for more details. 126 * 127 * You should have received a copy of the GNU General Public License 128 * along with this program. If not, see <http://www.gnu.org/licenses/>. 129 */ 130 131/* 132 * Test statx 133 * 134 * All tests should start with a description of _what_ we are testing. 135 * Non-trivial explanations of _how_ the code works should also go here. 136 * Include relevant links, Git commit hashes and CVE numbers. 137 * Inline comments should be avoided. 138 */ 139 140#include "tst_test.h" 141 142static void run(void) 143{ 144 tst_res(TPASS, "Doing hardly anything is easy"); 145} 146 147static struct tst_test test = { 148 .test_all = run, 149 .min_kver = "4.11", 150}; 151------------------------------------------------------------------------------ 152 153Starting with the +#include+ statement we copy in the main LTP test library 154headers. This includes the most common test API functions and the test harness 155initialisation code. It is important to note that this is a completely 156ordinary, independent C program, however +main()+ is missing because it is 157implemented in +tst_test.h+. 158 159We specify what code we want to run as part of the test using the +tst_test 160test+ structure. Various callbacks can be set by the test writer, including 161+test.test_all+, which we have set to +run()+. The test harness will execute 162this callback in a separate process (using +fork()+), forcibly terminating it 163if it does not return after +test.timeout+ seconds. 164 165We have also set +test.min_kver+ to the kernel version where +statx+ was 166introduced. The test library will determine the kernel version at runtime. If 167the version is less than 4.11 then the test harness will return +TCONF+, 168indicating that this test is not suitable for the current system 169configuration. 170 171Occasionally features are back ported to older kernel versions, so +statx+ may 172exist on kernels with a lower version. However we don't need to worry about 173that unless there is evidence of it happening. 174 175As mentioned in the code itself, you should specify what you are testing and 176the expected outcome, even if it is relatively simple. If your program flow is 177necessarily complex and difficult to understand (which is often the case when 178trying to manipulate the kernel into doing something bad), then a detailed 179explanation of how the code works is welcome. 180 181What you should not do, is use inline comments or include the same level of 182explanation which is written here. As a general rule, if something is easy to 183document, then the code should also be easy to read. So don't document the easy 184stuff (except for the basic test specification). 185 186Before continuing we should compile this and check that the basics work. In 187order to compile the test we need a +Makefile+ in the same subdirectory. If 188one already exists, then nothing needs to be done, otherwise add one with the 189following contents. 190 191[source,make] 192------------------------------------------------------------------------------ 193# Copyright (c) 2017 Linux Test Project 194# 195# This program is free software; you can redistribute it and/or 196# modify it under the terms of the GNU General Public License as 197# published by the Free Software Foundation; either version 2 of 198# the License, or (at your option) any later version. 199# 200# This program is distributed in the hope that it would be useful, 201# but WITHOUT ANY WARRANTY; without even the implied warranty of 202# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 203# GNU General Public License for more details. 204# 205# You should have received a copy of the GNU General Public License 206# along with this program. If not, see <http://www.gnu.org/licenses/>. 207 208top_srcdir ?= ../../../.. 209 210include $(top_srcdir)/include/mk/testcases.mk 211 212include $(top_srcdir)/include/mk/generic_leaf_target.mk 213 214------------------------------------------------------------------------------ 215 216This will automatically add +statx01.c+ as a build target producing a 217+statx01+ executable. Unless you have heavily deviated from the tutorial, and 218probably need to change +top_srcdir+, nothing else needs to be done. 219 220Normally, if you were starting a Makefile from scratch, then you would need to 221add +statx01+ as a build target. Specifying that you would like to run some 222program (e.g. +gcc+ or +clang+) to transform +statx01.c+ into +statx01+. Here 223we don't need to do that, but sometimes it is still necessary. For example, if 224we needed to link to the POSIX threading library, then we could add the 225following line after +testcases.mk+. 226 227[source,make] 228-------------------------------------------------------------------------------- 229statx01: CFLAGS += -pthread 230-------------------------------------------------------------------------------- 231 232Assuming you are in the test's subdirectory +testcases/kernel/syscalls/statx+, 233do 234 235[source,shell] 236-------------------------------------------------------------------------------- 237$ make 238$ ./statx01 239-------------------------------------------------------------------------------- 240 241This should build the test and then run it. However, even though the test is 242in the +syscalls+ directory it won't be automatically ran as part of the 243_syscalls_ test group (remember +./runltp -f syscalls+ from the +README+?). For 244this we need to add it to the +runtest+ file. So open +runtest/statx+ and add 245the lines starting with a +++. 246 247[source,diff] 248-------------------------------------------------------------------------------- 249 statvfs01 statvfs01 250 statvfs02 statvfs02 251 252+statx01 statx01 253+ 254 stime01 stime01 255 stime02 stime02 256 257-------------------------------------------------------------------------------- 258 259The +runtest+ files are in a two column format. The first column is the test 260name, which is mainly used by test runners for reporting and filtering. It is 261just a single string of text with no spaces. The second column, which can 262contain spaces, is passed to the shell in order to execute the test. Often it 263is just the executable name, but some tests also take arguments (the LTP has a 264library for argument parsing, by the way). 265 266If you haven't done so already, we should add all these new files to Git. It 267is vitally important that you do not make changes to the master branch. If you 268do then pulling changes from upstream becomes a major issue. So first of all 269create a new branch. 270 271[source,shell] 272-------------------------------------------------------------------------------- 273$ git checkout -b statx01 master 274-------------------------------------------------------------------------------- 275 276Now we want to add the files we have created or modified, but before doing a 277commit make sure you have configured Git correctly. You need to at least set 278your Name and e-mail address in +~/.gitconfig+, but there are some other 279settings which come in handy too. My relatively simple configuration is similar to 280the below 281 282[source,conf] 283-------------------------------------------------------------------------------- 284[user] 285 name = Sarah Jane 286 email = sjane@e-mail.address 287[core] 288 editor = emacs 289[sendemail] 290 smtpServer = smtp.server.address 291-------------------------------------------------------------------------------- 292 293Obviously you need to at least change your name and e-mail. The SMTP server is 294useful for +git send-mail+, which we will discuss later. The editor value is 295used for things like writing commits (without the +-m+ option). 296 297[source,shell] 298-------------------------------------------------------------------------------- 299$ git add -v :/testcases/kernel/syscalls/statx :/runtest/syscalls 300$ git commit -m "statx01: Add new test for statx syscall" 301-------------------------------------------------------------------------------- 302 303This should add all the new files in the +statx+ directory and the +runtest+ 304file. It is good practice to commit early and often. Later on we will do a 305Git-rebase, which allows us to clean up the commit history. So don't worry 306about how presentable your commit log is for now. Also don't hesitate to 307create a new branch when doing the exercises or experimenting. This will allow 308you to diverge from the tutorial and then easily come back again. 309 310I can't emphasize enough that Git makes things easy through branching and that 311things quickly get complicated if you don't do it. However if you do get into 312a mess, Git-reflog and Git-reset, will usually get you out of it. If you also 313mess that up then it may be possible to cherry pick 'dangling' commits out of 314the database into a branch. 315 3163.1 Report TCONF instead of TPASS 317~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 318 319Maybe the test should report "TCONF: Not implemented" instead or perhaps 320+TBROK+. Try changing it do so (see +doc/test-writing-guidelines.txt+ or 321https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[the 322Wiki]). 323 3243.2 Check Git ignores the executable 325~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 326 327Is your +.gitignore+ correct? 328 3293.3 Run checkpatch.pl on the source file 330~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 331 332The LTP follows the Linux style guidelines where possible. Check what happens 333if you run +kernel/linux/scripts/checkpatch.pl --no-tree -f statx01.c+ and 334correct any style issues. 335 3363.4 Install the LTP and run the test with runtest 337~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 338 339Run +statx01+ on its own; similar to the +madvise+ tests in the +README+. 340 3414. Call the system call 342----------------------- 343 344At the time of writing +statx+ has no +glibc+ wrapper. It is also fairly common 345for a distribution's C library version to be older than its kernel or it may use a 346cut down C library in comparison to the GNU one. So we must call +statx()+ 347using the general +syscall()+ interface. 348 349The LTP contains a library for dealing with the +syscall+ interface, which is 350located in +include/lapi+. System call numbers are listed against the relevant 351call in the +*.in+ files (e.g. +x86_64.in+) which are used to generate 352+syscalls.h+, which is the header you should include. On rare occasions you 353may find the system call number is missing from the +*.in+ files and will need 354to add it (see include/lapi/syscall/strip_syscall.awk). 355 356System call numbers vary between architectures, hence why there are multiple 357+*.in+ files for each architecture. You can find the various values for the 358+statx+ system call across a number of +uinstd.h+ files in the Linux kernel. 359 360Note that we don't use the system-call-identifier value available in 361+/usr/include/linux/uinstd.h+ because the kernel might be much newer than the 362user land development packages. 363 364For +statx+ we had to add +statx 332+ to +testcases/kernel/include/x86_64.in+, 365+statx 383+ to +testcases/kernel/include/powerpc.in+, etc. Now lets look at 366the code, which I will explain in more detail further down. 367 368[source,c] 369-------------------------------------------------------------------------------- 370/* 371 * Test statx 372 * 373 * Check if statx exists and what error code it returns when we give it dodgy 374 * data. 375 */ 376 377#include <stdint.h> 378#include "tst_test.h" 379#include "lapi/syscalls.h" 380 381struct statx_timestamp { 382 int64_t tv_sec; 383 uint32_t tv_nsec; 384 int32_t __reserved; 385}; 386 387struct statx { 388 uint32_t stx_mask; 389 uint32_t stx_blksize; 390 uint64_t stx_attributes; 391 uint32_t stx_nlink; 392 uint32_t stx_uid; 393 uint32_t stx_gid; 394 uint16_t stx_mode; 395 uint16_t __spare0[1]; 396 uint64_t stx_ino; 397 uint64_t stx_size; 398 uint64_t stx_blocks; 399 uint64_t stx_attributes_mask; 400 struct statx_timestamp stx_atime; 401 struct statx_timestamp stx_btime; 402 struct statx_timestamp stx_ctime; 403 struct statx_timestamp stx_mtime; 404 uint32_t stx_rdev_major; 405 uint32_t stx_rdev_minor; 406 uint32_t stx_dev_major; 407 uint32_t stx_dev_minor; 408 uint64_t __spare2[14]; 409}; 410 411static int sys_statx(int dirfd, const char *pathname, int flags, 412 unsigned int mask, struct statx *statxbuf) 413{ 414 return tst_syscall(__NR_statx, dirfd, pathname, flags, mask, statxbuf); 415} 416 417... 418-------------------------------------------------------------------------------- 419 420So the top part of the code is now boiler plate for calling +statx+. It is 421common for the kernel to be newer than the user land libraries and headers. So 422for new system calls like +statx+, we copy, with a few modifications, the 423relevant definitions into the LTP. This is somewhat like 'vendoring', although 424we are usually just copying headers required for interacting with the Kernel's 425ABI (Application Binary Interface), rather than internalising actual 426functionality. 427 428So from the top we include the +stdint.h+ library which gives us the standard 429+(u)int*_t+ type definitions. We use these in place of the Kernel type 430definitions such as +__u64+ in +linux/types.h+. We then have a couple of 431structure definitions which form part of the +statx+ API. These were copied 432from +include/uapi/linux/stat.h+ in the Kernel tree. 433 434After that, there is a wrapper function, which saves us from writing 435+tst_syscall(__NR_statx, ...+, every time we want to make a call to 436+statx+. This also provides a stub for when +statx+ is eventually integrated 437into the LTP library and also implemented by the C library. At that point we 438can switch to using the C library implementation if available or fallback to 439our own. 440 441The advantage of using the C library implementation is that it will often be 442better supported across multiple architectures. It will also mean we are using 443the system call in the same way most real programs would. Sometimes there are 444advantages to bypassing the C library, but in general it should not be our 445first choice. 446 447The final test should do a check during configuration (i.e. when we run 448+./configure+ before building) which checks if the +statx+ system call and 449associated structures exists. This requires writing an +m4+ file for use with 450+configure.ac+ which is processed during +make autotools+ and produces the 451configure script. 452 453For the time being though we shall just ignore this. All you need to know for 454now is that this is a problem which eventually needs to be dealt with and that 455there is a system in place to handle it. 456 457[source,c] 458-------------------------------------------------------------------------------- 459... 460 461static void run(void) 462{ 463 struct statx statxbuf = { 0 }; 464 465 TEST(sys_statx(0, NULL, 0, 0, &statxbuf)); 466 467 if (TST_RET == 0) 468 tst_res(TFAIL, "statx thinks it can stat NULL"); 469 else if (TST_ERR == EFAULT) 470 tst_res(TPASS, "statx set errno to EFAULT as expected"); 471 else 472 tst_res(TFAIL | TERRNO, "statx set errno to some unexpected value"); 473} 474 475static struct tst_test test = { 476 .test_all = run, 477 .min_kver = "4.11", 478}; 479-------------------------------------------------------------------------------- 480 481The +TEST+ macro sets +TST_RET+ to the return value of +tst_statx()+ and 482+TST_ERR+ to the value of +errno+ immediately after the functions 483return. This is mainly just for convenience, although it potentially could 484have other uses. 485 486We check whether the return value indicates success and if it doesn't also 487check the value of +errno+. The last call to +tst_res+ includes +TERRNO+, 488which will print the current error number and associated description in 489addition to the message we have provided. Note that it uses the current value 490of +errno+ not +TST_ERR+. 491 492What we should have done in the example above is use +TTERRNO+ which takes the 493value of +TST_ERR+. 494 495If we try to run the test on a kernel where +statx+ does not exist, then 496+tst_syscall+ will cause it to fail gracefully with +TCONF+. Where +TCONF+ 497indicates the test is not applicable to our configuration. 498 499The function +tst_syscall+ calls +tst_brk(TCONF,...)+ on failure. +tst_brk+ 500causes the test to exit immediately, which prevents any further test code from 501being run. 502 5034.1 What are the differences between tst_brk and tst_res? 504~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 505 506See +include/tst_test.h+ and the 507https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[test 508writing guide]. Also what do they have in common? 509 5104.2 What happens if you call tst_res(TINFO, ...) after sys_statx? 511~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 512 513Does the test still function correctly? 514 5154.3 Extend the test to handle other basic error conditions. 516~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 517 518For example, see if you can trigger +ENOENT+ instead. You shouldn't 519have to create any files, which is discussed in the next section. 520 5215. Setup, Cleanup and files 522--------------------------- 523 524Some tests require resources to be allocated, or system settings to be 525changed, before the test begins. This 'setup' only has to be done once at the 526beginning and at the end of the test needs to be removed or reverted. The 527'cleanup' also has to be done regardless of whether the test breaks. 528 529Fortunately, like most test libraries, we have setup and cleanup (teardown) 530callbacks. +setup+ is called once before +run+ and +cleanup+ is called once 531afterwards. Note that +run+ itself can be called multiple times by the test 532harness, but that +setup+ and +cleanup+ are only called once. 533 534If either your code, a +SAFE_*+ macro or a library function such as 535+tst_syscall+ call +tst_brk+, then +run+ will exit immediately and the 536+cleanup+ function is then called. Once 'cleanup' is completed, the test 537executable will then exit altogether abandoning any remaining iterations of 538+run+. 539 540For +statx+ we would like to create some files or file like objects which we 541have control over. Deciding where to create the files is easy, we just create 542it in the current working directory and let the LTP test harness handle where 543that should be by setting +.needs_tmpdir = 1+. 544 545[source,c] 546-------------------------------------------------------------------------------- 547/* 548 * Test statx 549 * 550 * Check if statx exists and what error code it returns when we give it dodgy 551 * data. Then stat a file and check it returns success. 552 */ 553 554#include <stdint.h> 555#include "tst_test.h" 556#include "lapi/syscalls.h" 557#include "lapi/fcntl.h" 558 559#define FNAME "file_to_stat" 560#define STATX_BASIC_STATS 0x000007ffU 561 562/*************** statx structure and wrapper goes here ! ***************/ 563 564... 565-------------------------------------------------------------------------------- 566 567We have added an extra include +lapi/fcntl.h+ which wraps the system header by 568the same name (+#include <fcntl.h>+). This header ensures we have definitions 569for recently added macros such as +AT_FDCWD+ by providing fall backs if the 570system header does not have them. The +lapi+ directory contains a number of 571headers like this. 572 573At some point we may wish to add +lapi/stat.h+ to provide a fall back for 574macros such as +STATX_BASIC_STATS+. However for the time being we have just 575defined it in the test. 576 577[source,c] 578-------------------------------------------------------------------------------- 579... 580 581static void setup(void) 582{ 583 SAFE_TOUCH(FNAME, 0777, NULL); 584} 585 586static void run(void) 587{ 588 struct statx statxbuf = { 0 }; 589 590 TEST(sys_statx(0, NULL, 0, 0, &statxbuf)); 591 if (TST_RET == 0) 592 tst_res(TFAIL, "statx thinks it can stat NULL"); 593 else if (TST_ERR == EFAULT) 594 tst_res(TPASS, "statx set errno to EFAULT as expected"); 595 else 596 tst_res(TFAIL | TERRNO, "statx set errno to some unexpected value"); 597 598 TEST(sys_statx(AT_FDCWD, FNAME, 0, STATX_BASIC_STATS, &statxbuf)); 599 if (TST_RET == 0) 600 tst_res(TPASS, "It returned zero so it must have worked!"); 601 else 602 tst_res(TFAIL | TERRNO, "statx can not stat a basic file"); 603} 604 605static struct tst_test test = { 606 .setup = setup, 607 .test_all = run, 608 .min_kver = "4.11", 609 .needs_tmpdir = 1 610}; 611-------------------------------------------------------------------------------- 612 613The +setup+ callback uses one of the LTP's +SAFE+ functions to create an empty 614file +file_to_stat+. Because we have set +.needs_tmpdir+, we can just create 615this file in the present working directory. We don't need to create a 616+cleanup+ callback yet because the LTP test harness will recursively delete 617the temporary directory and its contents. 618 619The +run+ function can be called multiple times by the test harness, however 620+setup+ and +cleanup+ callbacks will only be ran once. 621 622[WARNING] 623By this point you may have begun to explore the LTP library headers or older 624tests. In which case you will have come across functions from the old API such 625as +tst_brkm+. The old API is being phased out, so you should not use these 626functions. 627 628So far we haven't had to do any clean up. So our example doesn't answer the 629question "what happens if part of the clean up fails?". To answer this we are 630going to modify the test to ask the (highly contrived) question "What happens 631if I create and open a file, then create a hard-link to it, then call open 632again on the hard-link, then 'stat' the file". 633 634[source,c] 635-------------------------------------------------------------------------------- 636#define LNAME "file_to_stat_link" 637 638... 639 640static void setup(void) 641{ 642 fd = SAFE_OPEN(FNAME, O_CREAT, 0777); 643 SAFE_LINK(FNAME, LNAME); 644 lfd = SAFE_OPEN(LNAME, 0); 645} 646 647static void cleanup(void) 648{ 649 if (lfd != 0) 650 SAFE_CLOSE(lfd); 651 652 if (fd != 0) 653 SAFE_CLOSE(fd); 654} 655 656static void run(void) 657{ 658 ... 659 660 TEST(sys_statx(AT_FDCWD, LNAME, 0, STATX_BASIC_STATS, &statxbuf)); 661 if (TST_RET == 0) 662 tst_res(TPASS, "It returned zero so it must have worked!"); 663 else 664 tst_res(TFAIL | TERRNO, "statx can not stat a basic file"); 665} 666 667static struct tst_test test = { 668 .setup = setup, 669 .cleanup = cleanup, 670 .test_all = run, 671 .tcnt = 2, 672 .min_kver = "4.11", 673 .needs_tmpdir = 1 674}; 675-------------------------------------------------------------------------------- 676 677Because we are now opening a file, we need a +cleanup+ function to close the 678file descriptors. We have to manually close the files to ensure the temporary 679directory is deleted by the test harness (see the 680https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[test 681writing guidelines] for details). 682 683As a matter of good practice, the file descriptors are closed in reverse 684order. In some circumstances the order in which clean up is performed is 685significant. In that case resources created towards the end of 'setup' are 686dependent on ones near the beginning. So during 'cleanup' we remove the 687dependants before their dependencies. 688 689If, for some reason, the file descriptor +lfd+ became invalid during the test, 690but +fd+ was still open, we do not want +SAFE_CLOSE(lfd)+ to cause the 691+cleanup+ function to exit prematurely. If it did, then +fd+ would remain open 692which would cause problems on some file systems. 693 694Nor do we want to call +cleanup+ recursively. So during 'cleanup' +tst_brk+, 695and consequently the +SAFE+ functions, do not cause the test to exit with 696+TBROK+. Instead they just print an error message with +TWARN+. 697 698It is not entirely necessary to check if the file descriptors have a none zero 699value before attempting to close them. However it avoids a bunch of spurious 700warning messages if we fail to open +file_to_stat+. Test case failures can be 701difficult to interpret at the best of times, so avoid filling the log with 702noise. 703 7045.1 Check statx returns the correct number of hard links 705~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 706 707The field +statx.stx_nlink+ should be equal to 2, right? 708 7095.2 Git-branch 710~~~~~~~~~~~~~~ 711 712We are about to make some organisational changes to the test, so now would be 713a good time to branch. Then we can switch between the old and new versions, to 714check the behavior has not been changed by accident. 715 7166. Split the test 717----------------- 718 719In our current test, we have essentially rolled two different test cases into 720one. Firstly we check if an error is returned when bad arguments are provided 721and secondly we check what happens when we stat an actual file. Quite often it 722makes sense to call +tst_res+ multiple times in a single test case because we 723are checking different properties of the same result, but here we are clearly 724testing two different scenarios. 725 726So we should split the test in two. One obvious way to do this is to create 727+statx02.c+, but that seems like overkill in order to separate two simple test 728cases. So, for now at least, we are going to do it a different way. 729 730[source,c] 731-------------------------------------------------------------------------------- 732... 733 734static void run_stat_null(void) 735{ 736 struct statx statxbuf = { 0 }; 737 738 TEST(sys_statx(0, NULL, 0, 0, &statxbuf)); 739 if (TST_RET == 0) 740 tst_res(TFAIL, "statx thinks it can stat NULL"); 741 else if (TST_ERR == EFAULT) 742 tst_res(TPASS, "statx set errno to EFAULT as expected"); 743 else 744 tst_res(TFAIL | TERRNO, "statx set errno to some unexpected value"); 745} 746 747static void run_stat_symlink(void) 748{ 749 struct statx statxbuf = { 0 }; 750 751 TEST(sys_statx(AT_FDCWD, LNAME, 0, STATX_BASIC_STATS, &statxbuf)); 752 if (TST_RET == 0) 753 tst_res(TPASS, "It returned zero so it must have worked!"); 754 else 755 tst_res(TFAIL | TERRNO, "statx can not stat a basic file"); 756} 757 758static void run(unsigned int i) 759{ 760 switch(i) { 761 case 0: run_stat_null(); 762 case 1: run_stat_symlink(); 763 } 764} 765 766static struct tst_test test = { 767 .setup = setup, 768 .cleanup = cleanup, 769 .test = run, 770 .tcnt = 2, 771 .min_kver = "4.11", 772 .needs_tmpdir = 1 773}; 774-------------------------------------------------------------------------------- 775 776So we have used an alternative form of the +test+ or +run+ callback which 777accepts an index. Some tests use this index with an array of parameters and 778expected return values. Others do something similar to the above. The index 779can be used how you want so long as each iteration calls +tst_res+ in a 780meaningful way. 781 782If an iteration fails to return a result (i.e. call +tst_res+ with a value 783other than +TINFO+) then the test harness will report +TBROK+ and print the 784iteration which failed. This prevents a scenario in your test from silently 785failing due to some faulty logic. 786 7876.1 What is wrong with the switch statement? 788~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 789 790Were you paying attention? Also see the output of +checkpatch.pl+. 791 7926.2 Test a feature unique to statx 793~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 794 795So far we have not tested anything which is unique to +statx+. So, for 796example, you could check stx_btime is correct (possibly only to within a 797margin of error) and that it differs from +stx_mtime+ after writing to the 798file. 799 800Alternatively you could check that +stx_dev_major+ and +stx_dev_minor+ are set 801correctly. Note that the LTP has helper functions for creating devices and 802file systems (see 803https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines#2214-testing-with-a-block-device[section 8042.2.14] of the Test Writing Guidelines). 805 806This could be quite a challenging exercise. You may wish to tackle an 807altogether different test scenario instead. If you get stuck just move onto 808the next section and come back later. 809 8107. Submitting the test for review 811--------------------------------- 812 813Ignoring the fact we should probably create +lapi/stat.h+ along with a bunch 814of fallback logic in the build system. We can now get our test ready for 815submission. 816 817The first thing you need to do before considering submitting your test is run 818+scripts/checkpatch.pl --no-tree -f+ on +statx01.c+. Again, we use the kernel 819style guidelines where possible. Next you should create a new branch, this 820will allow you to reshape your commit history without fear. 821 822After that we have the pleasure of doing an interactive 'rebase' to clean up 823our commit history. In its current form the test only really needs a single 824commit, but if you have been using Git correctly then you should have 825many. The main reason we want to compress it to a single commit, is to make 826the LTP's Git-log readable. It also allows us to write a coherent description 827of the work as a whole in retrospective. Although, when adding a new test, the 828test description in the code will probably make the commit message redundant. 829 830Anyway, as an example, we shall look at my personal commit history from this 831tutorial and 'rebase' it. You should try following along with your own 832repository. First lets look at the commit history since we branched from 833master. 834 835[source,shell] 836-------------------------------------------------------------------------------- 837$ git log -oneline master..HEAD 838152d39fe7 (HEAD -> tutorial-rebase2, tutorial-rebase) tutorial: Start Submitting patch section 83970f7ce7ce statx01: Stop checkpatch from complaining 840bb0332bd7 tutorial: Fix review problems 8416a87a084a statx01: Fix review problems 842d784b1e85 test-writing-guidelines: Remove old API argument 843c26e1be7a fixup! tutorial 8441e24a5fb5 (me/tutorial-rebase) fixup! tutorial 845568a3f7be fixup! tutorial 84609dd2c829 statx: stage 6 847bfeef7902 statx: stage 5b 84876e03d714 statx: stage 5a 84998f5bc7ac statx: stage 4 8506f8c16438 statx: stage 3 (Add statx01) 8515d93b84d8 Add statx and other syscall numbers 8525ca627b78 tutorial: Add a step-by-step C test tutorial 853-------------------------------------------------------------------------------- 854 855So we have told git to show all the commits which don't exist in 'master', but 856are in +HEAD+, where +HEAD+ is the top of the current branch. The current 857branch is +tutorial-rebase2+ which I just created. I have already done one 858'rebase' and submitted a patch for review, so my original branch was just called 859+tutorial+. 860 861As usual my commit history is starting to look like a bit of mess! There is 862even a commit in there which should not be in the this branch (Remove old API 863argument), however it can be ignored for now and 'cherry picked' into a new branch 864later. 865 866For my patch I actually need at least two commits, one which contains the 867tutorial text and one which contains the test and associated files. So first 868of all I want to 'squash' (amalgamate) all the commits appended with 869+tutorial:+ into the bottom commit. 870 871[source,shell] 872-------------------------------------------------------------------------------- 873$ git rebase -i 5ca627b78\^ 874... 875-------------------------------------------------------------------------------- 876 877This begins an interactive 'rebase' where commit 5ca6427b78 is the earliest 878commit we want to edit. The +^+ symbol after the commit hash, specifies the 879commit before this one. The interactive 'rebase' command takes the last commit 880we want to keep unaltered as it's argument (in other words it takes a 881non-inclusive range). 882 883Upon entering a similar command you will be presented with a text file 884similar to the following. The file should be displayed in your text editor of 885choice, if it doesn't, then you may change the editor variable in +.gitconfig+ 886which was shown in section 3. 887 888[source,rebase] 889-------------------------------------------------------------------------------- 890pick 5ca627b78 tutorial: Add a step-by-step C test tutorial 891pick 5d93b84d8 Add statx and other syscall numbers 892pick 6f8c16438 statx: stage 3 (Add statx01) 893pick 98f5bc7ac statx: stage 4 894pick 76e03d714 statx: stage 5a 895pick bfeef7902 statx: stage 5b 896pick 09dd2c829 statx: stage 6 897pick 568a3f7be fixup! tutorial 898pick 1e24a5fb5 fixup! tutorial 899pick c26e1be7a fixup! tutorial 900pick d784b1e85 test-writing-guidelines: Remove old API argument 901pick 6a87a084a statx01: Fix review problems 902pick bb0332bd7 tutorial: Fix review problems 903pick 70f7ce7ce statx01: Stop checkpatch from complaining 904pick 152d39fe7 tutorial: Start Submitting patch section 905-------------------------------------------------------------------------------- 906 907The last commit from Git-log is shown at the top. The left hand column 908contains the commands we want to run on each commit. +pick+ just means we 909re-apply the commit as-is. We can reorder the lines to apply the commits in a 910different order, but we need to be careful when reordering commits to the same 911file. If your 'rebase' results in a merge conflict, then you have probably 912reordered some commits which contained changes to the same piece of code. 913 914Perhaps a better name for the interactive 'rebase' command would be 'replay'. As 915we pick a point in the commit history, undo all those commits before that 916point, then reapply them one at a time. During the replay we can reorder the 917commits, drop, merge, split and edit them, creating a new history. 918 919The commands I am going to use are +reword+ and +fixup+. The +reword+ command 920allows you to edit a single commit's message. The 'fixup' command 'squashes' a 921commit into the commit above/preceding it, merging the two commits into 922one. The commit which has +fixup+ applied has its commit message deleted. If 923you think a commit might have something useful in its message then you can use 924+squash+ instead. 925 926[source,rebase] 927-------------------------------------------------------------------------------- 928reword 5ca627b78 tutorial: Add a step-by-step C test tutorial 929fixup 568a3f7be fixup! tutorial 930fixup 1e24a5fb5 fixup! tutorial 931fixup c26e1be7a fixup! tutorial 932fixup bb0332bd7 tutorial: Fix review problems 933fixup 152d39fe7 tutorial: Start Submitting patch section 934fixup 276edecab tutorial: Save changes before rebase 935pick 5d93b84d8 Add statx and other syscall numbers 936pick 6f8c16438 statx: stage 3 (Add statx01) 937pick 98f5bc7ac statx: stage 4 938pick 76e03d714 statx: stage 5a 939pick bfeef7902 statx: stage 5b 940pick 09dd2c829 statx: stage 6 941pick d784b1e85 test-writing-guidelines: Remove old API argument 942pick 6a87a084a statx01: Fix review problems 943-------------------------------------------------------------------------------- 944 945So all the commits marked with +fixup+ will be re-played by Git immediately 946after 5ca62 at the top. A new commit will then be created with the amalgamated 947changes of all the commits and 5ca62's log message. It turns out that I didn't 948need to reword anything, but there is no harm in checking. It is easy to 949forget the +Signed-off-by:+ line. 950 951I could now do the same for the commits to the +statx+ test, making the commit 952message prefixes consistent. However I am not actually going to submit the 953test (yet). 954 955I won't attempt to show you this, but if you need to do the opposite and split 956apart a commit. It is also possible using Git-rebase by marking a line with 957+edit+. This will pause Git just after replaying the marked commit. You can 958then use a 'soft' Git-reset to bring the selected commit's changes back into 959the 'index' where you are then able to un-stage some parts before 960re-committing. 961 962You can also use +edit+ and +git commit --amend+ together to change a commit 963deep in your history, but without reseting the 'index'. The 'index' contains 964changes which you have staged with +git add+, but not yet committed. 965 966So now that the commit history has been cleaned up, we need to submit a patch 967to the mailing list or make a pull request on GitHub. The mailing list is the 968preferred place to make submissions and is more difficult for most people, so 969I will only cover that method. 970 971Just before we create the patch, we need to check that our changes will still 972apply to the master branch without problems. To do this we can use another 973type of 'rebase' and then try rebuilding and running the test. 974 975[source,shell] 976-------------------------------------------------------------------------------- 977$ git checkout master 978$ git pull origin 979$ git checkout tutorial-rebase2 980$ git rebase master 981-------------------------------------------------------------------------------- 982 983Above, I update the master branch and then replay our changes onto it using 984+git rebase master+. You may find that after the rebase there is a merge 985conflict. This will result in something which looks like the following (taken 986from a Makefile conflict which was caused by reordering commits in a 'rebase'). 987 988[source,diff] 989-------------------------------------------------------------------------------- 990<<<<<<< HEAD 991cve-2016-7117: LDFLAGS += -lpthread 992======= 993cve-2014-0196: LDFLAGS += -lpthread -lutil -lrt 994cve-2016-7117: LDFLAGS += -lpthread -lrt 995>>>>>>> 4dbfb8e79... Add -lrt 996-------------------------------------------------------------------------------- 997 998The first line tells us this is the beginning of a conflict. The third line 999separates the two conflicting pieces of content and the last line is the end 1000of the conflict. Usually, all you need to do is remove the lines you don't 1001want, stage the changes and continue the 'rebase' with +git rebase 1002--continue+. 1003 1004In order to create a patch e-mail we use +git format-patch+, we can then send 1005that e-mail using +git send-email+. It is also possible to import the patch 1006(+mbox+) file into a number of e-mail programs. 1007 1008[source,shell] 1009-------------------------------------------------------------------------------- 1010$ git format-patch -1 -v 2 -o output --to ltp@lists.linux.it fd3cc8596 1011output/v2-0001-tutorial-Add-a-step-by-step-C-test-tutorial.patch 1012-------------------------------------------------------------------------------- 1013 1014The first argument +-1+ specifies we want one commit from fd3cc8596 1015onwards. If we wanted this commit and the one after it we could specify +-2+ 1016instead. 1017 1018This is my second patch submission so I have used +-v 2+, which indicates this 1019is the second version of a patch set. The +-o+ option specifies the output 1020directory (literally called +output+). The +--to+ option adds the +To:+ e-mail 1021header, which I have set to the LTP mailing list. 1022 1023We can then send this patch with the following command sans +--dry-run+. 1024 1025[source,shell] 1026-------------------------------------------------------------------------------- 1027$ git send-email --dry-run output/v2-0001-tutorial-Add-a-step-by-step-C-test-tutorial.patch 1028-------------------------------------------------------------------------------- 1029 1030Git will ask some questions (which you an ignore) and then tell you what it 1031would do if this weren't a dry-run. In order for this to work you have to have 1032a valid SMTP server set in +.gitconfig+ and also be signed up to the LTP 1033mailing list under the same e-mail address you have configured in Git. You can 1034sign up at https://lists.linux.it/listinfo/ltp. 1035 10368. Doing code review 1037-------------------- 1038 1039While waiting for your test to be reviewed, you are invited and encouraged to 1040review other contributors' code. This may seem bizarre when you are completely 1041new to the project, but there are two important ways in which you can 1042contribute here: 1043 1044A. Point out logical errors in the code. 1045B. Improve your own understanding 1046 1047It doesn't matter whether you know the canonical way of writing an LTP test in 1048C. An error of logic, when properly explained, is usually indisputable. These 1049are the most important errors to find as they always result in false test 1050results. Once someone points out such an error it is usually obvious to 1051everyone that it is a bug and needs to be fixed. 1052 1053Obviously testing the patch is one way of finding errors. You can apply 1054patches using +git am+. Then it is just a case of compiling and running the 1055tests. 1056 1057Finally, reading and attempting to comment on other peoples patches, gives 1058you a better understanding of the reviewers perspective. This is better for 1059the project and for you. 1060 1061Style and organisational issues are best left to after you have found logical 1062errors. 1063 10649. Final notes 1065-------------- 1066 1067Hopefully you can now grasp the structure of an LTP test and have some idea of 1068what is available in the LTP test library. There are a vast number of library 1069functions available (mainly located in include and lib), some of which are 1070documented in the test writing guidelines and many of which are not. 1071 1072We have only scratched the surface of the immense technical complexity of 1073systems programming across multiple Kernel and C lib versions as well as 1074different hardware architectures. The important thing to take away from this 1075is that you have to be conscientious of what will happen on systems different 1076from yours. The LTP has a huge and varied user base, so situations you may 1077thing are unlikely can and do happen to somebody. 1078 1079Of course you don't want to spend time allowing for situations which may never 1080arise either, so you have to do your research and think about each situation 1081critically. The more systems you can test on before submitting your changes, 1082the better, although we understand not everyone has access to a lab. 1083 1084One important topic which has not been covered by this tutorial, is 1085multi-process or multi-threaded testing. The LTP library functions work inside 1086child processes and threads, but their semantics change slightly. There are 1087also various helper functions for synchronising and forking processes. For 1088more information see the Test Writing Guidelines (either at 1089https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[the 1090Wiki] or in ./doc), in particular sections 2.2.7 to 2.2.10 and 2.2.13. 1091 1092When it comes time to submit a test, the preferred way to do it is on the 1093mailing list although you can also use GitHub. The LTP follows similar rules 1094to the kernel for formatting and submitting patches. Generally speaking the 1095review cycle is easier for small patches, so try to make small changes or 1096additions where possible. 1097