1C Test Case Tutorial 2==================== 3 4NOTE: See also 5 https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[Test Writing Guidelines], 6 https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API]. 7 8This is a step-by-step tutorial on writing a simple C LTP test, where topics 9of the LTP and Linux kernel testing will be introduced gradually using a 10concrete example. Most sections will include exercises, some trivial and 11others not so much. If you find an exercise is leading you off at too much of 12a tangent, just leave it for later and move on. 13 14LTP tests can be written in C or Shell script. This tutorial is only for tests 15written in C using the new LTP test API. Note that while we go into some 16detail on using Git, this is not intended as a canonical or complete guide 17for Git. 18 190. Assumptions & Feedback 20------------------------- 21 22We assume the reader is familiar with C, Git and common Unix/Linux/GNU tools 23and has some general knowledge of Operating Systems. Experienced Linux 24developers may find it too verbose while people new to system level Linux 25development may find it overwhelming. 26 27Comments and feedback are welcome, please direct them to the mailing list (see 28+README+). 29 301. Getting Started 31------------------ 32 33Git-clone the main LTP repository as described in the +README+ and change 34directory to the checked-out Git repository. We recommend installing the LTP 35and running one of the tests mentioned in the Quick guide (in the +README+) to 36ensure you are starting from a good state. 37 38We also recommended cloning the Linux kernel repository for reference, this 39guide will refer to files and directories within the mainline kernel 4.12. 40 41[source,shell] 42------------------------------------------------------------------------------ 43$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 44------------------------------------------------------------------------------ 45 46There are a number of other repositories which are useful for reference as 47well, including the GNU C library +glibc+ and the alternative C library 48+musl+. Some system calls are partially or even entirely implemented in user 49land as part of the standard C library. So in these cases, the C library is an 50important reference. +glibc+ is the most common C library for Linux, however 51+musl+ is generally easier to understand. 52 53How system calls are implemented varies from one architecture to another and 54across kernel and C library versions. To find out whether a system call is 55actually accessing the kernel (whether it is actually a system call) on any 56given machine you can use the +strace+ utility. This intercepts system calls 57made by an executable and prints them. We will use this later in the tutorial. 58 592. Choose a System Call to test 60------------------------------- 61 62We will use the +statx()+ system call, to provide a concrete example of a 63test. At the time of writing there is no test for this call which was 64introduced in Linux kernel version 4.11. 65 66Linux system call specific tests are primarily contained in 67+testcases/kernel/syscalls+, but you should also +git grep+ the entire LTP 68repository to check for any existing usages of a system call. 69 70One way to find a system call which is not currently tested by the LTP is to 71look at +include/linux/syscalls.h+ in the kernel tree. 72 73Something the LTP excels at is ensuring bug-fixes are back ported to 74maintenance releases, so targeting a specific regression is another 75option. 76 772.1. Find an untested System call 78~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 79 80Try to find an untested system call which has a manual page (i.e. +man 81syscall+ produces a result). It is a good idea to Git-clone the latest kernel 82man-pages repository. 83 84[source,shell] 85------------------------------------------------------------------------------ 86$ git clone git://git.kernel.org/pub/scm/docs/man-pages/man-pages.git 87------------------------------------------------------------------------------ 88 89At the time of writing the difference between the latest man-pages release and 90the +HEAD+ of the repository (usually the latest commit) is well over 100 91commits. This represents about 9 weeks of changes. If you are using a stable 92Linux distribution, your man-pages package may well be years old. So as with 93the kernel, it is best to have the Git repository as a reference. 94 95You could also find a system call with untested parameters or use whatever it 96is you are planning to use the LTP for. 97 983. Create the test skeleton 99--------------------------- 100 101I shall call my test +statx01.c+, by the time you read this that file name 102will probably be taken, so increment the number in the file name as 103appropriate or replace +statx+ with the system call chosen in exercise 2.1. 104 105[source,shell] 106------------------------------------------------------------------------------ 107$ mkdir testcases/kernel/syscalls/statx 108$ cd testcases/kernel/syscalls/statx 109$ echo statx >> .gitignore 110------------------------------------------------------------------------------ 111 112Next open +statx01.c+ and add the following boilerplate. Make sure to change 113the copy right notice to your name/company, correct the test name and minimum 114kernel version if necessary. I will explain what the code does below. 115 116[source,c] 117------------------------------------------------------------------------------ 118// SPDX-License-Identifier: GPL-2.0-or-later 119/* 120 * Copyright (c) 2017 Instruction Ignorer <"can't"@be.bothered.com> 121 */ 122 123/* 124 * Test statx 125 * 126 * All tests should start with a description of _what_ we are testing. 127 * Non-trivial explanations of _how_ the code works should also go here. 128 * Include relevant links, Git commit hashes and CVE numbers. 129 * Inline comments should be avoided. 130 */ 131 132#include "tst_test.h" 133 134static void run(void) 135{ 136 tst_res(TPASS, "Doing hardly anything is easy"); 137} 138 139static struct tst_test test = { 140 .test_all = run, 141 .min_kver = "4.11", 142}; 143------------------------------------------------------------------------------ 144 145Starting with the +#include+ statement we copy in the main LTP test library 146headers. This includes the most common test API functions and the test harness 147initialisation code. It is important to note that this is a completely 148ordinary, independent C program, however +main()+ is missing because it is 149implemented in +tst_test.h+. 150 151We specify what code we want to run as part of the test using the +tst_test 152test+ structure. Various callbacks can be set by the test writer, including 153+test.test_all+, which we have set to +run()+. The test harness will execute 154this callback in a separate process (using +fork()+), forcibly terminating it 155if it does not return after +test.timeout+ seconds. 156 157We have also set +test.min_kver+ to the kernel version where +statx+ was 158introduced. The test library will determine the kernel version at runtime. If 159the version is less than 4.11 then the test harness will return +TCONF+, 160indicating that this test is not suitable for the current system 161configuration. 162 163Occasionally features are back ported to older kernel versions, so +statx+ may 164exist on kernels with a lower version. However we don't need to worry about 165that unless there is evidence of it happening. 166 167As mentioned in the code itself, you should specify what you are testing and 168the expected outcome, even if it is relatively simple. If your program flow is 169necessarily complex and difficult to understand (which is often the case when 170trying to manipulate the kernel into doing something bad), then a detailed 171explanation of how the code works is welcome. 172 173What you should not do, is use inline comments or include the same level of 174explanation which is written here. As a general rule, if something is easy to 175document, then the code should also be easy to read. So don't document the easy 176stuff (except for the basic test specification). 177 178Before continuing we should compile this and check that the basics work. In 179order to compile the test we need a +Makefile+ in the same subdirectory. If 180one already exists, then nothing needs to be done, otherwise add one with the 181following contents. 182 183[source,make] 184------------------------------------------------------------------------------ 185# SPDX-License-Identifier: GPL-2.0-or-later 186# Copyright (c) 2019 Linux Test Project 187 188top_srcdir ?= ../../../.. 189 190include $(top_srcdir)/include/mk/testcases.mk 191 192include $(top_srcdir)/include/mk/generic_leaf_target.mk 193 194------------------------------------------------------------------------------ 195 196This will automatically add +statx01.c+ as a build target producing a 197+statx01+ executable. Unless you have heavily deviated from the tutorial, and 198probably need to change +top_srcdir+, nothing else needs to be done. 199 200Normally, if you were starting a Makefile from scratch, then you would need to 201add +statx01+ as a build target. Specifying that you would like to run some 202program (e.g. +gcc+ or +clang+) to transform +statx01.c+ into +statx01+. Here 203we don't need to do that, but sometimes it is still necessary. For example, if 204we needed to link to the POSIX threading library, then we could add the 205following line after +testcases.mk+. 206 207[source,make] 208-------------------------------------------------------------------------------- 209statx01: CFLAGS += -pthread 210-------------------------------------------------------------------------------- 211 212Assuming you are in the test's subdirectory +testcases/kernel/syscalls/statx+, 213do 214 215[source,shell] 216-------------------------------------------------------------------------------- 217$ make 218$ ./statx01 219-------------------------------------------------------------------------------- 220 221This should build the test and then run it. However, even though the test is 222in the +syscalls+ directory it won't be automatically ran as part of the 223_syscalls_ test group (remember +./runltp -f syscalls+ from the +README+?). For 224this we need to add it to the +runtest+ file. So open +runtest/statx+ and add 225the lines starting with a +++. 226 227[source,diff] 228-------------------------------------------------------------------------------- 229 statvfs01 statvfs01 230 statvfs02 statvfs02 231 232+statx01 statx01 233+ 234 stime01 stime01 235 stime02 stime02 236 237-------------------------------------------------------------------------------- 238 239The +runtest+ files are in a two column format. The first column is the test 240name, which is mainly used by test runners for reporting and filtering. It is 241just a single string of text with no spaces. The second column, which can 242contain spaces, is passed to the shell in order to execute the test. Often it 243is just the executable name, but some tests also take arguments (the LTP has a 244library for argument parsing, by the way). 245 246If you haven't done so already, we should add all these new files to Git. It 247is vitally important that you do not make changes to the master branch. If you 248do then pulling changes from upstream becomes a major issue. So first of all 249create a new branch. 250 251[source,shell] 252-------------------------------------------------------------------------------- 253$ git checkout -b statx01 master 254-------------------------------------------------------------------------------- 255 256Now we want to add the files we have created or modified, but before doing a 257commit make sure you have configured Git correctly. You need to at least set 258your Name and e-mail address in +~/.gitconfig+, but there are some other 259settings which come in handy too. My relatively simple configuration is similar to 260the below 261 262[source,conf] 263-------------------------------------------------------------------------------- 264[user] 265 name = Sarah Jane 266 email = sjane@e-mail.address 267[core] 268 editor = emacs 269[sendemail] 270 smtpServer = smtp.server.address 271-------------------------------------------------------------------------------- 272 273Obviously you need to at least change your name and e-mail. The SMTP server is 274useful for +git send-mail+, which we will discuss later. The editor value is 275used for things like writing commits (without the +-m+ option). 276 277[source,shell] 278-------------------------------------------------------------------------------- 279$ git add -v :/testcases/kernel/syscalls/statx :/runtest/syscalls 280$ git commit -m "statx01: Add new test for statx syscall" 281-------------------------------------------------------------------------------- 282 283This should add all the new files in the +statx+ directory and the +runtest+ 284file. It is good practice to commit early and often. Later on we will do a 285Git-rebase, which allows us to clean up the commit history. So don't worry 286about how presentable your commit log is for now. Also don't hesitate to 287create a new branch when doing the exercises or experimenting. This will allow 288you to diverge from the tutorial and then easily come back again. 289 290I can't emphasize enough that Git makes things easy through branching and that 291things quickly get complicated if you don't do it. However if you do get into 292a mess, Git-reflog and Git-reset, will usually get you out of it. If you also 293mess that up then it may be possible to cherry pick 'dangling' commits out of 294the database into a branch. 295 2963.1 Report TCONF instead of TPASS 297~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 298 299Maybe the test should report "TCONF: Not implemented" instead or perhaps 300+TBROK+. Try changing it do so (see +doc/test-writing-guidelines.txt+ or 301https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[the 302Wiki]). 303 3043.2 Check Git ignores the executable 305~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 306 307Is your +.gitignore+ correct? 308 3093.3 Run make check 310~~~~~~~~~~~~~~~~~~ 311 312Check coding style with `make check` 313 (more in https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines#21-c-coding-style[C coding style]) 314 3153.4 Install the LTP and run the test with runtest 316~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 317 318Run +statx01+ on its own; similar to the +madvise+ tests in the +README+. 319 3204. Call the system call 321----------------------- 322 323At the time of writing +statx+ has no +glibc+ wrapper. It is also fairly common 324for a distribution's C library version to be older than its kernel or it may use a 325cut down C library in comparison to the GNU one. So we must call +statx()+ 326using the general +syscall()+ interface. 327 328The LTP contains a library for dealing with the +syscall+ interface, which is 329located in +include/lapi+. System call numbers are listed against the relevant 330call in the +*.in+ files (e.g. +x86_64.in+) which are used to generate 331+syscalls.h+, which is the header you should include. On rare occasions you 332may find the system call number is missing from the +*.in+ files and will need 333to add it (see +include/lapi/syscalls/strip_syscall.awk+). 334 335System call numbers vary between architectures, hence why there are multiple 336+*.in+ files for each architecture. You can find the various values for the 337+statx+ system call across a number of +uinstd.h+ files in the Linux kernel. 338 339Note that we don't use the system-call-identifier value available in 340+/usr/include/linux/uinstd.h+ because the kernel might be much newer than the 341user land development packages. 342 343For +statx+ we had to add +statx 332+ to +testcases/kernel/include/x86_64.in+, 344+statx 383+ to +testcases/kernel/include/powerpc.in+, etc. Now lets look at 345the code, which I will explain in more detail further down. 346 347[source,c] 348-------------------------------------------------------------------------------- 349/* 350 * Test statx 351 * 352 * Check if statx exists and what error code it returns when we give it dodgy 353 * data. 354 */ 355 356#include <stdint.h> 357#include "tst_test.h" 358#include "lapi/syscalls.h" 359 360struct statx_timestamp { 361 int64_t tv_sec; 362 uint32_t tv_nsec; 363 int32_t __reserved; 364}; 365 366struct statx { 367 uint32_t stx_mask; 368 uint32_t stx_blksize; 369 uint64_t stx_attributes; 370 uint32_t stx_nlink; 371 uint32_t stx_uid; 372 uint32_t stx_gid; 373 uint16_t stx_mode; 374 uint16_t __spare0[1]; 375 uint64_t stx_ino; 376 uint64_t stx_size; 377 uint64_t stx_blocks; 378 uint64_t stx_attributes_mask; 379 struct statx_timestamp stx_atime; 380 struct statx_timestamp stx_btime; 381 struct statx_timestamp stx_ctime; 382 struct statx_timestamp stx_mtime; 383 uint32_t stx_rdev_major; 384 uint32_t stx_rdev_minor; 385 uint32_t stx_dev_major; 386 uint32_t stx_dev_minor; 387 uint64_t __spare2[14]; 388}; 389 390static int sys_statx(int dirfd, const char *pathname, int flags, 391 unsigned int mask, struct statx *statxbuf) 392{ 393 return tst_syscall(__NR_statx, dirfd, pathname, flags, mask, statxbuf); 394} 395 396... 397-------------------------------------------------------------------------------- 398 399So the top part of the code is now boiler plate for calling +statx+. It is 400common for the kernel to be newer than the user land libraries and headers. So 401for new system calls like +statx+, we copy, with a few modifications, the 402relevant definitions into the LTP. This is somewhat like 'vendoring', although 403we are usually just copying headers required for interacting with the Kernel's 404ABI (Application Binary Interface), rather than internalising actual 405functionality. 406 407So from the top we include the +stdint.h+ library which gives us the standard 408+(u)int*_t+ type definitions. We use these in place of the Kernel type 409definitions such as +__u64+ in +linux/types.h+. We then have a couple of 410structure definitions which form part of the +statx+ API. These were copied 411from +include/uapi/linux/stat.h+ in the Kernel tree. 412 413After that, there is a wrapper function, which saves us from writing 414+tst_syscall(__NR_statx, ...+, every time we want to make a call to 415+statx+. This also provides a stub for when +statx+ is eventually integrated 416into the LTP library and also implemented by the C library. At that point we 417can switch to using the C library implementation if available or fallback to 418our own. 419 420The advantage of using the C library implementation is that it will often be 421better supported across multiple architectures. It will also mean we are using 422the system call in the same way most real programs would. Sometimes there are 423advantages to bypassing the C library, but in general it should not be our 424first choice. 425 426The final test should do a check during configuration (i.e. when we run 427+./configure+ before building) which checks if the +statx+ system call and 428associated structures exists. This requires writing an +m4+ file for use with 429+configure.ac+ which is processed during +make autotools+ and produces the 430configure script. 431 432For the time being though we shall just ignore this. All you need to know for 433now is that this is a problem which eventually needs to be dealt with and that 434there is a system in place to handle it. 435 436[source,c] 437-------------------------------------------------------------------------------- 438... 439 440static void run(void) 441{ 442 struct statx statxbuf = { 0 }; 443 444 TEST(sys_statx(0, NULL, 0, 0, &statxbuf)); 445 446 if (TST_RET == 0) 447 tst_res(TFAIL, "statx thinks it can stat NULL"); 448 else if (TST_ERR == EFAULT) 449 tst_res(TPASS, "statx set errno to EFAULT as expected"); 450 else 451 tst_res(TFAIL | TERRNO, "statx set errno to some unexpected value"); 452} 453 454static struct tst_test test = { 455 .test_all = run, 456 .min_kver = "4.11", 457}; 458-------------------------------------------------------------------------------- 459 460The +TEST+ macro sets +TST_RET+ to the return value of +tst_statx()+ and 461+TST_ERR+ to the value of +errno+ immediately after the functions 462return. This is mainly just for convenience, although it potentially could 463have other uses. 464 465We check whether the return value indicates success and if it doesn't also 466check the value of +errno+. The last call to +tst_res+ includes +TERRNO+, 467which will print the current error number and associated description in 468addition to the message we have provided. Note that it uses the current value 469of +errno+ not +TST_ERR+. 470 471What we should have done in the example above is use +TTERRNO+ which takes the 472value of +TST_ERR+. 473 474If we try to run the test on a kernel where +statx+ does not exist, then 475+tst_syscall+ will cause it to fail gracefully with +TCONF+. Where +TCONF+ 476indicates the test is not applicable to our configuration. 477 478The function +tst_syscall+ calls +tst_brk(TCONF,...)+ on failure. +tst_brk+ 479causes the test to exit immediately, which prevents any further test code from 480being run. 481 4824.1 What are the differences between tst_brk and tst_res? 483~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 484 485See +include/tst_test.h+ and the 486https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[test 487writing guide]. Also what do they have in common? 488 4894.2 What happens if you call tst_res(TINFO, ...) after sys_statx? 490~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 491 492Does the test still function correctly? 493 4944.3 Extend the test to handle other basic error conditions. 495~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 496 497For example, see if you can trigger +ENOENT+ instead. You shouldn't 498have to create any files, which is discussed in the next section. 499 5005. Setup, Cleanup and files 501--------------------------- 502 503Some tests require resources to be allocated, or system settings to be 504changed, before the test begins. This 'setup' only has to be done once at the 505beginning and at the end of the test needs to be removed or reverted. The 506'cleanup' also has to be done regardless of whether the test breaks. 507 508Fortunately, like most test libraries, we have setup and cleanup (teardown) 509callbacks. +setup+ is called once before +run+ and +cleanup+ is called once 510afterwards. Note that +run+ itself can be called multiple times by the test 511harness, but that +setup+ and +cleanup+ are only called once. 512 513If either your code, a +SAFE_*+ macro or a library function such as 514+tst_syscall+ call +tst_brk+, then +run+ will exit immediately and the 515+cleanup+ function is then called. Once 'cleanup' is completed, the test 516executable will then exit altogether abandoning any remaining iterations of 517+run+. 518 519For +statx+ we would like to create some files or file like objects which we 520have control over. Deciding where to create the files is easy, we just create 521it in the current working directory and let the LTP test harness handle where 522that should be by setting +.needs_tmpdir = 1+. 523 524[source,c] 525-------------------------------------------------------------------------------- 526/* 527 * Test statx 528 * 529 * Check if statx exists and what error code it returns when we give it dodgy 530 * data. Then stat a file and check it returns success. 531 */ 532 533#include <stdint.h> 534#include "tst_test.h" 535#include "lapi/syscalls.h" 536#include "lapi/fcntl.h" 537 538#define FNAME "file_to_stat" 539#define STATX_BASIC_STATS 0x000007ffU 540 541/*************** statx structure and wrapper goes here ! ***************/ 542 543... 544-------------------------------------------------------------------------------- 545 546We have added an extra include +lapi/fcntl.h+ which wraps the system header by 547the same name (+#include <fcntl.h>+). This header ensures we have definitions 548for recently added macros such as +AT_FDCWD+ by providing fall backs if the 549system header does not have them. The +lapi+ directory contains a number of 550headers like this. 551 552At some point we may wish to add +lapi/stat.h+ to provide a fall back for 553macros such as +STATX_BASIC_STATS+. However for the time being we have just 554defined it in the test. 555 556[source,c] 557-------------------------------------------------------------------------------- 558... 559 560static void setup(void) 561{ 562 SAFE_TOUCH(FNAME, 0777, NULL); 563} 564 565static void run(void) 566{ 567 struct statx statxbuf = { 0 }; 568 569 TEST(sys_statx(0, NULL, 0, 0, &statxbuf)); 570 if (TST_RET == 0) 571 tst_res(TFAIL, "statx thinks it can stat NULL"); 572 else if (TST_ERR == EFAULT) 573 tst_res(TPASS, "statx set errno to EFAULT as expected"); 574 else 575 tst_res(TFAIL | TERRNO, "statx set errno to some unexpected value"); 576 577 TEST(sys_statx(AT_FDCWD, FNAME, 0, STATX_BASIC_STATS, &statxbuf)); 578 if (TST_RET == 0) 579 tst_res(TPASS, "It returned zero so it must have worked!"); 580 else 581 tst_res(TFAIL | TERRNO, "statx can not stat a basic file"); 582} 583 584static struct tst_test test = { 585 .setup = setup, 586 .test_all = run, 587 .min_kver = "4.11", 588 .needs_tmpdir = 1 589}; 590-------------------------------------------------------------------------------- 591 592The +setup+ callback uses one of the LTP's +SAFE+ functions to create an empty 593file +file_to_stat+. Because we have set +.needs_tmpdir+, we can just create 594this file in the present working directory. We don't need to create a 595+cleanup+ callback yet because the LTP test harness will recursively delete 596the temporary directory and its contents. 597 598The +run+ function can be called multiple times by the test harness, however 599+setup+ and +cleanup+ callbacks will only be ran once. 600 601[WARNING] 602By this point you may have begun to explore the LTP library headers or older 603tests. In which case you will have come across functions from the old API such 604as +tst_brkm+. The old API is being phased out, so you should not use these 605functions. 606 607So far we haven't had to do any clean up. So our example doesn't answer the 608question "what happens if part of the clean up fails?". To answer this we are 609going to modify the test to ask the (highly contrived) question "What happens 610if I create and open a file, then create a hard-link to it, then call open 611again on the hard-link, then 'stat' the file". 612 613[source,c] 614-------------------------------------------------------------------------------- 615#define LNAME "file_to_stat_link" 616 617... 618 619static void setup(void) 620{ 621 fd = SAFE_OPEN(FNAME, O_CREAT, 0777); 622 SAFE_LINK(FNAME, LNAME); 623 lfd = SAFE_OPEN(LNAME, 0); 624} 625 626static void cleanup(void) 627{ 628 if (lfd != 0) 629 SAFE_CLOSE(lfd); 630 631 if (fd != 0) 632 SAFE_CLOSE(fd); 633} 634 635static void run(void) 636{ 637 ... 638 639 TEST(sys_statx(AT_FDCWD, LNAME, 0, STATX_BASIC_STATS, &statxbuf)); 640 if (TST_RET == 0) 641 tst_res(TPASS, "It returned zero so it must have worked!"); 642 else 643 tst_res(TFAIL | TERRNO, "statx can not stat a basic file"); 644} 645 646static struct tst_test test = { 647 .setup = setup, 648 .cleanup = cleanup, 649 .test_all = run, 650 .tcnt = 2, 651 .min_kver = "4.11", 652 .needs_tmpdir = 1 653}; 654-------------------------------------------------------------------------------- 655 656Because we are now opening a file, we need a +cleanup+ function to close the 657file descriptors. We have to manually close the files to ensure the temporary 658directory is deleted by the test harness (see the 659https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[test 660writing guidelines] for details). 661 662As a matter of good practice, the file descriptors are closed in reverse 663order. In some circumstances the order in which clean up is performed is 664significant. In that case resources created towards the end of 'setup' are 665dependent on ones near the beginning. So during 'cleanup' we remove the 666dependants before their dependencies. 667 668If, for some reason, the file descriptor +lfd+ became invalid during the test, 669but +fd+ was still open, we do not want +SAFE_CLOSE(lfd)+ to cause the 670+cleanup+ function to exit prematurely. If it did, then +fd+ would remain open 671which would cause problems on some file systems. 672 673Nor do we want to call +cleanup+ recursively. So during 'cleanup' +tst_brk+, 674and consequently the +SAFE+ functions, do not cause the test to exit with 675+TBROK+. Instead they just print an error message with +TWARN+. 676 677It is not entirely necessary to check if the file descriptors have a none zero 678value before attempting to close them. However it avoids a bunch of spurious 679warning messages if we fail to open +file_to_stat+. Test case failures can be 680difficult to interpret at the best of times, so avoid filling the log with 681noise. 682 6835.1 Check statx returns the correct number of hard links 684~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 685 686The field +statx.stx_nlink+ should be equal to 2, right? 687 6885.2 Git-branch 689~~~~~~~~~~~~~~ 690 691We are about to make some organisational changes to the test, so now would be 692a good time to branch. Then we can switch between the old and new versions, to 693check the behavior has not been changed by accident. 694 6956. Split the test 696----------------- 697 698In our current test, we have essentially rolled two different test cases into 699one. Firstly we check if an error is returned when bad arguments are provided 700and secondly we check what happens when we stat an actual file. Quite often it 701makes sense to call +tst_res+ multiple times in a single test case because we 702are checking different properties of the same result, but here we are clearly 703testing two different scenarios. 704 705So we should split the test in two. One obvious way to do this is to create 706+statx02.c+, but that seems like overkill in order to separate two simple test 707cases. So, for now at least, we are going to do it a different way. 708 709[source,c] 710-------------------------------------------------------------------------------- 711... 712 713static void run_stat_null(void) 714{ 715 struct statx statxbuf = { 0 }; 716 717 TEST(sys_statx(0, NULL, 0, 0, &statxbuf)); 718 if (TST_RET == 0) 719 tst_res(TFAIL, "statx thinks it can stat NULL"); 720 else if (TST_ERR == EFAULT) 721 tst_res(TPASS, "statx set errno to EFAULT as expected"); 722 else 723 tst_res(TFAIL | TERRNO, "statx set errno to some unexpected value"); 724} 725 726static void run_stat_symlink(void) 727{ 728 struct statx statxbuf = { 0 }; 729 730 TEST(sys_statx(AT_FDCWD, LNAME, 0, STATX_BASIC_STATS, &statxbuf)); 731 if (TST_RET == 0) 732 tst_res(TPASS, "It returned zero so it must have worked!"); 733 else 734 tst_res(TFAIL | TERRNO, "statx can not stat a basic file"); 735} 736 737static void run(unsigned int i) 738{ 739 switch(i) { 740 case 0: run_stat_null(); 741 case 1: run_stat_symlink(); 742 } 743} 744 745static struct tst_test test = { 746 .setup = setup, 747 .cleanup = cleanup, 748 .test = run, 749 .tcnt = 2, 750 .min_kver = "4.11", 751 .needs_tmpdir = 1 752}; 753-------------------------------------------------------------------------------- 754 755So we have used an alternative form of the +test+ or +run+ callback which 756accepts an index. Some tests use this index with an array of parameters and 757expected return values. Others do something similar to the above. The index 758can be used how you want so long as each iteration calls +tst_res+ in a 759meaningful way. 760 761If an iteration fails to return a result (i.e. call +tst_res+ with a value 762other than +TINFO+) then the test harness will report +TBROK+ and print the 763iteration which failed. This prevents a scenario in your test from silently 764failing due to some faulty logic. 765 7666.1 What is wrong with the switch statement? 767~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 768 769Were you paying attention? Also see the output of +make check+. 770 7716.2 Test a feature unique to statx 772~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 773 774So far we have not tested anything which is unique to +statx+. So, for 775example, you could check stx_btime is correct (possibly only to within a 776margin of error) and that it differs from +stx_mtime+ after writing to the 777file. 778 779Alternatively you could check that +stx_dev_major+ and +stx_dev_minor+ are set 780correctly. Note that the LTP has helper functions for creating devices and 781file systems (see 782https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines#2214-testing-with-a-block-device[section 7832.2.14] of the Test Writing Guidelines). 784 785This could be quite a challenging exercise. You may wish to tackle an 786altogether different test scenario instead. If you get stuck just move onto 787the next section and come back later. 788 7897. Submitting the test for review 790--------------------------------- 791 792Ignoring the fact we should probably create +lapi/stat.h+ along with a bunch 793of fallback logic in the build system. We can now get our test ready for 794submission. 795 796The first thing you need to do before considering submitting your test is run 797+make check-statx01+ or + make check+ in the test's directory. Again, we use 798the kernel style guidelines where possible. Next you should create a new 799branch, this will allow you to reshape your commit history without fear. 800 801After that we have the pleasure of doing an interactive 'rebase' to clean up 802our commit history. In its current form the test only really needs a single 803commit, but if you have been using Git correctly then you should have 804many. The main reason we want to compress it to a single commit, is to make 805the LTP's Git-log readable. It also allows us to write a coherent description 806of the work as a whole in retrospective. Although, when adding a new test, the 807test description in the code will probably make the commit message redundant. 808 809Anyway, as an example, we shall look at my personal commit history from this 810tutorial and 'rebase' it. You should try following along with your own 811repository. First lets look at the commit history since we branched from 812master. 813 814[source,shell] 815-------------------------------------------------------------------------------- 816$ git log -oneline master..HEAD 817152d39fe7 (HEAD -> tutorial-rebase2, tutorial-rebase) tutorial: Start Submitting patch section 81870f7ce7ce statx01: Stop checkpatch from complaining 819bb0332bd7 tutorial: Fix review problems 8206a87a084a statx01: Fix review problems 821d784b1e85 test-writing-guidelines: Remove old API argument 822c26e1be7a fixup! tutorial 8231e24a5fb5 (me/tutorial-rebase) fixup! tutorial 824568a3f7be fixup! tutorial 82509dd2c829 statx: stage 6 826bfeef7902 statx: stage 5b 82776e03d714 statx: stage 5a 82898f5bc7ac statx: stage 4 8296f8c16438 statx: stage 3 (Add statx01) 8305d93b84d8 Add statx and other syscall numbers 8315ca627b78 tutorial: Add a step-by-step C test tutorial 832-------------------------------------------------------------------------------- 833 834So we have told git to show all the commits which don't exist in 'master', but 835are in +HEAD+, where +HEAD+ is the top of the current branch. The current 836branch is +tutorial-rebase2+ which I just created. I have already done one 837'rebase' and submitted a patch for review, so my original branch was just called 838+tutorial+. 839 840As usual my commit history is starting to look like a bit of mess! There is 841even a commit in there which should not be in the this branch (Remove old API 842argument), however it can be ignored for now and 'cherry picked' into a new branch 843later. 844 845For my patch I actually need at least two commits, one which contains the 846tutorial text and one which contains the test and associated files. So first 847of all I want to 'squash' (amalgamate) all the commits appended with 848+tutorial:+ into the bottom commit. 849 850[source,shell] 851-------------------------------------------------------------------------------- 852$ git rebase -i 5ca627b78\^ 853... 854-------------------------------------------------------------------------------- 855 856This begins an interactive 'rebase' where commit 5ca6427b78 is the earliest 857commit we want to edit. The +^+ symbol after the commit hash, specifies the 858commit before this one. The interactive 'rebase' command takes the last commit 859we want to keep unaltered as it's argument (in other words it takes a 860non-inclusive range). 861 862Upon entering a similar command you will be presented with a text file 863similar to the following. The file should be displayed in your text editor of 864choice, if it doesn't, then you may change the editor variable in +.gitconfig+ 865which was shown in section 3. 866 867[source,rebase] 868-------------------------------------------------------------------------------- 869pick 5ca627b78 tutorial: Add a step-by-step C test tutorial 870pick 5d93b84d8 Add statx and other syscall numbers 871pick 6f8c16438 statx: stage 3 (Add statx01) 872pick 98f5bc7ac statx: stage 4 873pick 76e03d714 statx: stage 5a 874pick bfeef7902 statx: stage 5b 875pick 09dd2c829 statx: stage 6 876pick 568a3f7be fixup! tutorial 877pick 1e24a5fb5 fixup! tutorial 878pick c26e1be7a fixup! tutorial 879pick d784b1e85 test-writing-guidelines: Remove old API argument 880pick 6a87a084a statx01: Fix review problems 881pick bb0332bd7 tutorial: Fix review problems 882pick 70f7ce7ce statx01: Stop checkpatch from complaining 883pick 152d39fe7 tutorial: Start Submitting patch section 884-------------------------------------------------------------------------------- 885 886The last commit from Git-log is shown at the top. The left hand column 887contains the commands we want to run on each commit. +pick+ just means we 888re-apply the commit as-is. We can reorder the lines to apply the commits in a 889different order, but we need to be careful when reordering commits to the same 890file. If your 'rebase' results in a merge conflict, then you have probably 891reordered some commits which contained changes to the same piece of code. 892 893Perhaps a better name for the interactive 'rebase' command would be 'replay'. As 894we pick a point in the commit history, undo all those commits before that 895point, then reapply them one at a time. During the replay we can reorder the 896commits, drop, merge, split and edit them, creating a new history. 897 898The commands I am going to use are +reword+ and +fixup+. The +reword+ command 899allows you to edit a single commit's message. The 'fixup' command 'squashes' a 900commit into the commit above/preceding it, merging the two commits into 901one. The commit which has +fixup+ applied has its commit message deleted. If 902you think a commit might have something useful in its message then you can use 903+squash+ instead. 904 905[source,rebase] 906-------------------------------------------------------------------------------- 907reword 5ca627b78 tutorial: Add a step-by-step C test tutorial 908fixup 568a3f7be fixup! tutorial 909fixup 1e24a5fb5 fixup! tutorial 910fixup c26e1be7a fixup! tutorial 911fixup bb0332bd7 tutorial: Fix review problems 912fixup 152d39fe7 tutorial: Start Submitting patch section 913fixup 276edecab tutorial: Save changes before rebase 914pick 5d93b84d8 Add statx and other syscall numbers 915pick 6f8c16438 statx: stage 3 (Add statx01) 916pick 98f5bc7ac statx: stage 4 917pick 76e03d714 statx: stage 5a 918pick bfeef7902 statx: stage 5b 919pick 09dd2c829 statx: stage 6 920pick d784b1e85 test-writing-guidelines: Remove old API argument 921pick 6a87a084a statx01: Fix review problems 922-------------------------------------------------------------------------------- 923 924So all the commits marked with +fixup+ will be re-played by Git immediately 925after 5ca62 at the top. A new commit will then be created with the amalgamated 926changes of all the commits and 5ca62's log message. It turns out that I didn't 927need to reword anything, but there is no harm in checking. It is easy to 928forget the +Signed-off-by:+ line. 929 930I could now do the same for the commits to the +statx+ test, making the commit 931message prefixes consistent. However I am not actually going to submit the 932test (yet). 933 934I won't attempt to show you this, but if you need to do the opposite and split 935apart a commit. It is also possible using Git-rebase by marking a line with 936+edit+. This will pause Git just after replaying the marked commit. You can 937then use a 'soft' Git-reset to bring the selected commit's changes back into 938the 'index' where you are then able to un-stage some parts before 939re-committing. 940 941You can also use +edit+ and +git commit --amend+ together to change a commit 942deep in your history, but without reseting the 'index'. The 'index' contains 943changes which you have staged with +git add+, but not yet committed. 944 945So now that the commit history has been cleaned up, we need to submit a patch 946to the mailing list or make a pull request on GitHub. The mailing list is the 947preferred place to make submissions and is more difficult for most people, so 948I will only cover that method. 949 950Just before we create the patch, we need to check that our changes will still 951apply to the master branch without problems. To do this we can use another 952type of 'rebase' and then try rebuilding and running the test. 953 954[source,shell] 955-------------------------------------------------------------------------------- 956$ git checkout master 957$ git pull origin 958$ git checkout tutorial-rebase2 959$ git rebase master 960-------------------------------------------------------------------------------- 961 962Above, I update the master branch and then replay our changes onto it using 963+git rebase master+. You may find that after the rebase there is a merge 964conflict. This will result in something which looks like the following (taken 965from a Makefile conflict which was caused by reordering commits in a 'rebase'). 966 967[source,diff] 968-------------------------------------------------------------------------------- 969<<<<<<< HEAD 970cve-2016-7117: LDFLAGS += -lpthread 971======= 972cve-2014-0196: LDFLAGS += -lpthread -lutil -lrt 973cve-2016-7117: LDFLAGS += -lpthread -lrt 974>>>>>>> 4dbfb8e79... Add -lrt 975-------------------------------------------------------------------------------- 976 977The first line tells us this is the beginning of a conflict. The third line 978separates the two conflicting pieces of content and the last line is the end 979of the conflict. Usually, all you need to do is remove the lines you don't 980want, stage the changes and continue the 'rebase' with +git rebase 981--continue+. 982 983In order to create a patch e-mail we use +git format-patch+, we can then send 984that e-mail using +git send-email+. It is also possible to import the patch 985(+mbox+) file into a number of e-mail programs. 986 987[source,shell] 988-------------------------------------------------------------------------------- 989$ git format-patch -1 -v 2 -o output --to ltp@lists.linux.it fd3cc8596 990output/v2-0001-tutorial-Add-a-step-by-step-C-test-tutorial.patch 991-------------------------------------------------------------------------------- 992 993The first argument +-1+ specifies we want one commit from fd3cc8596 994onwards. If we wanted this commit and the one after it we could specify +-2+ 995instead. 996 997This is my second patch submission so I have used +-v 2+, which indicates this 998is the second version of a patch set. The +-o+ option specifies the output 999directory (literally called +output+). The +--to+ option adds the +To:+ e-mail 1000header, which I have set to the LTP mailing list. 1001 1002We can then send this patch with the following command sans +--dry-run+. 1003 1004[source,shell] 1005-------------------------------------------------------------------------------- 1006$ git send-email --dry-run output/v2-0001-tutorial-Add-a-step-by-step-C-test-tutorial.patch 1007-------------------------------------------------------------------------------- 1008 1009Git will ask some questions (which you an ignore) and then tell you what it 1010would do if this weren't a dry-run. In order for this to work you have to have 1011a valid SMTP server set in +.gitconfig+ and also be signed up to the LTP 1012mailing list under the same e-mail address you have configured in Git. You can 1013sign up at https://lists.linux.it/listinfo/ltp. 1014 10158. Doing code review 1016-------------------- 1017 1018While waiting for your test to be reviewed, you are invited and encouraged to 1019review other contributors' code. This may seem bizarre when you are completely 1020new to the project, but there are two important ways in which you can 1021contribute here: 1022 1023A. Point out logical errors in the code. 1024B. Improve your own understanding 1025 1026It doesn't matter whether you know the canonical way of writing an LTP test in 1027C. An error of logic, when properly explained, is usually indisputable. These 1028are the most important errors to find as they always result in false test 1029results. Once someone points out such an error it is usually obvious to 1030everyone that it is a bug and needs to be fixed. 1031 1032Obviously testing the patch is one way of finding errors. You can apply 1033patches using +git am+. Then it is just a case of compiling and running the 1034tests. 1035 1036Finally, reading and attempting to comment on other peoples patches, gives 1037you a better understanding of the reviewers perspective. This is better for 1038the project and for you. 1039 1040Style and organisational issues are best left to after you have found logical 1041errors. 1042 10439. Final notes 1044-------------- 1045 1046Hopefully you can now grasp the structure of an LTP test and have some idea of 1047what is available in the LTP test library. There are a vast number of library 1048functions available (mainly located in include and lib), some of which are 1049documented in the test writing guidelines and many of which are not. 1050 1051We have only scratched the surface of the immense technical complexity of 1052systems programming across multiple Kernel and C lib versions as well as 1053different hardware architectures. The important thing to take away from this 1054is that you have to be conscientious of what will happen on systems different 1055from yours. The LTP has a huge and varied user base, so situations you may 1056thing are unlikely can and do happen to somebody. 1057 1058Of course you don't want to spend time allowing for situations which may never 1059arise either, so you have to do your research and think about each situation 1060critically. The more systems you can test on before submitting your changes, 1061the better, although we understand not everyone has access to a lab. 1062 1063One important topic which has not been covered by this tutorial, is 1064multi-process or multi-threaded testing. The LTP library functions work inside 1065child processes and threads, but their semantics change slightly. There are 1066also various helper functions for synchronising and forking processes. For 1067more information see 1068https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API], 1069in particular sections 1070https://github.com/linux-test-project/ltp/wiki/C-Test-API#17-fork-ing[1.7 Fork()-ing] to 1071https://github.com/linux-test-project/ltp/wiki/C-Test-API#110-signals-and-signal-handlers[1.10 Signals and signal handlers] and 1072https://github.com/linux-test-project/ltp/wiki/C-Test-API#114-thread-safety-in-the-ltp-library[1.14 Thread-safety in the LTP library]. 1073 1074When it comes time to submit a test, the preferred way to do it is on the 1075mailing list although you can also use GitHub. The LTP follows similar rules 1076to the kernel for formatting and submitting patches. Generally speaking the 1077review cycle is easier for small patches, so try to make small changes or 1078additions where possible. 1079