1 2 3Linux Test Project HOWTO 4 510 October 2000 6 7Nate Straz 8 9Abstract 10 11This document explains some of the more in depth topics of 12the Linux Test Project and related testing issues. It does 13not cover basic installation procedures. See the INSTALL 14and README files in the tarball for that information. 15 161 Preface 17 18This document was written to help bring the community up 19to speed on the ins and outs of the Linux Test Project. 20 211.1 Copyright 22 23Copyright (c) 2000 by SGI, Inc. 24 25Please freely copy and distribute (sell or give away) this 26document in any format. It's requested that corrections 27and/or comments be forwarded to the document maintainer. 28You may create a derivative work and distribute it provided 29that you: 30 31* Send your derivative work (in the most suitable format 32 such as sgml) to the LDP (Linux Documentation Project) 33 or the like for posting on the Internet. If not the LDP, 34 then let the LDP know where it is available. 35 36* License the derivative work with this same license or use 37 GPL. Include a copyright notice and at least a pointer 38 to the license used. 39 40* Give due credit to previous authors and major contributors. 41 42If you're considering making a derived work other than a 43translation, it's requested that you discuss your plans 44with the current maintainer. 45 461.2 Disclaimer 47 48Use the information in this document at your own risk. I 49disavow any potential liability for the contents of this 50document. Use of the concepts, examples, and/or other content 51of this document is entirely at your own risk. 52 53All copyrights are owned by their owners, unless specifically 54noted otherwise. Use of a term in this document should 55not be regarded as affecting the validity of any trademark 56or service mark. 57 58Naming of particular products or brands should not be seen 59as endorsements. 60 61You are strongly recommended to take a backup of your system 62before major installation and backups at regular intervals. 63 642 Introduction 65 662.1 What is the Linux Test Project? 67 68The Linux Test Project (LTP) is an effort to create a set 69of tools and tests to verify the functionality and stability 70of the Linux kernel. We hope this will support Linux development 71by making unit testing more complete and minimizing user 72impact by building a barrier to keep bugs from making it 73to the user. 74 752.2 What is wrong with the current testing model? 76 77The Linux development community utilizes two important (some 78out argue most important) testing techniques in its normal 79operations: Design and Code Inspections. The intent of LTP 80is to support this by giving developers an ever growing 81set of tools to help identify any operational problems in 82their code that may be missed by human review. One of the 83toughest categories of problems to catch with inspection 84is that of interaction of features. With a continuously 85improving set of tests and tools, developers can get an 86indication of whether their changes may have broken some 87other functionality. 88 89There is no such thing as a perfect test base. It is only 90useful it if keeps up with new and changing functionality, 91and if it actually gets used. 92 932.3 Are you doing benchmarking? 94 95Not at this time. We are more interested in functional, 96regression, and stress testing the Linux kernel. Benchmarking 97may be workable to compare the performance among kernel 98versions. 99 1002.4 Are you doing standards testing? 101 102No, we are leaving that to the Linux Standards Base (LSB). 103 See the Linux Standards Base [http://www.linuxbase.org/||web site] 104for more information. 105 1063 Structure 107 108The basic building block of the test project is a test case 109that consists of a single action and a verification that 110the action worked. The result of the test case is usually 111restricted to PASS/FAIL. 112 113A test program is a runnable program that contains one or 114more test cases. Test programs often understand command 115line options which alter their behavior. The options could 116determine the amount of memory tested, the location of temporary 117files, the type of network packet used, or any other useful 118parameter. 119 120Test tags are used to pair a unique identifier with a test 121program and a set of command line options. Test tags are 122the basis for test suites. 123 1244 Writing Tests 125 126Writing a test case is a lot easier than most people think. 127 Any code that you write to examine how a part of the kernel 128works can be adapted into a test case. All that is needed 129is a way to report the result of the action to the rest 130of the world. There are several ways of doing this, some 131more involved than others. 132 1334.1 Exit Style Tests 134 135Probably the simplest way of reporting the results of a test 136case is the exit status of your program. If your test program 137encounters unexpected or incorrect results, exit the test 138program with a non-zero exit status, i.e. exit(1). Conversely, 139if your program completes as expected, return a zero exit 140status, i.e. exit(0). Any test driver should be able to 141handle this type of error reporting. If a test program has 142multiple test cases you won't know which test case failed, 143but you will know the program that failed. 144 1454.2 Formatted Output Tests 146 147The next easiest way of reporting the results is to write 148the results of each test case to standard output. This allows 149for the testing results to be more understandable to both 150the tester and the analysis tools. When the results are 151written in a standard way, tools can be used to analyze 152the results. 153 1545 Testing Tools 155 156The Linux Test Project has not yet decided on a "final" test 157harness. We have provided a simple solution with ltp-pan to 158make due until a complete solution has been found/created 159that compliments the Linux kernel development process. 160Several people have said we should use such and such a test 161harness. Until we find we need a large complex test harness, 162we will apply the KISS concept. 163 1645.1 Ltp-pan 165 166Ltp-pan is a simple test driver with the ability to keep track 167of orphaned processes and capture test output. It works 168by reading a list of test tags and command lines and runs 169them. By default ltp-pan will select a command randomly from 170the list of test tags, wait for it to finish. Through command 171line options you can run through the entire list sequentially, 172run n tests, keep n test running at all times, and buffer 173test output. Ltp-pan can be nested to create very complex test 174environments. 175 176Ltp-pan uses an active file, also called a zoo file to keep track 177of which tests are currently running. This file holds the 178pid, tag, and a portion of the command line. When you start 179ltp-pan it becomes a test tag in itself, thus it requires a 180name for itself. Ltp-pan updates the active file to show which 181test tags are currently running. When a test tag exits, 182ltp-pan will overwrite the first character with a '#'. The active 183file can be shared between multiple instances of ltp-pan so 184you know which tests were running when the system crashes 185by looking at one file. 186 187A ltp-pan file contains a list of test tags for ltp-pan to run. The 188format of a ltp-pan file is as follows: 189 190testtag testprogram -o one -p two other command line options 191 192# This is a comment. It is a good idea to describe the test 193 194# tags in your ltp-pan file. Tests programs can have different 195 196# behaviors depending on the command line options so it is 197 198# helpful to describe what each test tag is meant to verify 199or # provoke. 200 201# Some more test cases 202 203mm01 mmap001 -m 10000 204 205# 40 Mb mmap() test. 206 207# Creates a 10000 page mmap, touches all of the map, sync's 208 209# it, and munmap()s it. 210 211mm03 mmap001 -i 0 -I 1 -m 100 212 213# repetitive mmapping test. 214 215# Creates a one page map repetitively for one minute. 216 217dup02 dup02 218 219# Negative test for dup(2) with bad fd 220 221kill09 kill09 222 223# Basic test for kill(2) 224 225fs-suite01 ltp-pan -e -a fs-suite01.zoo -n fs-suite01 -f runtest/fs 226 227# run the entire set of file system tests 228 229The test tags are simple identifiers, no spaces are allowed. 230The test of the line is the program to run, which is done 231using execvp(3). Lines starting with '#' are comments and 232ignored by ltp-pan. It is a good practice to include descriptions 233with your test tags so you can have a reminder what a certain 234obscure test tag tries to do. 235 2365.1.1 Examples 237 238The most basic way to run ltp-pan is by passing the test program 239and parameters on the command line. This will run the single 240program once and wrap the output. 241 242$ ltp-pan -a ltp.zoo -n tutor sleep 4 243 244<<<test_start>>> 245 246tag=cmdln stime=971450564 247 248cmdline="sleep 4" 249 250contacts="" 251 252analysis=exit 253 254initiation_status="ok" 255 256<<<test_output>>> 257 258<<<execution_status>>> 259 260duration=103341903 termination_type=exited termination_id=0 261corefile=no cutime=0 cstime=0 262 263<<<test_end>>> 264 265$ cat ltp.zoo 266 267#9357,tutor,pan/ltp-pan -a ltp.zoo -n tutor sleep 4 268 269#9358,cmdln,sleep 4 270 271$ 272 273How it works 274 275This example shows the two parameters that are always required 276by ltp-pan, the active file and a test tag for ltp-pan. The "sleep 2774" on the end of the command line is a test program and 278parameters that ltp-pan should run. This test is given the tag 279"cmdln." Ltp-pan will run one test randomly, which ends 280up being cmdln since it is the only test that we told ltp-pan 281about. 282 283In the active file, ltp.zoo, ltp-pan writes the pid, test tag, 284and part of the command line for the currently running tests. 285The command lines are truncated so each line will fit on 286an 80 column display. When a test tag finishes, ltp-pan will 287place a '#' at the beginning of the line to mark it as available. 288Here you can see that cmdln and tutor, the name we gave 289ltp-pan, ran to completion. If the computer hangs, you can read 290this file to see which test programs were running. 291 292We have run one test once. Let's do something a little more 293exciting. Let's run one test several times, at the same 294time. 295 296$ ltp-pan -a ltp.zoo -n tutor -x 3 -s 3 -O /tmp sleep 1 297 298<<<test_start>>> 299 300tag=cmdln stime=971465653 301 302cmdline="sleep 1" 303 304contacts="" 305 306analysis=exit 307 308initiation_status="ok" 309 310<<<test_output>>> 311 312 313 314<<<execution_status>>> 315 316duration=103326814 termination_type=exited termination_id=0 317corefile=no 318 319cutime=1 cstime=0 320 321<<<test_end>>> 322 323<<<test_start>>> 324 325tag=cmdln stime=971465653 326 327cmdline="sleep 1" 328 329contacts="" 330 331analysis=exit 332 333initiation_status="ok" 334 335<<<test_output>>> 336 337 338 339<<<execution_status>>> 340 341duration=103326814 termination_type=exited termination_id=0 342corefile=no 343 344cutime=0 cstime=1 345 346<<<test_end>>> 347 348<<<test_start>>> 349 350tag=cmdln stime=971465653 351 352cmdline="sleep 1" 353 354contacts="" 355 356analysis=exit 357 358initiation_status="ok" 359 360<<<test_output>>> 361 362 363 364<<<execution_status>>> 365 366duration=103326814 termination_type=exited termination_id=0 367corefile=no 368 369cutime=0 cstime=0 370 371<<<test_end>>> 372 373How it works 374 375In this example we run another fake test from the command 376line, but we run it three times (-s 3) and keep three test 377tags active at the same time (-x 3). The -O parameter is 378a directory where temporary files can be created to buffer 379the output of each test tag. You can see in the output that 380cmdln ran three times. If the -O option were omitted, your 381test output would be mixed, making it almost worthless. 382 383* Using a ltp-pan file to run multiple tests 384 385* Nesting ltp-pan 386 387For more information on ltp-pan see the man page doc/man1/ltp-pan.1. 388 3895.2 Scanner 390 391Ltp-scanner is a results analysis tool that understands the rts 392style output which ltp-pan generates by default. It will produce 393a table summarizing which tests passed and which failed. 394 3956 To Do 396 397There are a lot of things that still need to be done to make 398this a complete kernel testing system. The following sections 399will discuss some of the to do items in detail. 400 4016.1 Configuration Analysis 402 403While the number of configuration options for the Linux kernel 404is seen as a strength to developers and users alike, it 405is a curse to testers. To create a powerful automated testing 406system, we need to be able to determine what the configuration 407on the booted box is and then determine which tests should 408be run on that box. 409 410The Linux kernel has hundreds of configuration options that 411can be set to compile the kernel. There are more options 412that can be set when you boot the kernel and while it is 413running. There are also many patches that can be applied 414to the kernel to add functionality or change behavior. 415 4166.2 Result Comparison 417 418A lot of testing will be done in the life of the Linux Test 419Project. Keeping track of the results from all the testing 420will require some infrastructure. It would be nice to take 421that output from a test machine, feed it to a program and 422receive a list of items that broke since the last run on 423that machine, or were fixed, or work on another test machine 424but not on this one. 425 4267 Contact information and updates 427 428URL: http://ltp.sourceforge.net/ 429 430mailing list: ltp@lists.linux.it 431 432list archive: http://lists.linux.it/pipermail/ltp/ 433 434Questions and comments should be sent to the LTP mailing 435list at ltp@lists.linux.it. To subscribe, please go to 436http://lists.linux.it/listinfo/ltp. 437 438The source is also available via CVS. See the web site for 439a web interface and check out instructions. 440 4418 Glossary 442 443Test IEEE/ANSI([footnote] Kit, Edward, Software Testing in the Real World: 444Improving the Process. P. 82. ACM Press, 1995.) : 445 (i) An activity in which a system or component is executed 446under specified conditions, the results are observed or 447record, and an evaluation is made of some aspect of the 448system or component. 449 (ii) A set of one or more test cases. 450 451Test Case A test assertion with a single result that 452 is being verified. This allows designations such as PASS 453 or FAIL to be applied to a single bit of functionality. 454 A single test case may be one of many test cases for 455 testing the complete functionality of a system. 456 IEEE/ANSI: 457 (i)A set of test inputs, execution conditions, and expected 458 results developed for a particular objective. 459 (ii) The smallest entity that is always executed as a unit, 460 from beginning to end. 461 462Test Driver A program that handles the execution of 463 test programs. It is responsible for starting the test 464 programs, capturing their output, and recording their 465 results. Ltp-pan is an example of a test driver. 466 467Test Framework A mechanism for organizing a group of 468 tests. Frameworks may have complex or very simple API's, 469 drivers and result logging mechanisms. Examples of frameworks 470 are TETware and DejaGnu. 471 472Test Harness A Test harness is the mechanism that connects 473 a test program to a test framework. It may be a specification 474 of exit codes, or a set of libraries for formatting messages 475 and determining exit codes. In TETware, the tet_result() 476 API is the test harness. 477 478Test Program A single invokable program. A test program 479 can contain one or more test cases. The test harness's 480 API allows for reporting/analysis of the individual test 481 cases. 482 483Test Suite A collection of tests programs, assertions, 484 cases grouped together under a framework. 485 486Test Tag An identifier that corresponds to a command 487 line which runs a test. The tag is a single word that 488 matches a test program with a set of command line arguments. 489