1<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" 2"http://www.w3.org/TR/html4/strict.dtd"> 3 4<html> 5 <head> 6 <meta name="generator" content= 7 "HTML Tidy for Linux/x86 (vers 1st March 2002), see www.w3.org"> 8 <!--tidy options: -i -wrap 78 --> 9 <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> 10 11 <title>A testing system for B2</title> 12<style type="text/css"> 13 hr { color: black } 14 p.revision { text-align: right; font-style: italic } 15 pre.code { margin-left: 2em } 16 pre.example { margin-left: 2em; border: solid black thin } 17 pre.output { margin-left: 2em } 18 img.banner { border: 0; float: left } 19 h1 { text-align: right } 20 br.clear { clear: left } 21 div.attention { color: red } 22 23</style> 24 </head> 25 26 <body> 27 <p><a href="../../../../index.htm"><img class="banner" height="86" width= 28 "277" alt="C++ Boost" src="../../../../boost.png"></a></p> 29 30 <h1>A testing system for B2<br class="clear"> 31 </h1> 32 <hr> 33 34 <dl class="page-index"> 35 <dt><a href="#sec-intro">Introduction for users</a></dt> 36 37 <dd> 38 <dl class="page-index"> 39 <dt><a href="#sec-command-line-options">Command line options</a></dt> 40 </dl> 41 </dd> 42 43 <dt><a href="#sec-developers">Introduction for developers</a></dt> 44 45 <dd> 46 <dl class="page-index"> 47 <dt><a href="#sec-intro-changing">Changing the working 48 directory</a></dt> 49 50 <dt><a href="#sec-intro-examining">Examining the working directory and 51 changing it</a></dt> 52 53 <dt><a href="#sec-intro-results">Test result</a></dt> 54 </dl> 55 </dd> 56 57 <dt><a href="#sec-reference">Reference documentation</a></dt> 58 59 <dd> 60 <dl class="page-index"> 61 <dt><a href="#method-__init__">Method __init__</a></dt> 62 63 <dt><a href="#method-set_tree">Method <tt>set_tree</tt></a></dt> 64 65 <dt><a href="#method-write">Method <tt>write</tt></a></dt> 66 67 <dt><a href="#method-copy">Method <tt>copy</tt></a></dt> 68 69 <dt><a href="#method-touch">Method <tt>touch</tt></a></dt> 70 71 <dt><a href="#method-run_build_system">Method 72 <tt>run_build_system</tt></a></dt> 73 74 <dt><a href="#method-read">Method <tt>read</tt></a></dt> 75 76 <dt><a href="#method-read_and_strip">Method 77 <tt>read_and_strip</tt></a></dt> 78 79 <dt><a href="#methods-expectations">Methods for declaring 80 expectations</a></dt> 81 82 <dt><a href="#methods-ignoring">Methods for ignoring 83 changes</a></dt> 84 85 <dt><a href="#methods-result">Methods for explicitly specifying 86 results</a></dt> 87 88 <dt><a href="#class-list">Helper class <tt>List</tt></a></dt> 89 </dl> 90 </dd> 91 </dl> 92 <hr> 93 94 <h2><a name="sec-intro">Introduction for users</a></h2> 95 96 <p>The testing system for B2 is a small set of Python modules and 97 scripts for automatically testing user-obversable behaviour. It uses 98 components from testing systems of <a href="http://www.scons.org">Scons</a> 99 and <a href="http://subversion.tigris.org">Subversion</a>, together with 100 some additional functionality.</p> 101 102 <p>To run the tests you need to:</p> 103 104 <ol> 105 <li>Get the source tree of B2 (located at <tt>tools/build</tt> 106 in Boost)</li> 107 108 <li>Have <a href="http://www.python.org">Python</a> installed. Version 109 2.1 is known to work.</li> 110 111 <li>Build Boost.Jam. See <a href= 112 "../engine/index.html">$boost_build_root/engine/index.html</a> for 113 instructions.</li> 114 115 <li>Configure at least one toolset. You can edit <tt>site-config.jam</tt> 116 or <tt>user-config.jam</tt> to add new toolsets. Or you can create file 117 <tt>test-config.jam</tt> in <tt>$boost_build_root/test</tt> directory. In 118 this case, <tt>site-config.jam</tt> and <tt>user-config.jam</tt> will be 119 ignored for testing.</li> 120 121 <li>Make sure that in the configuration jam file(s) that you use you generate 122 no console output, ie. with the Boost Build 'ECHO' rule. Such console output 123 in the configuration jam file(s) will cause a number of tests to automatically 124 fail which would otherwise succeed.</li> 125 </ol> 126 127 <p>When all is set, you can run all the tests using the <tt>test_all.py</tt> 128 script or you can run a specific test by starting its Python script 129 directly.</p> 130 131 <p>Examples:</p> 132 133<pre class="code"> 134python test_all.py 135python generators_test.py 136</pre> 137 138 <p>If everything is OK, you will see a list of passed tests. Otherwise, a 139 failure will be reported.</p> 140 141 <h3><a name="sec-command-line-options">Command line options</a></h3> 142 143 <p>Test scripts will use the toolset you configured to be the default or 144 you can specify a specific one on the command line:</p> 145 146<pre class="code"> 147python test_all.py borland 148python generators_test.py msvc-7.1 149</pre> 150 151 <p>Other test script flags you can specify on the command line are:</p> 152 153 <ul> 154 <li><tt>--default-bjam</tt> -- By default the test system will use the 155 Boost Jam executable found built in its default development build 156 location. This option makes it use the default one available on your 157 system, i.e. the one found in the system path.</li> 158 159 <li><tt>--preserve</tt> -- In case of a failed test its working 160 directory will be copied to the "failed_test" directory under the 161 current directory.</li> 162 163 <li><tt>--verbose</tt> -- Makes the test system and the run build system 164 display additional output. Note though that this may cause tests that 165 check the build system output to fail.</li> 166 </ul> 167 168 <h2><a name="sec-developers">Introduction for developers</a></h2> 169 170 <p>It is suggested that every new functionality come together with tests, 171 and that bugfixes are accompanied by tests. There's no need to say that 172 tests are good, but two points are extremely important:</p> 173 174 <ul> 175 <li>For an interpreted language like Jam, without any static checks, 176 testing is simply the only sefeguard we can have.</li> 177 178 <li>Good tests allow us to change internal design much more safely, and we 179 have not gotten everything nailed down yet.</li> 180 </ul> 181 182 <p>Adding a new test is simple:</p> 183 184 <ol> 185 <li>Go to <tt>$boost_build_root/test/test_all.py</tt> and add new test 186 name to the list at the end of the file. Suppose the test name is "hello". 187 </li> 188 189 <li>Add a new python module, in this example "hello.py", to do the actual 190 testing.</li> 191 </ol> 192 193 <p>The module, in general will perform these basic actions:</p> 194 195 <ol> 196 <li>Set up the initial working directory state</li> 197 198 <li> 199 Run the build system and check the results: 200 201 <ol> 202 <li>generated output,</li> 203 204 <li>changes made to the working directory,</li> 205 206 <li>new content of the working directory.</li> 207 </ol> 208 </li> 209 210 <li>Add, remove or touch files or change their content and then repeat 211 the previous step until satisfied.</li> 212 213 <li>Clean up</li> 214 </ol> 215 216 <p>The "hello.py" module might contain:</p> 217<pre class="example"> 218from BoostBuild import List 219 220# Create a temporary working directory 221t = BoostBuild.Tester() 222 223# Create the needed files 224t.write("jamroot.jam", "") 225t.write("jamfile.jam", """ 226exe hello : hello.cpp ; 227""") 228t.write("hello.cpp", """ 229int main() 230{ 231 return 0; 232} 233 234""") 235 236t.run_build_system() 237 238# First, create a list of three pathnames. 239file_list = List("bin/$toolset/debug/") * List("hello.exe hello.obj") 240# Second, assert that those files were added as result of the last build system invocation. 241t.expect_addition(file_list) 242 243# Invoke the build system once again. 244t.run_build_system("clean") 245# Check if the files added previously were removed. 246t.expect_removal(file_list) 247 248# Remove temporary directories 249t.cleanup() 250</pre> 251 252 <p>The <tt>test</tt> directory contains a file "template.py" which can be 253 used as a start for your own tests.</p> 254 255 <p>Overview of the most important methods of class <tt>Tester</tt> follows. 256 </p> 257 258 <h3><a name="sec-intro-changing">Changing the working directory</a></h3> 259 260 <p>The class <tt>Tester</tt> creates a temporary directory in its 261 constructor and changes to that directory. It can be modified by calling 262 these methods:</p> 263 264 <ul> 265 <li><tt>set_tree</tt> -- sets the content of the working directory to be 266 equal to the content of the specified directory. This method is 267 preferable when directory tree for testing is large.</li> 268 269 <li><tt>write</tt> -- sets the content of file in a working directory. 270 This is optimal if you want to create a directory tree with 3-4 small 271 files.</li> 272 273 <li><tt>touch</tt> -- changes the modification times of a file</li> 274 </ul> 275 276 <h3><a name="sec-intro-examining">Examining the working directory and 277 changing it</a></h3> 278 279 <p>The method <tt>read</tt>, inherited from the <tt>TestCmd</tt> class, can 280 be used to read any file in the working directory and check its content. 281 <tt>Tester</tt> adds another method for tracking changes. Whenever the build 282 system is run (using <a href="#method-run_build_system"><tt>run_build_system 283 </tt></a>), the working dir state before and after running is recorded. In 284 addition, difference between the two states -- i.e. lists of files that were 285 added, removed, modified or touched -- are stored in two member variables - 286 <tt>tree_difference</tt> and <tt>unexpected_difference</tt>.</p> 287 288 <p>After than, the test author may specify that some change is expected, for 289 example, by calling <tt>expect_addition("foo")</tt>. This call will check if 290 the file was indeed added, and if so, will remove its name from the list of 291 added files in <tt>unexpected_difference</tt>. Likewise, it is possible to 292 specify that some changes are not interesting, for example a call to 293 <tt>ignore("*.obj")</tt> will just remove every file with the ".obj" 294 extension from <tt>unexpected_difference</tt>.</p> 295 296 <p>When test has finished with expectations and ignoring, the member 297 <tt>unexpected_difference</tt> will contain the list of all changes not yet 298 accounted for. It is possible to assure that this list is empty by calling 299 the <tt>expect_nothing_more</tt> member function.</p> 300 301 <h3><a name="sec-intro-results">Test result</a></h3> 302 303 <p>Any of the <tt>expect*</tt> methods below will fail the test if the 304 expectation is not met. It is also possible to perform manually arbitrary 305 test and explicitly cause the test to either pass or fail. Ordinary 306 filesystem functions can be used to work with the directory tree. Methods 307 <tt>pass_test</tt> and <tt>fail_test</tt> are used to explicitly give the 308 test outcome.</p> 309 310 <p>Typically, after test termination, the working directory is erased. See 311 the <a href="#sec-command-line-options">"--preserve" command line option</a> 312 for information on how to preserve the working directory content for failed 313 tests for debugging purposes.</p> 314 315 <h2 id="sec-reference">Reference documentation</h2> 316 317 <p>The test system is composed of class <tt>Tester</tt>, derived form 318 <tt>TestCmd.TestCmd</tt>, and helper class <tt>List</tt>. <tt>Tester</tt> 319 and <tt>List</tt> methods are described below.</p> 320 321 <p>The documentation frequently refers to <tt>filename</tt>. In all cases, 322 files are specified in unix style: a sequence of components, separated by 323 "/". This is true on all platforms. In some contexts a list of files is 324 allowed. In those cases any object with a sequence interface is allowed.</p> 325 326 <h3><a name="method-__init__">Method <tt>__init__(self, arguments="", 327 executable="bjam", match=TestCmd.match_exact, boost_build_path=None, 328 translate_suffixes=True, pass_toolset=True, use_test_config=True, 329 ignore_toolset_requirements=True, workdir="", **keywords)</tt></a></h3> 330 331 <p><b>Optional arguments:</b></p> 332 333 <ul> 334 <li><tt>arguments</tt> 335 -- Arguments passed to the run executable.</li> 336 <li><tt>executable</tt> 337 -- Name of the executable to invoke.</li> 338 <li><tt>match</tt> 339 -- Function to use for compating actual and expected file contents. 340 </li> 341 <li><tt>boost_build_path</tt> 342 -- Boost build path to be passed to the run executable.</li> 343 <li><tt>translate_suffixes</tt> 344 -- Whether to update suffixes on the the file names passed from the 345 test script so they match those actually created by the current 346 toolset. For example, static library files are specified by using 347 the .lib suffix but when the 'gcc' toolset is used it actually 348 creates them using the .a suffix.</li> 349 <li><tt>pass_toolset</tt> 350 -- Whether the test system should pass the specified toolset to the 351 run executable.</li> 352 <li><tt>use_test_config</tt> 353 -- Whether the test system should tell the run executable to read in 354 the test_config.jam configuration file.</li> 355 <li><tt>ignore_toolset_requirements</tt> 356 -- Whether the test system should tell the run executable to ignore 357 toolset requirements.</li> 358 <li><tt>workdir</tt> 359 -- Indicates an absolute directory where the test will be run from. 360 </li> 361 </ul> 362 363 <p><b>Optional arguments inherited from the base class:</b></p> 364 365 <ul> 366 <li><tt>description</tt> 367 -- Test description string displayed in case of a failed test.</li> 368 <li><tt>subdir</tt> 369 -- List of subdirectories to automatically create under the working 370 directory. Each subdirectory needs to be specified separately 371 parent coming before its child.</li> 372 <li><tt>verbose</tt> 373 -- Flag that may be used to enable more verbose test system output. 374 Note that it does not also enable more verbose build system output 375 like the <a href="#sec-command-line-options">"--verbose" command 376 line option</a> does.</li> 377 </ul> 378 379 <p><b>Effects:</b></p> 380 381 <ol> 382 <li>Remembers the current working directory in member 383 <tt>original_workdir</tt>.</li> 384 385 <li>Determines the location of the executable (<code>bjam</code> by 386 default) and build system files, assuming that the current directory is 387 <tt>tools/build/test</tt>. Formulates jam invocation command, which 388 will include explicit setting for the <tt>BOOST_BUILD_PATH</tt> variable 389 and arguments passed to this methods, if any. This command will be used 390 by subsequent invocation of <a href="#method-run_build_system"><tt> 391 run_build_system</tt></a>. Finally, initializes the base class.</li> 392 393 <li>Changes the current working directory to the temporary working 394 directory created by the base constructor.</li> 395 396 <li>If you want to run a test in an existing directory, pass it as 397 <tt>workdir</tt>.</li> 398 399 <li> Most parameters passed to this constructor function may be overruled 400 for each specific test system run using <a href= 401 "#method-run_build_system"><tt>run_build_system</tt></a> parameters. 402 </ol> 403 404 <h3><a name="method-set_tree">Method <tt>set_tree(self, 405 tree_location)</tt></a></h3> 406 407 <p><b>Effects:</b></p> 408 409 <p>Replaces the content of the current working directory with the content 410 of directory at <tt>tree_location</tt>. If <tt>tree_location</tt> is not 411 absolute pathname, it will be treated as relative to 412 <tt>self.original_workdir</tt>. This methods also explicitly makes the 413 copied files writeable.</p> 414 415 <h3><a name="method-write">Method <tt>write(self, name, 416 content)</tt></a></h3> 417 418 <p><b>Effects:</b></p> 419 420 <p>Writes the specified content to the file given by <tt>name</tt> under 421 the temporary working directory. If the file already exists, it is 422 overwritten. Any required directories are automatically created.</p> 423 424 <h3><a name="method-copy">Method <tt>copy(self, src, dst)</tt></a></h3> 425 426 <p><b>Effects:</b></p> 427 428 <p>Equvivalent to <tt>self.write(self.read(src), dst)</tt>.</p> 429 430 <h3><a name="method-touch">Method <tt>touch(self, names)</tt></a></h3> 431 432 <p><b>Effects:</b></p> 433 434 <p>Sets the access and modification times for all files in <tt>names</tt> to 435 the current time. All the elements in <tt>names</tt> should be relative 436 paths.</p> 437 438 <h3><a name="method-run_build_system">Method <tt>run_build_system(self, 439 extra_args="", subdir="", stdout=None, stderr="", status=0, match=None, 440 pass_toolset=None, use_test_config=None, ignore_toolset_requirements=None, 441 expected_duration=None, **kw)</tt></a></h3> 442 443 <p><b>Effects:</b></p> 444 445 <ol> 446 <li>Stores the state of the working directory in 447 <tt>self.previous_tree</tt>.</li> 448 449 <li>Changes to <tt>subdir</tt>, if it is specified. It is relative to 450 the <tt>original_workdir</tt> or the workdir specified in 451 <tt>__init</tt>.</li> 452 453 <li>Invokes the <tt>bjam</tt> executable, passing <tt>extra_args</tt> 454 to it. The binary should be located under 455 <tt><test_invocation_dir>/../jam/src/bin.<platform></tt>. 456 This is to make sure tests use the version of jam build from CVS.</li> 457 458 <li>Compares the stdout, stderr and exit status of build system 459 invocation with values to appropriate parameters, if they are not 460 <tt>None</tt>. If any difference is found, the test fails.</li> 461 462 <li>If the <tt>expected_duration</tt> parameter is specified then it 463 represents the maximal allowed time in seconds for the test to run. The 464 test will be marked as failed if its duration is greater than the given 465 <tt>expected_duration</tt> parameter value.</li> 466 467 <li>Stores the new state of the working directory in <tt>self.tree</tt>. 468 Computes the difference between previous and current trees and stores them 469 in variables <tt>self.tree_difference</tt> and 470 <tt>self.unexpected_difference</tt>. Both variables are instances of class 471 <tt>tree.Trees_different</tt>, which have four attributes: 472 <tt>added_files</tt>, <tt>removed_files</tt>, <tt>modified_files</tt> and 473 <tt>touched_files</tt>. Each is a list of strings.</p></li> 474 </ol> 475 476 <h3><a name="method-read">Method <tt>read(self, name)</tt></a></h3> 477 478 <p><b>Effects:</b></p> 479 480 <p>Read the specified file and returns it content. Raises an exception is 481 the file is absent.</p> 482 483 <h3><a name="method-read_and_strip">Method <tt>read_and_strip(self, name) 484 </tt></a></h3> 485 486 <p><b>Effects:</b></p> 487 488 <p>Read the specified file and returns it content, after removing trailing 489 whitespace from every line. Raises an exception is the file is absent.</p> 490 491 <p><b>Rationale:</b></p> 492 493 <p>Although this method is questionable, there are a lot of cases when jam 494 or shells it uses insert spaces. It seems that introducing this method is 495 much simpler than dealing with all those cases.</p> 496 497 <h3><a name="methods-expectations">Methods for declaring expectations</a> 498 </h3> 499 500 <p>Accordingly to the number of changes kinds that are detected, there are 501 four methods that specify that test author expects a specific change to 502 occur. They check <tt>self.unexpected_difference</tt>, and if the change is 503 present there, it is removed. Otherwise, test fails.</p> 504 505 <p>Each method accepts a list of names. Those names use <tt>/</tt> path 506 separator on all systems. Additionally, the test system translates suffixes 507 appropriately. For the test to be portable, suffixes should use Windows 508 convention: <tt>exe</tt> for executables, <tt>dll</tt> for dynamic libraries 509 and <tt>lib</tt> for static libraries. Lastly, the string "$toolset" in file 510 names is replaced by the name of tested toolset.</p> 511 512 <p><b>Note:</b> The <tt>List</tt> helper class might be useful to create 513 lists of names.</p> 514 515 <p><b>Note:</b> The file content can be examined using the 516 <tt>TestCmd.read</tt> function.</p> 517 518 <p>The members are:</p> 519 520 <ul> 521 <li>expect_addition</li> 522 <li>expect_removal</li> 523 <li>expect_modification</li> 524 <li>expect_nothing</li> 525 </ul> 526 527 <p>Note that <tt>expect_modification</tt> is used to check that a either 528 file content or timestamp has changed. The rationale is that some compilers 529 change content even if sources does not change, and it's easier to have a 530 method which checks for both content and time changes.</p> 531 532 <p>There's also a member <tt>expect_nothing_more</tt>, which checks that all 533 the changes are either expected or ignored, in other words that 534 <tt>unexpected_difference</tt> is empty by now.</p> 535 536 <p>Lastly, there's a method to compare file content with expected content: 537 </p> 538 <tt>expect_content(self, name, content, exact=0)</tt> 539 540 <p>The method fails the test if the content of file identified by 'name' is 541 different from 'content'. If 'exact' is true, the file content is used 542 as-is, otherwise, two transformations are applied:</p> 543 544 <ul> 545 <li>The <tt>read_and_strip</tt> method is used to read the file, which 546 removes trailing whitespace</li> 547 548 <li>Each backslash in the file content is converted to forward slash.</li> 549 </ul> 550 551 <h3><a name="methods-ignoring">Methods for ignoring changes</a></h3> 552 553 <p>There are five methods which ignore changes made to the working tree. 554 They silently remove elements from <tt>self.unexpected_difference</tt>, and 555 don't generate error if element is not found. They accept shell style 556 wildcard.</p> 557 558 <p>The following methods correspond to four kinds of changes:</p> 559 560 <ul> 561 <li>ignore_addition(self, wildcard)</li> 562 <li>ignore_removal(self, wildcard)</li> 563 <li>ignore_modification(self, wildcard)</li> 564 <li>ignore_touch(self, wildcard)</li> 565 </ul> 566 567 <p>The method <tt>ignore(self, wildcard)</tt> ignores all the changes made 568 to files that match a wildcard.</p> 569 570 <h3><a name="methods-result">Methods for explicitly specifying results</a> 571 </h3> 572 573 <h4>Method <tt>pass_test(self, condition=1)</tt></h4> 574 575 <div class="attention"> 576 At this moment, the method should not be used. 577 </div> 578 579 <h4>Method <tt>fail_test(self, condition=1)</tt></h4> 580 581 <p><b>Effects:</b> Cause the test to fail if <tt>condition</tt> is true.</p> 582 583 <h3><a name="class-list">Helper class <tt>List</tt></a></h3> 584 The class has sequence interface and two additional methods. 585 586 <h4>Method <tt>__init__(self, string)</tt></h4> 587 588 <p><b>Effects:</b> Splits the string on unescaped spaces and tabs. The split 589 components can further be retrieved using standard sequence access.</p> 590 591 <h4>Method <tt>__mul__(self, other)</tt></h4> 592 593 <p><b>Effects:</b> Returns an <tt>List</tt> instance, which elements are all 594 possible concatenations of two string, first of which is from <tt>self</tt>, 595 and second of which is from <tt>other</tt>.</p> 596 597 <p>The class also defines <tt>__str__</tt> and <tt>__repr__</tt> methods. 598 Finally, there's <tt>__coerce__</tt> method which allows to convert strings 599 to instances of <tt>List</tt>.</p> 600 601 <p><b>Example:</b></p> 602<pre> 603 l = "a b" * List("c d") 604 for e in l: 605 print e 606</pre> 607 608 <p>will output:</p> 609<pre> 610 ac 611 ad 612 bc 613 bd 614 615</pre> 616 <hr> 617 <p class="revision">Last modified: May 02, 2008</p> 618 <p>© Copyright Vladimir Prus 2002, 2003, 2004, 2005.<br> 619 © Copyright Jurko Gospodnetic 2008.<br> 620 Distributed under the Boost Software License, Version 1.0. 621 (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)</p> 622 </body> 623</html> 624
