• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1================
2 lit TODO Items
3================
4
5Infrastructure
6==============
7
81. Change to always load suites, then resolve command line arguments?
9
10  Currently we expect each input argument to be a path on disk; we do a
11  recursive search to find the test suite for each item, but then we only do a
12  local search based at the input path to find tests. Additionally, for any path
13  that matches a file on disk we explicitly construct a test instance (bypassing
14  the formats on discovery implementation).
15
16  This has a couple problems:
17
18  * The test format doesn't have control over the test instances that result
19    from file paths.
20
21  * It isn't possible to specify virtual tests as inputs. For example, it is not
22    possible to specify an individual subtest to run with the googletest format.
23
24  * The test format doesn't have full control over the discovery of tests in
25    subdirectories.
26
27  Instead, we should move to a model whereby first all of the input specifiers
28  are resolved to test suites, and then the resolution of the input specifier is
29  delegated to each test suite. This could take a couple forms:
30
31  * We could resolve to test suites, then fully load each test suite, then have
32    a fixed process to map input specifiers to tests in the test suite
33    (presumably based on path-in-suite derivations). This has the benefit of
34    being consistent across all test formats, but the downside of requiring
35    loading the entire test suite.
36
37  * We could delegate all of the resolution of specifiers to the test
38    suite. This would allow formats that anticipate large test suites to manage
39    their own resolution for better performance. We could provide a default
40    resolution strategy that was similar to what we do now (start at subpaths
41    for directories, but allow the test format control over what happens for
42    individual tests).
43
442. Consider move to identifying all tests by path-to-test-suite and then path to
45   subtest, and don't use test suite names.
46
47  Currently the test suite name is presented as part of test names, but it has
48  no other useful function, and it is something that has to be skipped over to
49  cut-and-paste a name to subsequently use to rerun a test. If we just
50  represented each test suite by the path to its suite, then it would allow more
51  easy cut-and-paste of the test output lines. This has the downside that the
52  lines might get rather long.
53
543. Allow 'lit' driver to cooperate with test formats and suites to add options
55   (or at least sanitize accepted params).
56
57  We have started to use the --params method more and more extensively, and it is
58  cumbersome and error prone. Additionally, there are currently various options
59  ``lit`` honors that should more correctly be specified as belonging to the
60  ShTest test format.
61
62  It would be really nice if we could allow test formats and test suites to add
63  their own options to be parsed. The difficulty here, of course, is that we
64  don't know what test formats or test suites are in use until we have parsed the
65  input specifiers. For test formats we could ostensibly require all the possible
66  formats to be registered in order to have options, but for test suites we would
67  certainly have to load the suite before we can query it for what options it
68  understands.
69
70  That leaves us with the following options:
71
72  * Currently we could almost get away with parsing the input specifiers without
73    having done option parsing first (the exception is ``--config-prefix``) but
74    that isn't a very extensible design.
75
76  * We could make a distinction in the command line syntax for test format and
77    test suite options. For example, we could require something like::
78
79      lit -j 1 -sv input-specifier -- --some-format-option
80
81    which would be relatively easy to implement with optparser (I think).
82
83  * We could allow fully interspersed arguments by first extracting the options
84    lit knows about and parsing them, then dispatching the remainder to the
85    formats. This seems the most convenient for users, who are unlikely to care
86    about (or even be aware of) the distinction between the generic lit
87    infrastructure and format or suite specific options.
88
894. Eliminate duplicate execution models for ShTest tests.
90
91  Currently, the ShTest format uses tests written with shell-script like syntax,
92  and executes them in one of two ways. The first way is by converting them into
93  a bash script and literally executing externally them using bash. The second
94  way is through the use of an internal shell parser and shell execution code
95  (built on the subprocess module). The external execution mode is used on most
96  Unix systems that have bash, the internal execution mode is used on Windows.
97
98  Having two ways to do the same thing is error prone and leads to unnecessary
99  complexity in the testing environment. Additionally, because the mode that
100  converts scripts to bash doesn't try and validate the syntax, it is possible
101  to write tests that use bash shell features unsupported by the internal
102  shell. Such tests won't work on Windows but this may not be obvious to the
103  developer writing the test.
104
105  Another limitation is that when executing the scripts externally, the ShTest
106  format has no idea which commands fail, or what output comes from which
107  commands, so this limits how convenient the output of ShTest failures can be
108  and limits other features (for example, knowing what temporary files were
109  written).
110
111  We should eliminate having two ways of executing the same tests to reduce
112  platform differences and make it easier to develop new features in the ShTest
113  module. This is currently blocked on:
114
115  * The external execution mode is faster in some situations, because it avoids
116    being bottlenecked on the GIL. This can hopefully be obviated simply by
117    using --use-processes.
118
119  * Some tests in LLVM/Clang are explicitly disabled with the internal shell
120    (because they use features specific to bash). We would need to rewrite these
121    tests, or add additional features to the internal shell handling to allow
122    them to pass.
123
1245. Consider changing core to support setup vs. execute distinction.
125
126  Many of the existing test formats are cleanly divided into two phases, once
127  parses the test format and extracts XFAIL and REQUIRES information, etc., and
128  the other code actually executes the test.
129
130  We could make this distinction part of the core infrastructure and that would
131  enable a couple things:
132
133  * The REQUIREs handling could be lifted to the core, which is nice.
134
135  * This would provide a clear place to insert subtest support, because the
136    setup phase could be responsible for providing subtests back to the
137    core. That would provide part of the infrastructure to parallelize them, for
138    example, and would probably interact well with other possible features like
139    parameterized tests.
140
141  * This affords a clean implementation of --no-execute.
142
143  * One possible downside could be for test formats that cannot determine their
144    subtests without having executed the test. Supporting such formats would
145    either force the test to actually be executed in the setup stage (which
146    might be ok, as long as the API was explicitly phrased to support that), or
147    would mean we are forced into supporting subtests as return values from the
148    execute phase.
149
150  Any format can just keep all of its code in execute, presumably, so the only
151  cost of implementing this is its impact on the API and futures changes.
152
153
154Miscellaneous
155=============
156
157* Move temp directory name into local test config.
158
159* Support valgrind in all configs, and LLVM style valgrind.
160
161* Support ulimit.
162
163* Create an explicit test suite object (instead of using the top-level
164  TestingConfig object).
165
166* Introduce a wrapper class that has a ``subprocess.Popen`` like interface
167  but also supports killing the process and all its children and use this for
168  running tests.  This would allow us to implement platform specific methods
169  for killing a process's children which is needed for a per test timeout. On
170  POSIX platforms we can use process groups and on Windows we can probably use
171  job objects. This would not only allow us to remove the dependency on the
172  ``psutil`` module but would also be more reliable as the
173  ``lit.util.killProcessAndChildren()`` function which is currently used is
174  potentially racey (e.g. it might not kill a fork bomb completely).
175