• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1=========================
2Driver Design & Internals
3=========================
4
5.. contents::
6   :local:
7
8Introduction
9============
10
11This document describes the Clang driver. The purpose of this document
12is to describe both the motivation and design goals for the driver, as
13well as details of the internal implementation.
14
15Features and Goals
16==================
17
18The Clang driver is intended to be a production quality compiler driver
19providing access to the Clang compiler and tools, with a command line
20interface which is compatible with the gcc driver.
21
22Although the driver is part of and driven by the Clang project, it is
23logically a separate tool which shares many of the same goals as Clang:
24
25.. contents:: Features
26   :local:
27
28GCC Compatibility
29-----------------
30
31The number one goal of the driver is to ease the adoption of Clang by
32allowing users to drop Clang into a build system which was designed to
33call GCC. Although this makes the driver much more complicated than
34might otherwise be necessary, we decided that being very compatible with
35the gcc command line interface was worth it in order to allow users to
36quickly test clang on their projects.
37
38Flexible
39--------
40
41The driver was designed to be flexible and easily accommodate new uses
42as we grow the clang and LLVM infrastructure. As one example, the driver
43can easily support the introduction of tools which have an integrated
44assembler; something we hope to add to LLVM in the future.
45
46Similarly, most of the driver functionality is kept in a library which
47can be used to build other tools which want to implement or accept a gcc
48like interface.
49
50Low Overhead
51------------
52
53The driver should have as little overhead as possible. In practice, we
54found that the gcc driver by itself incurred a small but meaningful
55overhead when compiling many small files. The driver doesn't do much
56work compared to a compilation, but we have tried to keep it as
57efficient as possible by following a few simple principles:
58
59-  Avoid memory allocation and string copying when possible.
60-  Don't parse arguments more than once.
61-  Provide a few simple interfaces for efficiently searching arguments.
62
63Simple
64------
65
66Finally, the driver was designed to be "as simple as possible", given
67the other goals. Notably, trying to be completely compatible with the
68gcc driver adds a significant amount of complexity. However, the design
69of the driver attempts to mitigate this complexity by dividing the
70process into a number of independent stages instead of a single
71monolithic task.
72
73Internal Design and Implementation
74==================================
75
76.. contents::
77   :local:
78   :depth: 1
79
80Internals Introduction
81----------------------
82
83In order to satisfy the stated goals, the driver was designed to
84completely subsume the functionality of the gcc executable; that is, the
85driver should not need to delegate to gcc to perform subtasks. On
86Darwin, this implies that the Clang driver also subsumes the gcc
87driver-driver, which is used to implement support for building universal
88images (binaries and object files). This also implies that the driver
89should be able to call the language specific compilers (e.g. cc1)
90directly, which means that it must have enough information to forward
91command line arguments to child processes correctly.
92
93Design Overview
94---------------
95
96The diagram below shows the significant components of the driver
97architecture and how they relate to one another. The orange components
98represent concrete data structures built by the driver, the green
99components indicate conceptually distinct stages which manipulate these
100data structures, and the blue components are important helper classes.
101
102.. image:: DriverArchitecture.png
103   :align: center
104   :alt: Driver Architecture Diagram
105
106Driver Stages
107-------------
108
109The driver functionality is conceptually divided into five stages:
110
111#. **Parse: Option Parsing**
112
113   The command line argument strings are decomposed into arguments
114   (``Arg`` instances). The driver expects to understand all available
115   options, although there is some facility for just passing certain
116   classes of options through (like ``-Wl,``).
117
118   Each argument corresponds to exactly one abstract ``Option``
119   definition, which describes how the option is parsed along with some
120   additional metadata. The Arg instances themselves are lightweight and
121   merely contain enough information for clients to determine which
122   option they correspond to and their values (if they have additional
123   parameters).
124
125   For example, a command line like "-Ifoo -I foo" would parse to two
126   Arg instances (a JoinedArg and a SeparateArg instance), but each
127   would refer to the same Option.
128
129   Options are lazily created in order to avoid populating all Option
130   classes when the driver is loaded. Most of the driver code only needs
131   to deal with options by their unique ID (e.g., ``options::OPT_I``),
132
133   Arg instances themselves do not generally store the values of
134   parameters. In many cases, this would simply result in creating
135   unnecessary string copies. Instead, Arg instances are always embedded
136   inside an ArgList structure, which contains the original vector of
137   argument strings. Each Arg itself only needs to contain an index into
138   this vector instead of storing its values directly.
139
140   The clang driver can dump the results of this stage using the
141   ``-###`` flag (which must precede any actual command
142   line arguments). For example:
143
144   .. code-block:: console
145
146      $ clang -### -Xarch_i386 -fomit-frame-pointer -Wa,-fast -Ifoo -I foo t.c
147      Option 0 - Name: "-Xarch_", Values: {"i386", "-fomit-frame-pointer"}
148      Option 1 - Name: "-Wa,", Values: {"-fast"}
149      Option 2 - Name: "-I", Values: {"foo"}
150      Option 3 - Name: "-I", Values: {"foo"}
151      Option 4 - Name: "<input>", Values: {"t.c"}
152
153   After this stage is complete the command line should be broken down
154   into well defined option objects with their appropriate parameters.
155   Subsequent stages should rarely, if ever, need to do any string
156   processing.
157
158#. **Pipeline: Compilation Action Construction**
159
160   Once the arguments are parsed, the tree of subprocess jobs needed for
161   the desired compilation sequence are constructed. This involves
162   determining the input files and their types, what work is to be done
163   on them (preprocess, compile, assemble, link, etc.), and constructing
164   a list of Action instances for each task. The result is a list of one
165   or more top-level actions, each of which generally corresponds to a
166   single output (for example, an object or linked executable).
167
168   The majority of Actions correspond to actual tasks, however there are
169   two special Actions. The first is InputAction, which simply serves to
170   adapt an input argument for use as an input to other Actions. The
171   second is BindArchAction, which conceptually alters the architecture
172   to be used for all of its input Actions.
173
174   The clang driver can dump the results of this stage using the
175   ``-ccc-print-phases`` flag. For example:
176
177   .. code-block:: console
178
179      $ clang -ccc-print-phases -x c t.c -x assembler t.s
180      0: input, "t.c", c
181      1: preprocessor, {0}, cpp-output
182      2: compiler, {1}, assembler
183      3: assembler, {2}, object
184      4: input, "t.s", assembler
185      5: assembler, {4}, object
186      6: linker, {3, 5}, image
187
188   Here the driver is constructing seven distinct actions, four to
189   compile the "t.c" input into an object file, two to assemble the
190   "t.s" input, and one to link them together.
191
192   A rather different compilation pipeline is shown here; in this
193   example there are two top level actions to compile the input files
194   into two separate object files, where each object file is built using
195   ``lipo`` to merge results built for two separate architectures.
196
197   .. code-block:: console
198
199      $ clang -ccc-print-phases -c -arch i386 -arch x86_64 t0.c t1.c
200      0: input, "t0.c", c
201      1: preprocessor, {0}, cpp-output
202      2: compiler, {1}, assembler
203      3: assembler, {2}, object
204      4: bind-arch, "i386", {3}, object
205      5: bind-arch, "x86_64", {3}, object
206      6: lipo, {4, 5}, object
207      7: input, "t1.c", c
208      8: preprocessor, {7}, cpp-output
209      9: compiler, {8}, assembler
210      10: assembler, {9}, object
211      11: bind-arch, "i386", {10}, object
212      12: bind-arch, "x86_64", {10}, object
213      13: lipo, {11, 12}, object
214
215   After this stage is complete the compilation process is divided into
216   a simple set of actions which need to be performed to produce
217   intermediate or final outputs (in some cases, like ``-fsyntax-only``,
218   there is no "real" final output). Phases are well known compilation
219   steps, such as "preprocess", "compile", "assemble", "link", etc.
220
221#. **Bind: Tool & Filename Selection**
222
223   This stage (in conjunction with the Translate stage) turns the tree
224   of Actions into a list of actual subprocess to run. Conceptually, the
225   driver performs a top down matching to assign Action(s) to Tools. The
226   ToolChain is responsible for selecting the tool to perform a
227   particular action; once selected the driver interacts with the tool
228   to see if it can match additional actions (for example, by having an
229   integrated preprocessor).
230
231   Once Tools have been selected for all actions, the driver determines
232   how the tools should be connected (for example, using an inprocess
233   module, pipes, temporary files, or user provided filenames). If an
234   output file is required, the driver also computes the appropriate
235   file name (the suffix and file location depend on the input types and
236   options such as ``-save-temps``).
237
238   The driver interacts with a ToolChain to perform the Tool bindings.
239   Each ToolChain contains information about all the tools needed for
240   compilation for a particular architecture, platform, and operating
241   system. A single driver invocation may query multiple ToolChains
242   during one compilation in order to interact with tools for separate
243   architectures.
244
245   The results of this stage are not computed directly, but the driver
246   can print the results via the ``-ccc-print-bindings`` option. For
247   example:
248
249   .. code-block:: console
250
251      $ clang -ccc-print-bindings -arch i386 -arch ppc t0.c
252      # "i386-apple-darwin9" - "clang", inputs: ["t0.c"], output: "/tmp/cc-Sn4RKF.s"
253      # "i386-apple-darwin9" - "darwin::Assemble", inputs: ["/tmp/cc-Sn4RKF.s"], output: "/tmp/cc-gvSnbS.o"
254      # "i386-apple-darwin9" - "darwin::Link", inputs: ["/tmp/cc-gvSnbS.o"], output: "/tmp/cc-jgHQxi.out"
255      # "ppc-apple-darwin9" - "gcc::Compile", inputs: ["t0.c"], output: "/tmp/cc-Q0bTox.s"
256      # "ppc-apple-darwin9" - "gcc::Assemble", inputs: ["/tmp/cc-Q0bTox.s"], output: "/tmp/cc-WCdicw.o"
257      # "ppc-apple-darwin9" - "gcc::Link", inputs: ["/tmp/cc-WCdicw.o"], output: "/tmp/cc-HHBEBh.out"
258      # "i386-apple-darwin9" - "darwin::Lipo", inputs: ["/tmp/cc-jgHQxi.out", "/tmp/cc-HHBEBh.out"], output: "a.out"
259
260   This shows the tool chain, tool, inputs and outputs which have been
261   bound for this compilation sequence. Here clang is being used to
262   compile t0.c on the i386 architecture and darwin specific versions of
263   the tools are being used to assemble and link the result, but generic
264   gcc versions of the tools are being used on PowerPC.
265
266#. **Translate: Tool Specific Argument Translation**
267
268   Once a Tool has been selected to perform a particular Action, the
269   Tool must construct concrete Commands which will be executed during
270   compilation. The main work is in translating from the gcc style
271   command line options to whatever options the subprocess expects.
272
273   Some tools, such as the assembler, only interact with a handful of
274   arguments and just determine the path of the executable to call and
275   pass on their input and output arguments. Others, like the compiler
276   or the linker, may translate a large number of arguments in addition.
277
278   The ArgList class provides a number of simple helper methods to
279   assist with translating arguments; for example, to pass on only the
280   last of arguments corresponding to some option, or all arguments for
281   an option.
282
283   The result of this stage is a list of Commands (executable paths and
284   argument strings) to execute.
285
286#. **Execute**
287
288   Finally, the compilation pipeline is executed. This is mostly
289   straightforward, although there is some interaction with options like
290   ``-pipe``, ``-pass-exit-codes`` and ``-time``.
291
292Additional Notes
293----------------
294
295The Compilation Object
296^^^^^^^^^^^^^^^^^^^^^^
297
298The driver constructs a Compilation object for each set of command line
299arguments. The Driver itself is intended to be invariant during
300construction of a Compilation; an IDE should be able to construct a
301single long lived driver instance to use for an entire build, for
302example.
303
304The Compilation object holds information that is particular to each
305compilation sequence. For example, the list of used temporary files
306(which must be removed once compilation is finished) and result files
307(which should be removed if compilation fails).
308
309Unified Parsing & Pipelining
310^^^^^^^^^^^^^^^^^^^^^^^^^^^^
311
312Parsing and pipelining both occur without reference to a Compilation
313instance. This is by design; the driver expects that both of these
314phases are platform neutral, with a few very well defined exceptions
315such as whether the platform uses a driver driver.
316
317ToolChain Argument Translation
318^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
319
320In order to match gcc very closely, the clang driver currently allows
321tool chains to perform their own translation of the argument list (into
322a new ArgList data structure). Although this allows the clang driver to
323match gcc easily, it also makes the driver operation much harder to
324understand (since the Tools stop seeing some arguments the user
325provided, and see new ones instead).
326
327For example, on Darwin ``-gfull`` gets translated into two separate
328arguments, ``-g`` and ``-fno-eliminate-unused-debug-symbols``. Trying to
329write Tool logic to do something with ``-gfull`` will not work, because
330Tool argument translation is done after the arguments have been
331translated.
332
333A long term goal is to remove this tool chain specific translation, and
334instead force each tool to change its own logic to do the right thing on
335the untranslated original arguments.
336
337Unused Argument Warnings
338^^^^^^^^^^^^^^^^^^^^^^^^
339
340The driver operates by parsing all arguments but giving Tools the
341opportunity to choose which arguments to pass on. One downside of this
342infrastructure is that if the user misspells some option, or is confused
343about which options to use, some command line arguments the user really
344cared about may go unused. This problem is particularly important when
345using clang as a compiler, since the clang compiler does not support
346anywhere near all the options that gcc does, and we want to make sure
347users know which ones are being used.
348
349To support this, the driver maintains a bit associated with each
350argument of whether it has been used (at all) during the compilation.
351This bit usually doesn't need to be set by hand, as the key ArgList
352accessors will set it automatically.
353
354When a compilation is successful (there are no errors), the driver
355checks the bit and emits an "unused argument" warning for any arguments
356which were never accessed. This is conservative (the argument may not
357have been used to do what the user wanted) but still catches the most
358obvious cases.
359
360Relation to GCC Driver Concepts
361-------------------------------
362
363For those familiar with the gcc driver, this section provides a brief
364overview of how things from the gcc driver map to the clang driver.
365
366-  **Driver Driver**
367
368   The driver driver is fully integrated into the clang driver. The
369   driver simply constructs additional Actions to bind the architecture
370   during the *Pipeline* phase. The tool chain specific argument
371   translation is responsible for handling ``-Xarch_``.
372
373   The one caveat is that this approach requires ``-Xarch_`` not be used
374   to alter the compilation itself (for example, one cannot provide
375   ``-S`` as an ``-Xarch_`` argument). The driver attempts to reject
376   such invocations, and overall there isn't a good reason to abuse
377   ``-Xarch_`` to that end in practice.
378
379   The upside is that the clang driver is more efficient and does little
380   extra work to support universal builds. It also provides better error
381   reporting and UI consistency.
382
383-  **Specs**
384
385   The clang driver has no direct correspondent for "specs". The
386   majority of the functionality that is embedded in specs is in the
387   Tool specific argument translation routines. The parts of specs which
388   control the compilation pipeline are generally part of the *Pipeline*
389   stage.
390
391-  **Toolchains**
392
393   The gcc driver has no direct understanding of tool chains. Each gcc
394   binary roughly corresponds to the information which is embedded
395   inside a single ToolChain.
396
397   The clang driver is intended to be portable and support complex
398   compilation environments. All platform and tool chain specific code
399   should be protected behind either abstract or well defined interfaces
400   (such as whether the platform supports use as a driver driver).
401