README.md
1# TensorFlow Op CodeGen Machinery (Experimental)
2
3## Usage
4
5```
6usage: generate_cpp [flags] OpName1 [OpName2 ...]
7Flags:
8 --help=false bool Print this help message.
9 --category="" string Category for generated ops (e.g. 'math', 'array').
10 --namespace="" string Compact C++ namespace, default is 'tensorflow::ops'.
11 --output_dir="" string Directory into which output files will be generated.
12 --source_dir="" string The tensorflow root directory, e.g. 'tensorflow/' for in-source include paths. Any path underneath the tensorflow root is also accepted.
13 --api_dirs="" string Comma-separated list of directories containing API definitions.
14```
15
16## Design
17
18### Generator Framework
19
20The generator framework is a loose Model/View/Controller arrangement:
21
22The *Model* classes live in the ***model/*** directory. They are representations
23of the `OpDef` and `ApiDef` protos, normalized and resolved.
24
25> _For example, an `OpDef` proto's `ArgDef` members contain a type string, which
26> must be dereferenced to an `AttrDef` by name to determine its type. This
27> `AttrDef` proto message in turn contains a type string which may need to be
28> parsed as "list(type)". Other `AttrDef` messages are not types, but instead
29> argument-like modifiers. In contrast, the generator model `ArgSpec` contains a
30> resolved `ArgType` which provides a boolean `is_list()` method directly, and
31> the model `OpSpec` provides a list of only the argument-like attributes. In
32> addition to convenience, this should aid consistency between generated code in
33> each target language._
34
35The *Controller* is in the ***common/*** directory. It is the workhorse used by
36the language generators; it digests the Op registry and API definitions to build
37the model and provides utilities for the language generators.
38
39The *View* and rendering classes map the language-independent Model classes
40(`OpSpec`, `ArgSpec`, `AttrSpec`, etc.) to language-specific `SourceCode`. The
41framework does not impose any design on the language-specific generators, but
42provides some utilities, and the C++ generator is a complete example.
43
44### C++ Generator
45
46The `CppGenerator` class is the interface to the `cpp/` language directory.
47Given a config, it can generate source code for a .cc or .h file as a string or
48write it to a target file.
49
50The `CppFileRenderer` is the main renderer used by the generator; it renders an
51entire file. The `CppConfig` defines if it is operating in header or source
52mode.
53
54"Views" are stateless and intended to be low-level building blocks: a direct
55language-specific representation of the model classes. For example, an `ArgView`
56is initialized from an `ArgSpec` (which was created initially from an `ArgDef`
57proto message). Where they may have some similar methods between the model and
58view, the view methods are language-specific.
59
60For instance, the C++ generator's `ArgView::VariableName()` method is an
61language-formatted name usable as a variable representing the model `ArgSpec`
62object. In contrast, the `ArgSpec::name()` method in the model refers to the
63canonical name of the object in the proto.
64
65Where views are a representation of the *input* model, in the C++ generator,
66"renderers" then use these views to build the *output* `SourceCode`; Renderers
67understand the language at the statement/directive level and target a functional
68section of the output, such as a block comment or an entire method or file.
69
70Other differences between views and renderers:
71
72* Renderers are stateful, modifying a referenced SourceCode. Views are
73 stateless and their public methods are all const, returning strings.
74* Renderers are context-dependent, e.g. a method signature will include
75 default values when in "declaration" mode but not "definition" mode. A view
76 of some argument object simply knows its default value and does not care the
77 context.
78* In terms of dependencies, `Renderers` use `Views` and other `Renderers`.
79 However, `Renderers` do **not** reference the model directly (e.g.
80 `OpSpec`). This is because if a renderer needs to reference part of the
81 model, it should get a language specific representation.
82
83### Extending to Additional Languages
84
85The design for the C++ generator should apply to other languages, and the
86underlying generator framework (the model and controller) try to be agnostic. In
87fact, some elements of the C++ design could be formalized (such as the
88rendering/view framework) or re-used (e.g. `cpp:Renderer` could likely be shared
89with C and Java as a common C-style language renderer base class).
90
91Abstracted and condensed from the C++ generator, the overall control flow could
92be described as follows:
93
94From main() in *generate_lang_main.cc*:
95
96* Call `tensorflow::port::InitMain` and parse any flags
97* Initialize config objects (e.g. `PathConfig`, `LangConfig` from flags)
98* Initialize a new `LangGenerator` from these config objects
99* Call this generator to create/write `SourceCode` to a file
100
101In class `LangGenerator` in *lang_generator.cc*:
102
103* Initialize a new `Controller` from the config objects
104* Call this controller to build the Op models (`OpSpec`)
105* Initialize a new language-specific `View` for each model object
106* Create a blank `SourceCode` rendering target (for each output file)
107* Initialize a new `LangFileRenderer` from this target source code, the model
108 `View` objects, and config objects
109* Call this renderer to generate the target `SourceCode`
110
111The dependencies are as follows:
112
113* `lang::Generator` depends on `Controller`, `Model`, `lang::Renderers`,
114 `lang::Views`
115* `lang::Renderer` depends on `lang::View` (and `lang::Renderer` peers)
116* `lang::View` depends on the model (e.g. `OpSpec`) (and `lang::View` peers)
117