1# TensorFlow Op CodeGen Machinery (Experimental) 2 3## Usage 4 5``` 6usage: generate_cpp [flags] OpName1 [OpName2 ...] 7Flags: 8 --help=false bool Print this help message. 9 --category="" string Category for generated ops (e.g. 'math', 'array'). 10 --namespace="" string Compact C++ namespace, default is 'tensorflow::ops'. 11 --output_dir="" string Directory into which output files will be generated. 12 --source_dir="" string The tensorflow root directory, e.g. 'tensorflow/' for in-source include paths. Any path underneath the tensorflow root is also accepted. 13 --api_dirs="" string Comma-separated list of directories containing API definitions. 14``` 15 16## Design 17 18### Generator Framework 19 20The generator framework is a loose Model/View/Controller arrangement: 21 22The *Model* classes live in the ***model/*** directory. They are representations 23of the `OpDef` and `ApiDef` protos, normalized and resolved. 24 25> _For example, an `OpDef` proto's `ArgDef` members contain a type string, which 26> must be dereferenced to an `AttrDef` by name to determine its type. This 27> `AttrDef` proto message in turn contains a type string which may need to be 28> parsed as "list(type)". Other `AttrDef` messages are not types, but instead 29> argument-like modifiers. In contrast, the generator model `ArgSpec` contains a 30> resolved `ArgType` which provides a boolean `is_list()` method directly, and 31> the model `OpSpec` provides a list of only the argument-like attributes. In 32> addition to convenience, this should aid consistency between generated code in 33> each target language._ 34 35The *Controller* is in the ***common/*** directory. It is the workhorse used by 36the language generators; it digests the Op registry and API definitions to build 37the model and provides utilities for the language generators. 38 39The *View* and rendering classes map the language-independent Model classes 40(`OpSpec`, `ArgSpec`, `AttrSpec`, etc.) to language-specific `SourceCode`. The 41framework does not impose any design on the language-specific generators, but 42provides some utilities, and the C++ generator is a complete example. 43 44### C++ Generator 45 46The `CppGenerator` class is the interface to the `cpp/` language directory. 47Given a config, it can generate source code for a .cc or .h file as a string or 48write it to a target file. 49 50The `CppFileRenderer` is the main renderer used by the generator; it renders an 51entire file. The `CppConfig` defines if it is operating in header or source 52mode. 53 54"Views" are stateless and intended to be low-level building blocks: a direct 55language-specific representation of the model classes. For example, an `ArgView` 56is initialized from an `ArgSpec` (which was created initially from an `ArgDef` 57proto message). Where they may have some similar methods between the model and 58view, the view methods are language-specific. 59 60For instance, the C++ generator's `ArgView::VariableName()` method is an 61language-formatted name usable as a variable representing the model `ArgSpec` 62object. In contrast, the `ArgSpec::name()` method in the model refers to the 63canonical name of the object in the proto. 64 65Where views are a representation of the *input* model, in the C++ generator, 66"renderers" then use these views to build the *output* `SourceCode`; Renderers 67understand the language at the statement/directive level and target a functional 68section of the output, such as a block comment or an entire method or file. 69 70Other differences between views and renderers: 71 72* Renderers are stateful, modifying a referenced SourceCode. Views are 73 stateless and their public methods are all const, returning strings. 74* Renderers are context-dependent, e.g. a method signature will include 75 default values when in "declaration" mode but not "definition" mode. A view 76 of some argument object simply knows its default value and does not care the 77 context. 78* In terms of dependencies, `Renderers` use `Views` and other `Renderers`. 79 However, `Renderers` do **not** reference the model directly (e.g. 80 `OpSpec`). This is because if a renderer needs to reference part of the 81 model, it should get a language specific representation. 82 83### Extending to Additional Languages 84 85The design for the C++ generator should apply to other languages, and the 86underlying generator framework (the model and controller) try to be agnostic. In 87fact, some elements of the C++ design could be formalized (such as the 88rendering/view framework) or re-used (e.g. `cpp:Renderer` could likely be shared 89with C and Java as a common C-style language renderer base class). 90 91Abstracted and condensed from the C++ generator, the overall control flow could 92be described as follows: 93 94From main() in *generate_lang_main.cc*: 95 96* Call `tensorflow::port::InitMain` and parse any flags 97* Initialize config objects (e.g. `PathConfig`, `LangConfig` from flags) 98* Initialize a new `LangGenerator` from these config objects 99* Call this generator to create/write `SourceCode` to a file 100 101In class `LangGenerator` in *lang_generator.cc*: 102 103* Initialize a new `Controller` from the config objects 104* Call this controller to build the Op models (`OpSpec`) 105* Initialize a new language-specific `View` for each model object 106* Create a blank `SourceCode` rendering target (for each output file) 107* Initialize a new `LangFileRenderer` from this target source code, the model 108 `View` objects, and config objects 109* Call this renderer to generate the target `SourceCode` 110 111The dependencies are as follows: 112 113* `lang::Generator` depends on `Controller`, `Model`, `lang::Renderers`, 114 `lang::Views` 115* `lang::Renderer` depends on `lang::View` (and `lang::Renderer` peers) 116* `lang::View` depends on the model (e.g. `OpSpec`) (and `lang::View` peers) 117