1# TensorFlow SavedModel 2 3[TOC] 4 5## Overview 6 7SavedModel is the universal serialization format for 8[TensorFlow](https://www.tensorflow.org/) models. 9 10SavedModel provides a language-neutral format to save machine-learning models 11that is recoverable and hermetic. It enables higher-level systems and tools to 12produce, consume and transform TensorFlow models. 13 14## Guides 15* [Using the SavedModel Format](https://www.tensorflow.org/guide/saved_model) 16* [Save and load Keras models](https://www.tensorflow.org/guide/keras/save_and_serialize) 17* [Save and load with checkpointing in Keras](https://www.tensorflow.org/tutorials/keras/save_and_load) 18* [Training checkpoints](https://www.tensorflow.org/guide/checkpoint) 19* [Save and load a model using a distribution strategy](https://www.tensorflow.org/tutorials/distribute/save_and_load) 20 21 22## [Public API](https://www.tensorflow.org/api_docs/python/tf/saved_model) 23* [`tf.saved_model.save`](https://www.tensorflow.org/api_docs/python/tf/saved_model/save) 24* [`tf.saved_model.load`](https://www.tensorflow.org/api_docs/python/tf/saved_model/load) 25* [`tf.saved_model.SaveOptions`](https://www.tensorflow.org/api_docs/python/tf/saved_model/SaveOptions) 26* [`tf.saved_model.LoadOptions`](https://www.tensorflow.org/api_docs/python/tf/saved_model/LoadOptions) 27* [`tf.saved_model.Asset`](https://www.tensorflow.org/api_docs/python/tf/saved_model/Asset) 28* [`tf.saved_model.contains_saved_model`](https://www.tensorflow.org/api_docs/python/tf/saved_model/contains_saved_model) 29 30### Related Modules and Functions 31* [`tf.keras.models.save_model`](https://www.tensorflow.org/api_docs/python/tf/keras/models/save_model) 32* [`tf.keras.models.load_model`](https://www.tensorflow.org/api_docs/python/tf/keras/models/load_model) 33* [`tf.train.Checkpoint`](https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint) 34 35 36## The SavedModel Format 37A SavedModel directory has the following structure: 38 39``` 40assets/ 41assets.extra/ 42variables/ 43 variables.data-?????-of-????? 44 variables.index 45saved_model.pb 46``` 47 48* SavedModel protocol buffer 49 * [`saved_model.pb`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/protobuf/saved_model.proto) 50 or `saved_model.pbtxt` 51 * Includes the graph definitions as `MetaGraphDef` protocol buffers. 52* Assets 53 * Subfolder called `assets`. 54 * Contains auxiliary files such as vocabularies, etc. 55* Extra assets 56 * Subfolder where higher-level libraries and users can add their own 57 assets that co-exist with the model, but are not loaded by the graph. 58 * This subfolder is not managed by the SavedModel libraries. 59* Variables 60 * Subfolder called `variables`. 61 * `variables.data-?????-of-?????` 62 * `variables.index` 63 64--- 65 66## SavedModel in TensorFlow 1.x 67 68SavedModel had slightly different semantics in TF 1.x. Conventions that are 69generally only supported in TF 1.x are noted as such. 70 71### Features 72 73The following is a summary of the features in SavedModel: 74 75* (TF1-only) Multiple graphs sharing a single set of variables and assets can be added to a 76 single SavedModel. Each graph is associated with a specific set of tags to 77 allow identification during a load or restore operation. 78* (TF1-only) Support for `SignatureDefs` 79 * Graphs that are used for inference tasks typically have a set of inputs 80 and outputs. This is called a `Signature`. 81 * SavedModel uses [SignatureDefs](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/protobuf/meta_graph.proto) 82 to allow generic support for signatures that may need to be saved with the graphs. 83 * For commonly used SignatureDefs in the context of TensorFlow Serving, 84 please see documentation [here](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/signature_defs.md). 85* Support for `Assets`. 86 * For cases where ops depend on external files for initialization, such as 87 vocabularies, SavedModel supports this via `assets`. 88 * Assets are copied to the SavedModel location and can be read when loading 89 a specific meta graph def. 90* Support to clear devices before generating the SavedModel. 91 92The following is a summary of features that are NOT supported in SavedModel. 93Higher-level frameworks and tools that use SavedModel may provide these. 94 95* Implicit versioning. 96* Garbage collection. 97* Atomic writes to the SavedModel location. 98 99### TF1 SavedModel Background 100SavedModel manages and builds upon existing TensorFlow primitives such as 101`TensorFlow Saver` and `MetaGraphDef`. Specifically, SavedModel wraps a [TensorFlow Saver](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/python/training/saver.py). 102The Saver is primarily used to generate the variable checkpoints. SavedModel 103will replace the existing [TensorFlow Inference Model Format](https://github.com/tensorflow/tensorflow/tree/r1.15/tensorflow/contrib/session_bundle#tensorflow-inference-model-format) 104as the canonical way to export TensorFlow graphs for serving. 105 106 107### APIs 108The APIs for building and loading a SavedModel are described in this section. 109 110#### (TF1-only) Builder 111The SavedModel [builder](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/saved_model/builder.py) 112is implemented in Python. 113 114The `SavedModelBuilder` class provides functionality to save multiple meta graph 115defs, associated variables and assets. 116 117To build a SavedModel, the first meta graph must be saved with variables. 118Subsequent meta graphs will simply be saved with their graph definitions. If 119assets need to be saved and written or copied to disk, they can be provided 120when the meta graph def is added. If multiple meta graph defs are associated 121with an asset of the same name, only the first version is retained. 122 123#### (TF1-only) Tags 124Each meta graph added to the SavedModel must be annotated with user specified 125tags, which reflect the meta graph capabilities or use-cases. 126More specifically, these tags typically annotate a meta graph with its 127functionality (e.g. serving or training), and possibly hardware specific aspects 128such as GPU. 129In the SavedModel, the meta graph def whose tag-set exactly matches those 130specified in the loader API, will be the one loaded by the loader. 131If no meta graph def is found matching the specified tags, an error is returned. 132For example, a loader with a requirement to serve on GPU hardware would be able 133to load only meta graph annotated with tags='serve,gpu' by specifying this set 134of tags in tensorflow::LoadSavedModel(...). 135 136 137#### Usage 138The typical usage of `builder` is as follows: 139 140~~~python 141export_dir = ... 142... 143builder = tf.saved_model.builder.SavedModelBuilder(export_dir) 144with tf.Session(graph=tf.Graph()) as sess: 145 ... 146 builder.add_meta_graph_and_variables(sess, 147 [tf.saved_model.tag_constants.TRAINING], 148 signature_def_map=foo_signatures, 149 assets_collection=foo_assets) 150... 151with tf.Session(graph=tf.Graph()) as sess: 152 ... 153 builder.add_meta_graph(["bar-tag", "baz-tag"]) 154... 155builder.save() 156~~~ 157 158#### (TF1-only) Stripping Default valued attributes 159The SavedModelBuilder class allows users to control whether default-valued 160attributes must be stripped from the NodeDefs while adding a meta graph to the 161SavedModel bundle. Both `SavedModelBuilder.add_meta_graph_and_variables` and 162`SavedModelBuilder.add_meta_graph` methods accept a Boolean flag 163`strip_default_attrs` that controls this behavior. 164 165If `strip_default_attrs` is `False`, the exported MetaGraphDef will have the 166default valued attributes in all it's NodeDef instances. This can break forward 167compatibility with a sequence of events such as the following: 168 169* An existing Op (`Foo`) is updated to include a new attribute (`T`) with a 170 default (`bool`) at version 101. 171* A model producer (such as a Trainer) binary picks up this change 172 (version 101) to the OpDef and re-exports an existing model that uses Op `Foo`. 173* A model consumer (such as Tensorflow Serving) running an older binary 174 (version 100) doesn't have attribute `T` for Op `Foo`, but tries to import 175 this model. The model consumer doesn't recognize attribute `T` in a NodeDef 176 that uses Op `Foo` and therefore fails to load the model. 177 178By setting `strip_default_attrs` to `True`, the model producers can strip away 179any default valued attributes in the NodeDefs. This helps ensure that newly 180added attributes with defaults don't cause older model consumers to fail loading 181models regenerated with newer training binaries. 182 183TIP: If you care about forward compatibility, then set `strip_default_attrs` 184to `True` while using `SavedModelBuilder.add_meta_graph_and_variables` and 185`SavedModelBuilder.add_meta_graph`. 186 187### Loader 188The SavedModel loader is implemented in C++ and Python. 189 190#### (TF1-only) Python 191The Python version of the SavedModel [loader](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/saved_model/loader.py) 192provides load and restore capability for a SavedModel. The `load` operation 193requires the session in which to restore the graph definition and variables, the 194tags used to identify the meta graph def to load and the location of the 195SavedModel. Upon a load, the subset of variables and assets supplied as part of 196the specific meta graph def, will be restored into the supplied session. 197 198~~~python 199export_dir = ... 200... 201with tf.Session(graph=tf.Graph()) as sess: 202 tf.saved_model.loader.load(sess, [tag_constants.TRAINING], export_dir) 203 ... 204~~~ 205 206#### C++ 207The C++ version of the SavedModel [loader](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/cc/saved_model/loader.h) 208provides an API to load a SavedModel from a path, while allowing 209`SessionOptions` and `RunOptions`. Similar to the Python version, the C++ 210version requires the tags associated with the graph to be loaded, to be 211specified. The loaded version of SavedModel is referred to as `SavedModelBundle` 212and contains the meta graph def and the session within which it is loaded. 213 214~~~c++ 215const string export_dir = ... 216SavedModelBundle bundle; 217... 218LoadSavedModel(session_options, run_options, export_dir, {kSavedModelTagTrain}, 219 &bundle); 220~~~ 221 222### Constants 223SavedModel offers the flexibility to build and load TensorFlow graphs for a 224variety of use-cases. For the set of most common expected use-cases, 225SavedModel's APIs provide a set of constants in Python and C++ that are easy to 226reuse and share across tools consistently. 227 228#### (TF1-specific) Tag constants 229Sets of tags can be used to uniquely identify a `MetaGraphDef` saved in a 230SavedModel. A subset of commonly used tags is specified in: 231 232* [Python](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/saved_model/tag_constants.py) 233* [C++](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/cc/saved_model/tag_constants.h). 234 235#### (TF1-specific) Signature constants 236SignatureDefs are used to define the signature of a computation supported in a 237TensorFlow graph. Commonly used input keys, output keys and method names are 238defined in: 239 240* [Python](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/saved_model/signature_constants.py) 241* [C++](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/cc/saved_model/signature_constants.h). 242