• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1.. _module-pw_protobuf_compiler:
2
3--------------------
4pw_protobuf_compiler
5--------------------
6The Protobuf compiler module provides build system integration and wrapper
7scripts for generating source code for Protobuf definitions.
8
9Generator support
10=================
11Protobuf code generation is currently supported for the following generators:
12
13+-------------+----------------+-----------------------------------------------+
14| Generator   | Code           | Notes                                         |
15+-------------+----------------+-----------------------------------------------+
16| pw_protobuf | ``pwpb``       | Compiles using ``pw_protobuf``.               |
17+-------------+----------------+-----------------------------------------------+
18| Nanopb      | ``nanopb``     | Compiles using Nanopb. The build argument     |
19|             |                | ``dir_pw_third_party_nanopb`` must be set to  |
20|             |                | point to a local nanopb installation.         |
21+-------------+----------------+-----------------------------------------------+
22| Nanopb RPC  | ``nanopb_rpc`` | Compiles pw_rpc service and client code for   |
23|             |                | nanopb. Requires a nanopb installation.       |
24+-------------+----------------+-----------------------------------------------+
25| Raw RPC     | ``raw_rpc``    | Compiles raw binary pw_rpc service code.      |
26+-------------+----------------+-----------------------------------------------+
27| Go          | ``go``         | Compiles using the standard Go protobuf       |
28|             |                | plugin with gRPC service support.             |
29+-------------+----------------+-----------------------------------------------+
30| Python      | ``python``     | Compiles using the standard Python protobuf   |
31|             |                | plugin, creating a ``pw_python_package``.     |
32+-------------+----------------+-----------------------------------------------+
33
34GN template
35===========
36The ``pw_proto_library`` GN template is provided by the module.
37
38It defines a collection of protobuf files that should be compiled together. The
39template creates a sub-target for each supported generator, named
40``<target_name>.<generator>``. These sub-targets generate their respective
41protobuf code, and expose it to the build system appropriately (e.g. a
42``pw_source_set`` for C/C++).
43
44For example, given the following target:
45
46.. code-block::
47
48  pw_proto_library("test_protos") {
49    sources = [ "my_test_protos/test.proto" ]
50  }
51
52``test_protos.pwpb`` compiles code for pw_protobuf, and ``test_protos.nanopb``
53compiles using Nanopb (if it's installed).
54
55Protobuf code is only generated when a generator sub-target is listed as a
56dependency of another GN target.
57
58GN permits using abbreviated labels when the target name matches the directory
59name (e.g. ``//foo`` for ``//foo:foo``). For consistency with this, the
60sub-targets for each generator are aliased to the directory when the target name
61is the same. For example, these two labels are equivalent:
62
63.. code-block::
64
65  //path/to/my_protos:my_protos.pwpb
66  //path/to/my_protos:pwpb
67
68``pw_python_package`` subtargets are also available on the ``python`` subtarget:
69
70.. code-block::
71
72  //path/to/my_protos:my_protos.python.lint
73  //path/to/my_protos:python.lint
74
75**Arguments**
76
77* ``sources``: List of input .proto files.
78* ``deps``: List of other pw_proto_library dependencies.
79* ``inputs``: Other files on which the protos depend (e.g. nanopb ``.options``
80  files).
81* ``prefix``: A prefix to add to the source protos prior to compilation. For
82  example, a source called ``"foo.proto"`` with ``prefix = "nested"`` will be
83  compiled with protoc as ``"nested/foo.proto"``.
84* ``strip_prefix``: Remove this prefix from the source protos. All source and
85  input files must be nested under this path.
86* ``python_package``: Label of Python package to which to add the proto modules.
87
88**Example**
89
90.. code-block::
91
92  import("$dir_pw_protobuf_compiler/proto.gni")
93
94  pw_proto_library("my_protos") {
95    sources = [
96      "my_protos/foo.proto",
97      "my_protos/bar.proto",
98    ]
99  }
100
101  pw_proto_library("my_other_protos") {
102    sources = [ "some/other/path/baz.proto" ]  # imports foo.proto
103
104    # This removes the "some/other/path" prefix from the proto files.
105    strip_prefix = "some/other/path"
106
107    # This adds the "my_other_protos/" prefix to the proto files.
108    prefix = "my_other_protos"
109
110    # Proto libraries depend on other proto libraries directly.
111    deps = [ ":my_protos" ]
112  }
113
114  source_set("my_cc_code") {
115    sources = [
116      "foo.cc",
117      "bar.cc",
118      "baz.cc",
119    ]
120
121    # When depending on protos in a source_set, specify the generator suffix.
122    deps = [ ":my_other_protos.pwpb" ]
123  }
124
125From C++, ``baz.proto`` included as follows:
126
127.. code-block:: cpp
128
129  #include "my_other_protos/baz.pwpb.h"
130
131From Python, ``baz.proto`` is imported as follows:
132
133.. code-block:: python
134
135  from my_other_protos import baz_pb2
136
137Proto file structure
138--------------------
139Protobuf source files must be nested under another directory when they are
140compiled. This ensures that they can be packaged properly in Python. The first
141directory is used as the Python package name, so must be unique across the
142build. The ``prefix`` option may be used to set this directory.
143
144Using ``prefix`` and ``strip_prefix`` together allows remapping proto files to
145a completely different path. This can be useful when working with protos defined
146in external libraries. For example, consider this proto library:
147
148.. code-block::
149
150  pw_proto_library("external_protos") {
151    sources = [
152      "//other/external/some_library/src/protos/alpha.proto",
153      "//other/external/some_library/src/protos/beta.proto,
154      "//other/external/some_library/src/protos/internal/gamma.proto",
155    ]
156    strip_prefix = "//other/external/some_library/src/protos"
157    prefix = "some_library"
158  }
159
160These protos will be compiled by protoc as if they were in this file structure:
161
162.. code-block::
163
164  some_library/
165  ├── alpha.proto
166  ├── beta.proto
167  └── internal
168      └── gamma.proto
169
170.. _module-pw_protobuf_compiler-add-to-python-package:
171
172Adding Python proto modules to an existing package
173--------------------------------------------------
174By default, generated Python proto modules are organized into their own Python
175package. These proto modules can instead be added to an existing Python package
176declared with ``pw_python_package``. This is done by setting the
177``python_package`` argument on the ``pw_proto_library`` and the
178``proto_library`` argument on the ``pw_python_package``.
179
180For example, the protos declared in ``my_protos`` will be nested in the Python
181package declared by ``my_package``.
182
183.. code-block::
184
185  pw_proto_library("my_protos") {
186    sources = [ "hello.proto ]
187    prefix = "foo"
188    python_package = ":my_package"
189  }
190
191  pw_python_pacakge("my_package") {
192    generate_setup = {
193      name = "foo"
194      version = "1.0"
195    }
196    sources = [ "foo/cool_module.py" ]
197    proto_library = ":my_protos"
198  }
199
200The ``hello_pb2.py`` proto module can be used alongside other files in the
201``foo`` package.
202
203.. code-block:: python
204
205  from foo import cool_module, hello_pb2
206
207Working with externally defined protos
208--------------------------------------
209``pw_proto_library`` targets may be used to build ``.proto`` sources from
210existing projects. In these cases, it may be necessary to supply the
211``strip_prefix`` argument, which specifies the protobuf include path to use for
212``protoc``. If only a single external protobuf is being compiled, the
213``python_module_as_package`` option can be used to override the requirement that
214the protobuf be nested under a directory. This option generates a Python package
215with the same name as the proto file, so that the generated proto can be
216imported as if it were a standalone Python module.
217
218For example, the ``pw_proto_library`` target for Nanopb sets
219``python_module_as_package`` to ``nanopb_pb2``.
220
221.. code-block::
222
223  pw_proto_library("proto") {
224    strip_prefix = "$dir_pw_third_party_nanopb/generator/proto"
225    sources = [ "$dir_pw_third_party_nanopb/generator/proto/nanopb.proto" ]
226    python_module_as_package = "nanopb_pb2"
227  }
228
229In Python, this makes ``nanopb.proto`` available as ``import nanopb_pb2`` via
230the ``nanopb_pb2`` Python package. In C++, ``nanopb.proto`` is accessed as
231``#include "nanopb.pwpb.h"``.
232
233The ``python_module_as_package`` feature should only be used when absolutely
234necessary --- for example, to support proto files that include
235``import "nanopb.proto"``.
236
237CMake
238=====
239CMake provides a ``pw_proto_library`` function with similar features as the
240GN template. The CMake build only supports building firmware code, so
241``pw_proto_library`` does not generate a Python package.
242
243**Arguments**
244
245* ``NAME``: the base name of the libraries to create
246* ``SOURCES``: .proto source files
247* ``DEPS``: dependencies on other ``pw_proto_library`` targets
248* ``PREFIX``: prefix add to the proto files
249* ``STRIP_PREFIX``: prefix to remove from the proto files
250* ``INPUTS``: files to include along with the .proto files (such as Nanopb
251  .options files)
252
253**Example**
254
255 .. code-block:: cmake
256
257  include($ENV{PW_ROOT}/pw_build/pigweed.cmake)
258  include($ENV{PW_ROOT}/pw_protobuf_compiler/proto.cmake)
259
260  pw_proto_library(my_module.my_protos
261    SOURCES
262      my_protos/foo.proto
263      my_protos/bar.proto
264  )
265
266  pw_proto_library(my_module.my_protos
267    SOURCES
268      my_protos/foo.proto
269      my_protos/bar.proto
270  )
271
272  pw_proto_library(my_module.my_other_protos
273    SOURCES
274      some/other/path/baz.proto  # imports foo.proto
275
276    # This removes the "some/other/path" prefix from the proto files.
277    STRIP_PREFIX
278      some/other/path
279
280    # This adds the "my_other_protos/" prefix to the proto files.
281    PREFIX
282      my_other_protos
283
284    # Proto libraries depend on other proto libraries directly.
285    DEPS
286      my_module.my_protos
287  )
288
289  add_library(my_module.my_cc_code
290      foo.cc
291      bar.cc
292      baz.cc
293  )
294
295  # When depending on protos in a source_set, specify the generator suffix.
296  target_link_libraries(my_module.my_cc_code PUBLIC
297    my_module.my_other_protos.pwpb
298  )
299
300These proto files are accessed in C++ the same as in the GN build:
301
302.. code-block:: cpp
303
304  #include "my_other_protos/baz.pwpb.h"
305