1.. _module-pw_protobuf_compiler: 2 3-------------------- 4pw_protobuf_compiler 5-------------------- 6The Protobuf compiler module provides build system integration and wrapper 7scripts for generating source code for Protobuf definitions. 8 9Generator support 10================= 11Protobuf code generation is currently supported for the following generators: 12 13+-------------+----------------+-----------------------------------------------+ 14| Generator | Code | Notes | 15+-------------+----------------+-----------------------------------------------+ 16| pw_protobuf | ``pwpb`` | Compiles using ``pw_protobuf``. | 17+-------------+----------------+-----------------------------------------------+ 18| Nanopb | ``nanopb`` | Compiles using Nanopb. The build argument | 19| | | ``dir_pw_third_party_nanopb`` must be set to | 20| | | point to a local nanopb installation. | 21+-------------+----------------+-----------------------------------------------+ 22| Nanopb RPC | ``nanopb_rpc`` | Compiles pw_rpc service and client code for | 23| | | nanopb. Requires a nanopb installation. | 24+-------------+----------------+-----------------------------------------------+ 25| Raw RPC | ``raw_rpc`` | Compiles raw binary pw_rpc service code. | 26+-------------+----------------+-----------------------------------------------+ 27| Go | ``go`` | Compiles using the standard Go protobuf | 28| | | plugin with gRPC service support. | 29+-------------+----------------+-----------------------------------------------+ 30| Python | ``python`` | Compiles using the standard Python protobuf | 31| | | plugin, creating a ``pw_python_package``. | 32+-------------+----------------+-----------------------------------------------+ 33 34GN template 35=========== 36The ``pw_proto_library`` GN template is provided by the module. 37 38It defines a collection of protobuf files that should be compiled together. The 39template creates a sub-target for each supported generator, named 40``<target_name>.<generator>``. These sub-targets generate their respective 41protobuf code, and expose it to the build system appropriately (e.g. a 42``pw_source_set`` for C/C++). 43 44For example, given the following target: 45 46.. code-block:: 47 48 pw_proto_library("test_protos") { 49 sources = [ "my_test_protos/test.proto" ] 50 } 51 52``test_protos.pwpb`` compiles code for pw_protobuf, and ``test_protos.nanopb`` 53compiles using Nanopb (if it's installed). 54 55Protobuf code is only generated when a generator sub-target is listed as a 56dependency of another GN target. 57 58GN permits using abbreviated labels when the target name matches the directory 59name (e.g. ``//foo`` for ``//foo:foo``). For consistency with this, the 60sub-targets for each generator are aliased to the directory when the target name 61is the same. For example, these two labels are equivalent: 62 63.. code-block:: 64 65 //path/to/my_protos:my_protos.pwpb 66 //path/to/my_protos:pwpb 67 68``pw_python_package`` subtargets are also available on the ``python`` subtarget: 69 70.. code-block:: 71 72 //path/to/my_protos:my_protos.python.lint 73 //path/to/my_protos:python.lint 74 75**Arguments** 76 77* ``sources``: List of input .proto files. 78* ``deps``: List of other pw_proto_library dependencies. 79* ``inputs``: Other files on which the protos depend (e.g. nanopb ``.options`` 80 files). 81* ``prefix``: A prefix to add to the source protos prior to compilation. For 82 example, a source called ``"foo.proto"`` with ``prefix = "nested"`` will be 83 compiled with protoc as ``"nested/foo.proto"``. 84* ``strip_prefix``: Remove this prefix from the source protos. All source and 85 input files must be nested under this path. 86* ``python_package``: Label of Python package to which to add the proto modules. 87 88**Example** 89 90.. code-block:: 91 92 import("$dir_pw_protobuf_compiler/proto.gni") 93 94 pw_proto_library("my_protos") { 95 sources = [ 96 "my_protos/foo.proto", 97 "my_protos/bar.proto", 98 ] 99 } 100 101 pw_proto_library("my_other_protos") { 102 sources = [ "some/other/path/baz.proto" ] # imports foo.proto 103 104 # This removes the "some/other/path" prefix from the proto files. 105 strip_prefix = "some/other/path" 106 107 # This adds the "my_other_protos/" prefix to the proto files. 108 prefix = "my_other_protos" 109 110 # Proto libraries depend on other proto libraries directly. 111 deps = [ ":my_protos" ] 112 } 113 114 source_set("my_cc_code") { 115 sources = [ 116 "foo.cc", 117 "bar.cc", 118 "baz.cc", 119 ] 120 121 # When depending on protos in a source_set, specify the generator suffix. 122 deps = [ ":my_other_protos.pwpb" ] 123 } 124 125From C++, ``baz.proto`` included as follows: 126 127.. code-block:: cpp 128 129 #include "my_other_protos/baz.pwpb.h" 130 131From Python, ``baz.proto`` is imported as follows: 132 133.. code-block:: python 134 135 from my_other_protos import baz_pb2 136 137Proto file structure 138-------------------- 139Protobuf source files must be nested under another directory when they are 140compiled. This ensures that they can be packaged properly in Python. The first 141directory is used as the Python package name, so must be unique across the 142build. The ``prefix`` option may be used to set this directory. 143 144Using ``prefix`` and ``strip_prefix`` together allows remapping proto files to 145a completely different path. This can be useful when working with protos defined 146in external libraries. For example, consider this proto library: 147 148.. code-block:: 149 150 pw_proto_library("external_protos") { 151 sources = [ 152 "//other/external/some_library/src/protos/alpha.proto", 153 "//other/external/some_library/src/protos/beta.proto, 154 "//other/external/some_library/src/protos/internal/gamma.proto", 155 ] 156 strip_prefix = "//other/external/some_library/src/protos" 157 prefix = "some_library" 158 } 159 160These protos will be compiled by protoc as if they were in this file structure: 161 162.. code-block:: 163 164 some_library/ 165 ├── alpha.proto 166 ├── beta.proto 167 └── internal 168 └── gamma.proto 169 170.. _module-pw_protobuf_compiler-add-to-python-package: 171 172Adding Python proto modules to an existing package 173-------------------------------------------------- 174By default, generated Python proto modules are organized into their own Python 175package. These proto modules can instead be added to an existing Python package 176declared with ``pw_python_package``. This is done by setting the 177``python_package`` argument on the ``pw_proto_library`` and the 178``proto_library`` argument on the ``pw_python_package``. 179 180For example, the protos declared in ``my_protos`` will be nested in the Python 181package declared by ``my_package``. 182 183.. code-block:: 184 185 pw_proto_library("my_protos") { 186 sources = [ "hello.proto ] 187 prefix = "foo" 188 python_package = ":my_package" 189 } 190 191 pw_python_pacakge("my_package") { 192 generate_setup = { 193 name = "foo" 194 version = "1.0" 195 } 196 sources = [ "foo/cool_module.py" ] 197 proto_library = ":my_protos" 198 } 199 200The ``hello_pb2.py`` proto module can be used alongside other files in the 201``foo`` package. 202 203.. code-block:: python 204 205 from foo import cool_module, hello_pb2 206 207Working with externally defined protos 208-------------------------------------- 209``pw_proto_library`` targets may be used to build ``.proto`` sources from 210existing projects. In these cases, it may be necessary to supply the 211``strip_prefix`` argument, which specifies the protobuf include path to use for 212``protoc``. If only a single external protobuf is being compiled, the 213``python_module_as_package`` option can be used to override the requirement that 214the protobuf be nested under a directory. This option generates a Python package 215with the same name as the proto file, so that the generated proto can be 216imported as if it were a standalone Python module. 217 218For example, the ``pw_proto_library`` target for Nanopb sets 219``python_module_as_package`` to ``nanopb_pb2``. 220 221.. code-block:: 222 223 pw_proto_library("proto") { 224 strip_prefix = "$dir_pw_third_party_nanopb/generator/proto" 225 sources = [ "$dir_pw_third_party_nanopb/generator/proto/nanopb.proto" ] 226 python_module_as_package = "nanopb_pb2" 227 } 228 229In Python, this makes ``nanopb.proto`` available as ``import nanopb_pb2`` via 230the ``nanopb_pb2`` Python package. In C++, ``nanopb.proto`` is accessed as 231``#include "nanopb.pwpb.h"``. 232 233The ``python_module_as_package`` feature should only be used when absolutely 234necessary --- for example, to support proto files that include 235``import "nanopb.proto"``. 236 237CMake 238===== 239CMake provides a ``pw_proto_library`` function with similar features as the 240GN template. The CMake build only supports building firmware code, so 241``pw_proto_library`` does not generate a Python package. 242 243**Arguments** 244 245* ``NAME``: the base name of the libraries to create 246* ``SOURCES``: .proto source files 247* ``DEPS``: dependencies on other ``pw_proto_library`` targets 248* ``PREFIX``: prefix add to the proto files 249* ``STRIP_PREFIX``: prefix to remove from the proto files 250* ``INPUTS``: files to include along with the .proto files (such as Nanopb 251 .options files) 252 253**Example** 254 255 .. code-block:: cmake 256 257 include($ENV{PW_ROOT}/pw_build/pigweed.cmake) 258 include($ENV{PW_ROOT}/pw_protobuf_compiler/proto.cmake) 259 260 pw_proto_library(my_module.my_protos 261 SOURCES 262 my_protos/foo.proto 263 my_protos/bar.proto 264 ) 265 266 pw_proto_library(my_module.my_protos 267 SOURCES 268 my_protos/foo.proto 269 my_protos/bar.proto 270 ) 271 272 pw_proto_library(my_module.my_other_protos 273 SOURCES 274 some/other/path/baz.proto # imports foo.proto 275 276 # This removes the "some/other/path" prefix from the proto files. 277 STRIP_PREFIX 278 some/other/path 279 280 # This adds the "my_other_protos/" prefix to the proto files. 281 PREFIX 282 my_other_protos 283 284 # Proto libraries depend on other proto libraries directly. 285 DEPS 286 my_module.my_protos 287 ) 288 289 add_library(my_module.my_cc_code 290 foo.cc 291 bar.cc 292 baz.cc 293 ) 294 295 # When depending on protos in a source_set, specify the generator suffix. 296 target_link_libraries(my_module.my_cc_code PUBLIC 297 my_module.my_other_protos.pwpb 298 ) 299 300These proto files are accessed in C++ the same as in the GN build: 301 302.. code-block:: cpp 303 304 #include "my_other_protos/baz.pwpb.h" 305