1.. _module-pw_stream: 2 3.. cpp:namespace-push:: pw::stream 4 5========= 6pw_stream 7========= 8.. pigweed-module:: 9 :name: pw_stream 10 11``pw_stream`` provides a foundational interface for streaming data from one part 12of a system to another. In the simplest use cases, this is basically a memcpy 13behind a reusable interface that can be passed around the system. On the other 14hand, the flexibility of this interface means a ``pw_stream`` could terminate is 15something more complex, like a UART stream or flash memory. 16 17-------- 18Overview 19-------- 20At the most basic level, ``pw_stream``'s interfaces provide very simple handles 21to enabling streaming data from one location in a system to an endpoint. 22 23Example: 24 25.. code-block:: cpp 26 27 Status DumpSensorData(pw::stream::Writer& writer) { 28 static char temp[64]; 29 ImuSample imu_sample; 30 imu.GetSample(&info); 31 size_t bytes_written = imu_sample.AsCsv(temp, sizeof(temp)); 32 return writer.Write(temp, bytes_written); 33 } 34 35In this example, ``DumpSensorData()`` only cares that it has access to a 36:cpp:class:`Writer` that it can use to stream data to using ``Writer::Write()``. 37The :cpp:class:`Writer` itself can be backed by anything that can act as a data 38"sink." 39 40--------------------- 41pw::stream Interfaces 42--------------------- 43There are three basic capabilities of a stream: 44 45* Reading -- Bytes can be read from the stream. 46* Writing -- Bytes can be written to the stream. 47* Seeking -- The position in the stream can be changed. 48 49``pw_stream`` provides a family of stream classes with different capabilities. 50The most basic class, :cpp:class:`Stream` guarantees no functionality, while the 51most capable class, :cpp:class:`SeekableReaderWriter` supports reading, writing, 52and seeking. 53 54Usage overview 55============== 56.. list-table:: 57 :header-rows: 1 58 59 * - pw::stream Interfaces 60 - Accept in APIs? 61 - Extend to create new stream? 62 * - :cpp:class:`pw::stream::Stream` 63 - ❌ 64 - ❌ 65 * - | :cpp:class:`pw::stream::Reader` 66 | :cpp:class:`pw::stream::Writer` 67 | :cpp:class:`pw::stream::ReaderWriter` 68 - ✅ 69 - ❌ 70 * - | :cpp:class:`pw::stream::SeekableReader` 71 | :cpp:class:`pw::stream::SeekableWriter` 72 | :cpp:class:`pw::stream::SeekableReaderWriter` 73 - ✅ 74 - ✅ 75 * - | :cpp:class:`pw::stream::RelativeSeekableReader` 76 | :cpp:class:`pw::stream::RelativeSeekableWriter` 77 | :cpp:class:`pw::stream::RelativeSeekableReaderWriter` 78 - ✅ (rarely) 79 - ✅ 80 * - | :cpp:class:`pw::stream::NonSeekableReader` 81 | :cpp:class:`pw::stream::NonSeekableWriter` 82 | :cpp:class:`pw::stream::NonSeekableReaderWriter` 83 - ❌ 84 - ✅ 85 86 87Interface documentation 88======================= 89Summary documentation for the ``pw_stream`` interfaces is below. See the API 90comments in `pw_stream/public/pw_stream/stream.h 91<https://cs.pigweed.dev/pigweed/+/main:pw_stream/public/pw_stream/stream.h>`_ 92for full details. 93 94.. doxygenclass:: pw::stream::Stream 95 :members: 96 :private-members: 97 98Reader interfaces 99----------------- 100.. doxygenclass:: pw::stream::Reader 101 :members: 102 103.. doxygenclass:: pw::stream::SeekableReader 104 :members: 105 106.. doxygenclass:: pw::stream::RelativeSeekableReader 107 :members: 108 109.. doxygenclass:: pw::stream::NonSeekableReader 110 :members: 111 112Writer interfaces 113----------------- 114.. doxygenclass:: pw::stream::Writer 115 :members: 116 117.. doxygenclass:: pw::stream::SeekableWriter 118 :members: 119 120.. doxygenclass:: pw::stream::RelativeSeekableWriter 121 :members: 122 123.. doxygenclass:: pw::stream::NonSeekableWriter 124 :members: 125 126 127ReaderWriter interfaces 128----------------------- 129.. doxygenclass:: pw::stream::ReaderWriter 130 :members: 131 132.. doxygenclass:: pw::stream::SeekableReaderWriter 133 :members: 134 135.. doxygenclass:: pw::stream::RelativeSeekableReaderWriter 136 :members: 137 138.. doxygenclass:: pw::stream::NonSeekableReaderWriter 139 :members: 140 141--------------- 142Implementations 143--------------- 144``pw_stream`` includes a few stream implementations for general use. 145 146.. cpp:class:: MemoryWriter : public SeekableWriter 147 148 The ``MemoryWriter`` class implements the :cpp:class:`Writer` interface by 149 backing the data destination with an **externally-provided** memory buffer. 150 ``MemoryWriterBuffer`` extends ``MemoryWriter`` to internally provide a memory 151 buffer. 152 153 The ``MemoryWriter`` can be accessed like a standard C++ container. The 154 contents grow as data is written. 155 156.. cpp:class:: MemoryReader : public SeekableReader 157 158 The ``MemoryReader`` class implements the :cpp:class:`Reader` interface by 159 backing the data source with an **externally-provided** memory buffer. 160 161.. cpp:class:: NullStream : public SeekableReaderWriter 162 163 ``NullStream`` is a no-op stream implementation, similar to ``/dev/null``. 164 Writes are always dropped. Reads always return ``OUT_OF_RANGE``. Seeks have no 165 effect. 166 167.. cpp:class:: CountingNullStream : public SeekableReaderWriter 168 169 ``CountingNullStream`` is a no-op stream implementation, like 170 :cpp:class:`NullStream`, that counts the number of bytes written. 171 172 .. cpp:function:: size_t bytes_written() const 173 174 Returns the number of bytes provided to previous ``Write()`` calls. 175 176.. cpp:class:: StdFileWriter : public SeekableWriter 177 178 ``StdFileWriter`` wraps an ``std::ofstream`` with the :cpp:class:`Writer` 179 interface. 180 181.. cpp:class:: StdFileReader : public SeekableReader 182 183 ``StdFileReader`` wraps an ``std::ifstream`` with the :cpp:class:`Reader` 184 interface. 185 186.. cpp:class:: SocketStream : public NonSeekableReaderWriter 187 188 ``SocketStream`` wraps posix-style TCP sockets with the :cpp:class:`Reader` 189 and :cpp:class:`Writer` interfaces. It can be used to connect to a TCP server, 190 or to communicate with a client via the ``ServerSocket`` class. 191 192.. cpp:class:: ServerSocket 193 194 ``ServerSocket`` wraps a posix server socket, and produces a 195 :cpp:class:`SocketStream` for each accepted client connection. 196 197------------------ 198Why use pw_stream? 199------------------ 200 201Standard API 202============ 203``pw_stream`` provides a standard way for classes to express that they have the 204ability to write data. Writing to one sink versus another sink is a matter of 205just passing a reference to the appropriate :cpp:class:`Writer`. 206 207As an example, imagine dumping sensor data. If written against a random HAL 208or one-off class, there's porting work required to write to a different sink 209(imagine writing over UART vs dumping to flash memory). Building a "dumping" 210implementation against the :cpp:class:`Writer` interface prevents a dependency 211on a bespoke API that would require porting work. 212 213Similarly, after building a :cpp:class:`Writer` implementation for a Sink that 214data could be dumped to, that same :cpp:class:`Writer` can be reused for other 215contexts that already write data to the :cpp:class:`pw::stream::Writer` 216interface. 217 218Before: 219 220.. code-block:: cpp 221 222 // Not reusable, depends on `Uart`. 223 void DumpSensorData(Uart& uart) { 224 static char temp[64]; 225 ImuSample imu_sample; 226 imu.GetSample(&info); 227 size_t bytes_written = imu_sample.AsCsv(temp, sizeof(temp)); 228 uart.Transmit(temp, bytes_written, /*timeout_ms=*/ 200); 229 } 230 231After: 232 233.. code-block:: cpp 234 235 // Reusable; no more Uart dependency! 236 Status DumpSensorData(Writer& writer) { 237 static char temp[64]; 238 ImuSample imu_sample; 239 imu.GetSample(&info); 240 size_t bytes_written = imu_sample.AsCsv(temp, sizeof(temp)); 241 return writer.Write(temp, bytes_written); 242 } 243 244Reduce intermediate buffers 245=========================== 246Often functions that write larger blobs of data request a buffer is passed as 247the destination that data should be written to. This *requires* a buffer to be 248allocated, even if the data only exists in that buffer for a very short period 249of time before it's written somewhere else. 250 251In situations where data read from somewhere will immediately be written 252somewhere else, a :cpp:class:`Writer` interface can cut out the middleman 253buffer. 254 255Before: 256 257.. code-block:: cpp 258 259 // Requires an intermediate buffer to write the data as CSV. 260 void DumpSensorData(Uart& uart) { 261 char temp[64]; 262 ImuSample imu_sample; 263 imu.GetSample(&info); 264 size_t bytes_written = imu_sample.AsCsv(temp, sizeof(temp)); 265 uart.Transmit(temp, bytes_written, /*timeout_ms=*/ 200); 266 } 267 268After: 269 270.. code-block:: cpp 271 272 // Both DumpSensorData() and RawSample::AsCsv() use a Writer, eliminating the 273 // need for an intermediate buffer. 274 Status DumpSensorData(Writer& writer) { 275 RawSample imu_sample; 276 imu.GetSample(&info); 277 return imu_sample.AsCsv(writer); 278 } 279 280Prevent buffer overflow 281======================= 282When copying data from one buffer to another, there must be checks to ensure the 283copy does not overflow the destination buffer. As this sort of logic is 284duplicated throughout a codebase, there's more opportunities for bound-checking 285bugs to sneak in. ``Writers`` manage this logic internally rather than pushing 286the bounds checking to the code that is moving or writing the data. 287 288Similarly, since only the :cpp:class:`Writer` has access to any underlying 289buffers, it's harder for functions that share a :cpp:class:`Writer` to 290accidentally clobber data written by others using the same buffer. 291 292Before: 293 294.. code-block:: cpp 295 296 Status BuildPacket(Id dest, span<const std::byte> payload, 297 span<std::byte> dest) { 298 Header header; 299 if (dest.size_bytes() + payload.size_bytes() < sizeof(Header)) { 300 return Status::ResourceExhausted(); 301 } 302 header.dest = dest; 303 header.src = DeviceId(); 304 header.payload_size = payload.size_bytes(); 305 306 memcpy(dest.data(), &header, sizeof(header)); 307 // Forgetting this line would clobber buffer contents. Also, using 308 // a temporary span instead could leave `dest` to be misused elsewhere in 309 // the function. 310 dest = dest.subspan(sizeof(header)); 311 memcpy(dest.data(), payload.data(), payload.size_bytes()); 312 } 313 314After: 315 316.. code-block:: cpp 317 318 Status BuildPacket(Id dest, span<const std::byte> payload, Writer& writer) { 319 Header header; 320 header.dest = dest; 321 header.src = DeviceId(); 322 header.payload_size = payload.size_bytes(); 323 324 writer.Write(header); 325 return writer.Write(payload); 326 } 327 328------------ 329Design notes 330------------ 331 332Sync & Flush 333============ 334The :cpp:class:`pw::stream::Stream` API does not include ``Sync()`` or 335``Flush()`` functions. There no mechanism in the :cpp:class:`Stream` API to 336synchronize a :cpp:class:`Reader`'s potentially buffered input with its 337underlying data source. This must be handled by the implementation if required. 338Similarly, the :cpp:class:`Writer` implementation is responsible for flushing 339any buffered data to the sink. 340 341``Flush()`` and ``Sync()`` were excluded from :cpp:class:`Stream` for a few 342reasons: 343 344* The semantics of when to call ``Flush()``/``Sync()`` on the stream are 345 unclear. The presence of these methods complicates using a :cpp:class:`Reader` 346 or :cpp:class:`Writer`. 347* Adding one or two additional virtual calls increases the size of all 348 :cpp:class:`Stream` vtables. 349 350.. _module-pw_stream-class-hierarchy: 351 352Class hierarchy 353=============== 354All ``pw_stream`` classes inherit from a single, common base with all possible 355functionality: :cpp:class:`pw::stream::Stream`. This structure has 356some similarities with Python's `io module 357<https://docs.python.org/3/library/io.html>`_ and C#'s `Stream class 358<https://docs.microsoft.com/en-us/dotnet/api/system.io.stream>`_. 359 360An alternative approach is to have the reading, writing, and seeking portions of 361the interface provided by different entities. This is how Go's `io 362package <https://pkg.go.dev/io>`_ and C++'s `input/output library 363<https://en.cppreference.com/w/cpp/io>`_ are structured. 364 365We chose to use a single base class for a few reasons: 366 367* The inheritance hierarchy is simple and linear. Despite the linear 368 hierarchy, combining capabilities is natural with classes like 369 :cpp:class:`ReaderWriter`. 370 371 In C++, separate interfaces for each capability requires either a complex 372 virtual inheritance hierarchy or entirely separate hierarchies for each 373 capability. Separate hierarchies can become cumbersome when trying to 374 combine multiple capabilities. A :cpp:class:`SeekableReaderWriter` would 375 have to implement three different interfaces, which means three different 376 vtables and three vtable pointers in each instance. 377* Stream capabilities are clearly expressed in the type system, while 378 naturally supporting optional functionality. A :cpp:class:`Reader` may 379 or may not support :cpp:func:`Stream::Seek`. Applications that can handle 380 seek failures gracefully way use seek on any :cpp:class:`Reader`. If seeking 381 is strictly necessary, an API can accept a :cpp:class:`SeekableReader` 382 instead. 383 384 Expressing optional functionality in the type system is cumbersome when 385 there are distinct interfaces for each capability. ``Reader``, ``Writer``, 386 and ``Seeker`` interfaces would not be sufficient. To match the flexibility 387 of the current structure, there would have to be separate optional versions 388 of each interface, and classes for various combinations. :cpp:class:`Stream` 389 would be an "OptionalReaderOptionalWriterOptionalSeeker" in this model. 390* Code reuse is maximized. For example, a single 391 :cpp:func:`Stream::ConservativeLimit` implementation supports many stream 392 implementations. 393 394Virtual interfaces 395================== 396``pw_stream`` uses virtual functions. Virtual functions enable runtime 397polymorphism. The same code can be used with any stream implementation. 398 399Virtual functions have inherently has more overhead than a regular function 400call. However, this is true of any polymorphic API. Using a C-style ``struct`` 401of function pointers makes different trade-offs but still has more overhead than 402a regular function call. 403 404For many use cases, the overhead of virtual calls insignificant. However, in 405some extremely performance-sensitive contexts, the flexibility of the virtual 406interface may not justify the performance cost. 407 408Asynchronous APIs 409================= 410At present, ``pw_stream`` is synchronous. All :cpp:class:`Stream` API calls are 411expected to block until the operation is complete. This might be undesirable 412for slow operations, like writing to NOR flash. 413 414Pigweed has not yet established a pattern for asynchronous C++ APIs. The 415:cpp:class:`Stream` class may be extended in the future to add asynchronous 416capabilities, or a separate ``AsyncStream`` could be created. 417 418.. cpp:namespace-pop:: 419 420------------ 421Dependencies 422------------ 423* :ref:`module-pw_assert` 424* :ref:`module-pw_preprocessor` 425* :ref:`module-pw_status` 426* :ref:`module-pw_span` 427 428------ 429Zephyr 430------ 431To enable ``pw_stream`` for Zephyr add ``CONFIG_PIGWEED_STREAM=y`` to the 432project's configuration. 433 434---- 435Rust 436---- 437Pigweed centric analogs to Rust ``std``'s ``Read``, ``Write``, ``Seek`` traits 438as well as a basic ``Cursor`` implementation are provided by the 439`pw_stream crate </rustdoc/pw_stream/>`_. 440 441 442.. toctree:: 443 :hidden: 444 :maxdepth: 1 445 446 backends 447 448------ 449Python 450------ 451There are legacy Python utilities used for reading and writing a serial device 452for RPC purposes. 453 454.. toctree:: 455 :hidden: 456 457 Python <py/docs> 458