1.. _module-pw_stream: 2 3.. cpp:namespace-push:: pw::stream 4 5========= 6pw_stream 7========= 8 9``pw_stream`` provides a foundational interface for streaming data from one part 10of a system to another. In the simplest use cases, this is basically a memcpy 11behind a reusable interface that can be passed around the system. On the other 12hand, the flexibility of this interface means a ``pw_stream`` could terminate is 13something more complex, like a UART stream or flash memory. 14 15-------- 16Overview 17-------- 18At the most basic level, ``pw_stream``'s interfaces provide very simple handles 19to enabling streaming data from one location in a system to an endpoint. 20 21Example: 22 23.. code-block:: cpp 24 25 Status DumpSensorData(pw::stream::Writer& writer) { 26 static char temp[64]; 27 ImuSample imu_sample; 28 imu.GetSample(&info); 29 size_t bytes_written = imu_sample.AsCsv(temp, sizeof(temp)); 30 return writer.Write(temp, bytes_written); 31 } 32 33In this example, ``DumpSensorData()`` only cares that it has access to a 34:cpp:class:`Writer` that it can use to stream data to using ``Writer::Write()``. 35The :cpp:class:`Writer` itself can be backed by anything that can act as a data 36"sink." 37 38--------------------- 39pw::stream Interfaces 40--------------------- 41There are three basic capabilities of a stream: 42 43 * Reading -- Bytes can be read from the stream. 44 * Writing -- Bytes can be written to the stream. 45 * Seeking -- The position in the stream can be changed. 46 47``pw_stream`` provides a family of stream classes with different capabilities. 48The most basic class, :cpp:class:`Stream` guarantees no functionality, while the 49most capable class, :cpp:class:`SeekableReaderWriter` supports reading, writing, 50and seeking. 51 52Usage overview 53============== 54 55+-------------------------------------------+-----------------+------------------------------+ 56| pw::stream Interfaces | Accept in APIs? | Extend to create new stream? | 57+===========================================+=================+==============================+ 58| :cpp:class:`Stream` | ❌ | ❌ | 59+-------------------------------------------+-----------------+------------------------------+ 60| :cpp:class:`Reader` | ✅ | ❌ | 61| | | | 62| :cpp:class:`Writer` | | | 63| | | | 64| :cpp:class:`ReaderWriter` | | | 65+-------------------------------------------+-----------------+------------------------------+ 66| :cpp:class:`SeekableReader` | ✅ | ✅ | 67| | | | 68| :cpp:class:`SeekableWriter` | | | 69| | | | 70| :cpp:class:`SeekableReaderWriter` | | | 71+-------------------------------------------+-----------------+------------------------------+ 72| :cpp:class:`RelativeSeekableReader` | ✅ (rarely) | ✅ | 73| | | | 74| :cpp:class:`RelativeSeekableWriter` | | | 75| | | | 76| :cpp:class:`RelativeSeekableReaderWriter` | | | 77+-------------------------------------------+-----------------+------------------------------+ 78| :cpp:class:`NonSeekableReader` | ❌ | ✅ | 79| | | | 80| :cpp:class:`NonSeekableWriter` | | | 81| | | | 82| :cpp:class:`NonSeekableReaderWriter` | | | 83+-------------------------------------------+-----------------+------------------------------+ 84 85Interface documentation 86======================= 87Summary documentation for the ``pw_stream`` interfaces is below. See the API 88comments in `pw_stream/public/pw_stream/stream.h 89<https://cs.opensource.google/pigweed/pigweed/+/main:pw_stream/public/pw_stream/stream.h>`_ 90for full details. 91 92.. cpp:class:: Stream 93 94 A generic stream that may support reading, writing, and seeking, but makes no 95 guarantees about whether any operations are supported. Stream serves as the 96 base for the Reader, Writer, and ReaderWriter interfaces. 97 98 Stream cannot be extended directly. Instead, work with one of the derived 99 classes that explicitly supports the required functionality. Stream should 100 almost never be used in APIs; accept a derived class with the required 101 capabilities instead. 102 103 All Stream methods are blocking. They return when the requested operation 104 completes. 105 106 **Public methods** 107 108 .. cpp:function:: bool readable() const 109 110 True if :cpp:func:`Read` is supported. 111 112 .. cpp:function:: bool writable() const 113 114 True if :cpp:func:`Write` is supported. 115 116 .. cpp:function:: bool seekable() const 117 118 True if :cpp:func:`Seek` is supported. 119 120 .. cpp:function:: Result<ByteSpan> Read(ByteSpan buffer) 121 .. cpp:function:: Result<ByteSpan> Read(void* buffer, size_t size_bytes) 122 123 Reads data from the stream into the provided buffer, if supported. As many 124 bytes as are available up to the buffer size are copied into the buffer. 125 Remaining bytes may by read in subsequent calls. 126 127 Returns: 128 129 * OK - Between 1 and dest.size_bytes() were successfully read. Returns 130 the span of read bytes. 131 * UNIMPLEMENTED - This stream does not support writing. 132 * FAILED_PRECONDITION - The Reader is not in state to read data. 133 * RESOURCE_EXHAUSTED - Unable to read any bytes at this time. No bytes 134 read. Try again once bytes become available. 135 * OUT_OF_RANGE - Reader has been exhausted, similar to EOF. No bytes were 136 read, no more will be read. 137 138 .. cpp:function:: Status Write(ConstByteSpan data) 139 140 Writes the provided data to the stream, if supported. 141 142 Returns: 143 144 * OK - Data was successfully accepted by the stream. 145 * UNIMPLEMENTED - This stream does not support writing. 146 * FAILED_PRECONDITION - The writer is not in a state to accept data. 147 * RESOURCE_EXHAUSTED - The writer was unable to write all of requested data 148 at this time. No data was written. 149 * OUT_OF_RANGE - The Writer has been exhausted, similar to EOF. No data was 150 written; no more will be written. 151 152 153 .. cpp:function:: Status Seek(ptrdiff_t offset, Whence origin = kBeginning) 154 155 Changes the current read & write position in the stream, if supported. 156 157 Returns: 158 159 * OK - Successfully updated the position. 160 * UNIMPLEMENTED - Seeking is not supported for this stream. 161 * OUT_OF_RANGE - Attempted to seek beyond the bounds of the stream. The 162 position is unchanged. 163 164 .. cpp:function:: size_t Tell() const 165 166 Returns the current read & write position in the stream, if supported. 167 Returns ``Stream::kUnknownPosition`` (``size_t(-1)``) if unsupported. 168 169 .. cpp:function:: size_t ConservativeReadLimit() const 170 171 Likely minimum bytes available to read. Returns ``kUnlimited`` 172 (``size_t(-1)``) if there is no limit or it is unknown. 173 174 .. cpp:function:: size_t ConservativeWriteLimit() const 175 176 Likely minimum bytes available to write. Returns ``kUnlimited`` 177 (``size_t(-1)``) if there is no limit or it is unknown. 178 179 **Private virtual methods** 180 181 Stream's public methods are non-virtual. The public methods call private 182 virtual methods that are implemented by derived classes. 183 184 .. cpp:function:: private virtual StatusWithSize DoRead(ByteSpan destination) 185 186 Virtual :cpp:func:`Read` function implemented by derived classes. 187 188 .. cpp:function:: private virtual Status DoWrite(ConstByteSpan data) 189 190 Virtual :cpp:func:`Write` function implemented by derived classes. 191 192 .. cpp:function:: private virtual Status DoSeek(ptrdiff_t offset, Whence origin) 193 194 Virtual :cpp:func:`Seek` function implemented by derived classes. 195 196 .. cpp:function:: private virtual size_t DoTell() const 197 198 Virtual :cpp:func:`Tell` function optionally implemented by derived classes. 199 The default implementation always returns ``kUnknownPosition``. 200 201 .. cpp:function:: private virtual size_t ConservativeLimit(LimitType limit_type) 202 203 Virtual function optionally implemented by derived classes that is used for 204 :cpp:func:`ConservativeReadLimit` and :cpp:func:`ConservativeWriteLimit`. 205 The default implementation returns ``kUnlimited`` or ``0`` depending on 206 whether the stream is readable/writable. 207 208Reader interfaces 209----------------- 210.. cpp:class:: Reader : public Stream 211 212 A Stream that supports writing but not reading. The Write() method is hidden. 213 214 Use in APIs when: 215 * Must read from, but not write to, a stream. 216 * May or may not need seeking. Use a SeekableReader& if seeking is 217 required. 218 219 Inherit from when: 220 * Reader cannot be extended directly. Instead, extend SeekableReader, 221 NonSeekableReader, or (rarely) RelativeSeekableReader, as appropriate. 222 223 A Reader may or may not support seeking. Check seekable() or try calling 224 Seek() to determine if the stream is seekable. 225 226.. cpp:class:: SeekableReader : public RelativeSeekableReader 227 228 A Reader that fully supports seeking. 229 230 Use in APIs when: 231 * Absolute seeking is required. Use Reader& if seeking is not required or 232 seek failures can be handled gracefully. 233 234 Inherit from when: 235 * Implementing a reader that supports absolute seeking. 236 237.. cpp:class:: RelativeSeekableReader : public Reader 238 239 A Reader that at least partially supports seeking. Seeking within some range 240 of the current position works, but seeking beyond that or from other origins 241 may or may not be supported. The extent to which seeking is possible is NOT 242 exposed by this API. 243 244 Use in APIs when: 245 * Relative seeking is required. Usage in APIs should be rare; generally 246 Reader should be used instead. 247 248 Inherit from when: 249 * Implementing a Reader that can only support seeking near the current 250 position. 251 252 A buffered Reader that only supports seeking within its buffer is a good 253 example of a RelativeSeekableReader. 254 255.. cpp:class:: NonSeekableReader : public Reader 256 257 A Reader that does not support seeking. The Seek() method is hidden. 258 259 Use in APIs when: 260 * Do NOT use in APIs! If seeking is not required, use Reader& instead. 261 262 Inherit from when: 263 * Implementing a Reader that does not support seeking. 264 265Writer interfaces 266----------------- 267.. cpp:class:: Writer : public Stream 268 269 A Stream that supports writing but not reading. The Read() method is hidden. 270 271 Use in APIs when: 272 * Must write to, but not read from, a stream. 273 * May or may not need seeking. Use a SeekableWriter& if seeking is 274 required. 275 276 Inherit from when: 277 * Writer cannot be extended directly. Instead, extend SeekableWriter, 278 NonSeekableWriter, or (rarely) RelativeSeekableWriter, as appropriate. 279 280 A Writer may or may not support seeking. Check seekable() or try calling 281 Seek() to determine if the stream is seekable. 282 283.. cpp:class:: SeekableWriter : public RelativeSeekableWriter 284 285 A Writer that fully supports seeking. 286 287 Use in APIs when: 288 * Absolute seeking is required. Use Writer& if seeking is not required or 289 seek failures can be handled gracefully. 290 291 Inherit from when: 292 * Implementing a writer that supports absolute seeking. 293 294 295.. cpp:class:: RelativeSeekableWriter : public Writer 296 297 A Writer that at least partially supports seeking. Seeking within some range 298 of the current position works, but seeking beyond that or from other origins 299 may or may not be supported. The extent to which seeking is possible is NOT 300 exposed by this API. 301 302 Use in APIs when: 303 * Relative seeking is required. Usage in APIs should be rare; generally 304 Writer should be used instead. 305 306 Inherit from when: 307 * Implementing a Writer that can only support seeking near the current 308 position. 309 310 A buffered Writer that only supports seeking within its buffer is a good 311 example of a RelativeSeekableWriter. 312 313.. cpp:class:: NonSeekableWriter : public Writer 314 315 A Writer that does not support seeking. The Seek() method is hidden. 316 317 Use in APIs when: 318 * Do NOT use in APIs! If seeking is not required, use Writer& instead. 319 320 Inherit from when: 321 * Implementing a Writer that does not support seeking. 322 323ReaderWriter interfaces 324----------------------- 325.. cpp:class:: ReaderWriter : public Stream 326 327 A Stream that supports both reading and writing. 328 329 Use in APIs when: 330 * Must both read from and write to a stream. 331 * May or may not need seeking. Use a SeekableReaderWriter& if seeking is 332 required. 333 334 Inherit from when: 335 * Cannot extend ReaderWriter directly. Instead, extend 336 SeekableReaderWriter, NonSeekableReaderWriter, or (rarely) 337 RelativeSeekableReaderWriter, as appropriate. 338 339 A ReaderWriter may or may not support seeking. Check seekable() or try 340 calling Seek() to determine if the stream is seekable. 341 342.. cpp:class:: SeekableReaderWriter : public RelativeSeekableReaderWriter 343 344 A ReaderWriter that fully supports seeking. 345 346 Use in APIs when: 347 * Absolute seeking is required. Use ReaderWriter& if seeking is not 348 required or seek failures can be handled gracefully. 349 350 Inherit from when: 351 * Implementing a writer that supports absolute seeking. 352 353.. cpp:class:: RelativeSeekableReaderWriter : public ReaderWriter 354 355 A ReaderWriter that at least partially supports seeking. Seeking within some 356 range of the current position works, but seeking beyond that or from other 357 origins may or may not be supported. The extent to which seeking is possible 358 is NOT exposed by this API. 359 360 Use in APIs when: 361 * Relative seeking is required. Usage in APIs should be rare; generally 362 ReaderWriter should be used instead. 363 364 Inherit from when: 365 * Implementing a ReaderWriter that can only support seeking near the 366 current position. 367 368 A buffered ReaderWriter that only supports seeking within its buffer is a 369 good example of a RelativeSeekableReaderWriter. 370 371.. cpp:class:: NonSeekableReaderWriter : public ReaderWriter 372 373 A ReaderWriter that does not support seeking. The Seek() method is hidden. 374 375 Use in APIs when: 376 * Do NOT use in APIs! If seeking is not required, use ReaderWriter& 377 instead. 378 379 Inherit from when: 380 * Implementing a ReaderWriter that does not support seeking. 381 382--------------- 383Implementations 384--------------- 385``pw_stream`` includes a few stream implementations for general use. 386 387.. cpp:class:: MemoryWriter : public SeekableWriter 388 389 The ``MemoryWriter`` class implements the :cpp:class:`Writer` interface by 390 backing the data destination with an **externally-provided** memory buffer. 391 ``MemoryWriterBuffer`` extends ``MemoryWriter`` to internally provide a memory 392 buffer. 393 394 The ``MemoryWriter`` can be accessed like a standard C++ container. The 395 contents grow as data is written. 396 397.. cpp:class:: MemoryReader : public SeekableReader 398 399 The ``MemoryReader`` class implements the :cpp:class:`Reader` interface by 400 backing the data source with an **externally-provided** memory buffer. 401 402.. cpp:class:: NullReaderWriter : public SeekableReaderWriter 403 404 ``NullReaderWriter`` is a no-op stream implementation, similar to 405 ``/dev/null``. Writes are always dropped. Reads always return 406 ``OUT_OF_RANGE``. Seeks have no effect. 407 408.. cpp:class:: StdFileWriter : public SeekableWriter 409 410 ``StdFileWriter`` wraps an ``std::ofstream`` with the :cpp:class:`Writer` 411 interface. 412 413.. cpp:class:: StdFileReader : public SeekableReader 414 415 ``StdFileReader`` wraps an ``std::ifstream`` with the :cpp:class:`Reader` 416 interface. 417 418------------------ 419Why use pw_stream? 420------------------ 421 422Standard API 423============ 424``pw_stream`` provides a standard way for classes to express that they have the 425ability to write data. Writing to one sink versus another sink is a matter of 426just passing a reference to the appropriate :cpp:class:`Writer`. 427 428As an example, imagine dumping sensor data. If written against a random HAL 429or one-off class, there's porting work required to write to a different sink 430(imagine writing over UART vs dumping to flash memory). Building a "dumping" 431implementation against the :cpp:class:`Writer` interface prevents a dependency 432on a bespoke API that would require porting work. 433 434Similarly, after building a :cpp:class:`Writer` implementation for a Sink that 435data could be dumped to, that same :cpp:class:`Writer` can be reused for other 436contexts that already write data to the :cpp:class:`pw::stream::Writer` 437interface. 438 439Before: 440 441.. code-block:: cpp 442 443 // Not reusable, depends on `Uart`. 444 void DumpSensorData(Uart& uart) { 445 static char temp[64]; 446 ImuSample imu_sample; 447 imu.GetSample(&info); 448 size_t bytes_written = imu_sample.AsCsv(temp, sizeof(temp)); 449 uart.Transmit(temp, bytes_written, /*timeout_ms=*/ 200); 450 } 451 452After: 453 454.. code-block:: cpp 455 456 // Reusable; no more Uart dependency! 457 Status DumpSensorData(Writer& writer) { 458 static char temp[64]; 459 ImuSample imu_sample; 460 imu.GetSample(&info); 461 size_t bytes_written = imu_sample.AsCsv(temp, sizeof(temp)); 462 return writer.Write(temp, bytes_written); 463 } 464 465Reduce intermediate buffers 466=========================== 467Often functions that write larger blobs of data request a buffer is passed as 468the destination that data should be written to. This *requires* a buffer to be 469allocated, even if the data only exists in that buffer for a very short period 470of time before it's written somewhere else. 471 472In situations where data read from somewhere will immediately be written 473somewhere else, a :cpp:class:`Writer` interface can cut out the middleman 474buffer. 475 476Before: 477 478.. code-block:: cpp 479 480 // Requires an intermediate buffer to write the data as CSV. 481 void DumpSensorData(Uart& uart) { 482 char temp[64]; 483 ImuSample imu_sample; 484 imu.GetSample(&info); 485 size_t bytes_written = imu_sample.AsCsv(temp, sizeof(temp)); 486 uart.Transmit(temp, bytes_written, /*timeout_ms=*/ 200); 487 } 488 489After: 490 491.. code-block:: cpp 492 493 // Both DumpSensorData() and RawSample::AsCsv() use a Writer, eliminating the 494 // need for an intermediate buffer. 495 Status DumpSensorData(Writer& writer) { 496 RawSample imu_sample; 497 imu.GetSample(&info); 498 return imu_sample.AsCsv(writer); 499 } 500 501Prevent buffer overflow 502======================= 503When copying data from one buffer to another, there must be checks to ensure the 504copy does not overflow the destination buffer. As this sort of logic is 505duplicated throughout a codebase, there's more opportunities for bound-checking 506bugs to sneak in. ``Writers`` manage this logic internally rather than pushing 507the bounds checking to the code that is moving or writing the data. 508 509Similarly, since only the :cpp:class:`Writer` has access to any underlying 510buffers, it's harder for functions that share a :cpp:class:`Writer` to 511accidentally clobber data written by others using the same buffer. 512 513Before: 514 515.. code-block:: cpp 516 517 Status BuildPacket(Id dest, span<const std::byte> payload, 518 span<std::byte> dest) { 519 Header header; 520 if (dest.size_bytes() + payload.size_bytes() < sizeof(Header)) { 521 return Status::ResourceExhausted(); 522 } 523 header.dest = dest; 524 header.src = DeviceId(); 525 header.payload_size = payload.size_bytes(); 526 527 memcpy(dest.data(), &header, sizeof(header)); 528 // Forgetting this line would clobber buffer contents. Also, using 529 // a temporary span instead could leave `dest` to be misused elsewhere in 530 // the function. 531 dest = dest.subspan(sizeof(header)); 532 memcpy(dest.data(), payload.data(), payload.size_bytes()); 533 } 534 535After: 536 537.. code-block:: cpp 538 539 Status BuildPacket(Id dest, span<const std::byte> payload, Writer& writer) { 540 Header header; 541 header.dest = dest; 542 header.src = DeviceId(); 543 header.payload_size = payload.size_bytes(); 544 545 writer.Write(header); 546 return writer.Write(payload); 547 } 548 549------------ 550Design notes 551------------ 552 553Sync & Flush 554============ 555The :cpp:class:`pw::stream::Stream` API does not include ``Sync()`` or 556``Flush()`` functions. There no mechanism in the :cpp:class:`Stream` API to 557synchronize a :cpp:class:`Reader`'s potentially buffered input with its 558underlying data source. This must be handled by the implementation if required. 559Similarly, the :cpp:class:`Writer` implementation is responsible for flushing 560any buffered data to the sink. 561 562``Flush()`` and ``Sync()`` were excluded from :cpp:class:`Stream` for a few 563reasons: 564 565 * The semantics of when to call ``Flush()``/``Sync()`` on the stream are 566 unclear. The presence of these methods complicates using a 567 :cpp:class:`Reader` or :cpp:class:`Writer`. 568 * Adding one or two additional virtual calls increases the size of all 569 :cpp:class:`Stream` vtables. 570 571Class hierarchy 572=============== 573All ``pw_stream`` classes inherit from a single, common base with all possible 574functionality: :cpp:class:`pw::stream::Stream`. This structure has 575some similarities with Python's `io module 576<https://docs.python.org/3/library/io.html>`_ and C#'s `Stream class 577<https://docs.microsoft.com/en-us/dotnet/api/system.io.stream>`_. 578 579An alternative approach is to have the reading, writing, and seeking portions of 580the interface provided by different entities. This is how Go's `io 581<https://pkg.go.dev/io package>`_ and C++'s `input/output library 582<https://en.cppreference.com/w/cpp/io>`_ are structured. 583 584We chose to use a single base class for a few reasons: 585 586* The inheritance hierarchy is simple and linear. Despite the linear 587 hierarchy, combining capabilities is natural with classes like 588 :cpp:class:`ReaderWriter`. 589 590 In C++, separate interfaces for each capability requires either a complex 591 virtual inheritance hierarchy or entirely separate hierarchies for each 592 capability. Separate hierarchies can become cumbersome when trying to 593 combine multiple capabilities. A :cpp:class:`SeekableReaderWriter` would 594 have to implement three different interfaces, which means three different 595 vtables and three vtable pointers in each instance. 596* Stream capabilities are clearly expressed in the type system, while 597 naturally supporting optional functionality. A :cpp:class:`Reader` may 598 or may not support :cpp:func:`Stream::Seek`. Applications that can handle 599 seek failures gracefully way use seek on any :cpp:class:`Reader`. If seeking 600 is strictly necessary, an API can accept a :cpp:class:`SeekableReader` 601 instead. 602 603 Expressing optional functionality in the type system is cumbersome when 604 there are distinct interfaces for each capability. ``Reader``, ``Writer``, 605 and ``Seeker`` interfaces would not be sufficient. To match the flexibility 606 of the current structure, there would have to be separate optional versions 607 of each interface, and classes for various combinations. :cpp:class:`Stream` 608 would be an "OptionalReaderOptionalWriterOptionalSeeker" in this model. 609* Code reuse is maximized. For example, a single 610 :cpp:func:`Stream::ConservativeLimit` implementation supports many stream 611 implementations. 612 613Virtual interfaces 614================== 615``pw_stream`` uses virtual functions. Virtual functions enable runtime 616polymorphism. The same code can be used with any stream implementation. 617 618Virtual functions have inherently has more overhead than a regular function 619call. However, this is true of any polymorphic API. Using a C-style ``struct`` 620of function pointers makes different trade-offs but still has more overhead than 621a regular function call. 622 623For many use cases, the overhead of virtual calls insignificant. However, in 624some extremely performance-sensitive contexts, the flexibility of the virtual 625interface may not justify the performance cost. 626 627Asynchronous APIs 628================= 629At present, ``pw_stream`` is synchronous. All :cpp:class:`Stream` API calls are 630expected to block until the operation is complete. This might be undesirable 631for slow operations, like writing to NOR flash. 632 633Pigweed has not yet established a pattern for asynchronous C++ APIs. The 634:cpp:class:`Stream` class may be extended in the future to add asynchronous 635capabilities, or a separate ``AsyncStream`` could be created. 636 637------------ 638Dependencies 639------------ 640 * :ref:`module-pw_assert` 641 * :ref:`module-pw_preprocessor` 642 * :ref:`module-pw_status` 643 * :ref:`module-pw_span` 644 645.. cpp:namespace-pop:: 646 647Zephyr 648====== 649To enable ``pw_stream`` for Zephyr add ``CONFIG_PIGWEED_STREAM=y`` to the 650project's configuration. 651