1.. _module-pw_log_tokenized: 2 3---------------- 4pw_log_tokenized 5---------------- 6.. pigweed-module:: 7 :name: pw_log_tokenized 8 9The ``pw_log_tokenized`` module contains utilities for tokenized logging. It 10connects ``pw_log`` to ``pw_tokenizer`` and supports 11:ref:`module-pw_log-tokenized-args`. 12 13C++ backend 14=========== 15``pw_log_tokenized`` provides a backend for ``pw_log`` that tokenizes log 16messages with the ``pw_tokenizer`` module. The log level, 16-bit tokenized 17module name, and flags bits are passed through the payload argument. The macro 18eventually passes logs to the :c:func:`pw_log_tokenized_HandleLog` function, 19which must be implemented by the application. 20 21.. doxygenfunction:: pw_log_tokenized_HandleLog 22 23Example implementation: 24 25.. code-block:: cpp 26 27 extern "C" void pw_log_tokenized_HandleLog( 28 uint32_t payload, const uint8_t message[], size_t size) { 29 // The metadata object provides the log level, module token, and flags. 30 // These values can be recorded and used for runtime filtering. 31 pw::log_tokenized::Metadata metadata(payload); 32 33 if (metadata.level() < current_log_level) { 34 return; 35 } 36 37 if (metadata.flags() & HIGH_PRIORITY_LOG != 0) { 38 EmitHighPriorityLog(metadata.module(), message, size); 39 } else { 40 EmitLowPriorityLog(metadata.module(), message, size); 41 } 42 } 43 44See the documentation for :ref:`module-pw_tokenizer` for further details. 45 46Metadata in the format string 47----------------------------- 48With tokenized logging, the log format string is converted to a 32-bit token. 49Regardless of how long the format string is, it's always represented by a 32-bit 50token. Because of this, metadata can be packed into the tokenized string with 51no cost. 52 53``pw_log_tokenized`` uses a simple key-value format to encode metadata in a 54format string. Each field starts with the ``■`` (U+25A0 "Black Square") 55character, followed by the key name, the ``♦`` (U+2666 "Black Diamond Suit") 56character, and then the value. The string is encoded as UTF-8. Key names are 57comprised of alphanumeric ASCII characters and underscore and start with a 58letter. 59 60.. code-block:: 61 62 "■key1♦contents1■key2♦contents2■key3♦contents3" 63 64This format makes the message easily machine parseable and human readable. It is 65extremely unlikely to conflict with log message contents due to the characters 66used. 67 68``pw_log_tokenized`` uses three fields: ``msg``, ``module``, and ``file``. 69Implementations may add other fields, but they will be ignored by the 70``pw_log_tokenized`` tooling. 71 72.. code-block:: 73 74 "■msg♦Hyperdrive %d set to %f■module♦engine■file♦propulsion/hyper.cc" 75 76Using key-value pairs allows placing the fields in any order. 77``pw_log_tokenized`` places the message first. This is prefered when tokenizing 78C code because the tokenizer only hashes a fixed number of characters. If the 79file were first, the long path might take most of the hashed characters, 80increasing the odds of a collision with other strings in that file. In C++, all 81characters in the string are hashed, so the order is not important. 82 83The format string is created by the :c:macro:`PW_LOG_TOKENIZED_FORMAT_STRING` 84macro. 85 86.. doxygendefine:: PW_LOG_TOKENIZED_FORMAT_STRING 87 88Metadata in the tokenizer payload argument 89------------------------------------------- 90``pw_log_tokenized`` packs runtime-accessible metadata into a 32-bit integer 91which is passed as the "payload" argument for ``pw_log_tokenizer``'s global 92handler with payload facade. Packing this metadata into a single word rather 93than separate arguments reduces the code size significantly. 94 95Four items are packed into the payload argument: 96 97- Log level -- Used for runtime log filtering by level. 98- Line number -- Used to track where a log message originated. 99- Log flags -- Implementation-defined log flags. 100- Tokenized :c:macro:`PW_LOG_MODULE_NAME` -- Used for runtime log filtering by 101 module. 102 103Configuring metadata bit fields 104^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 105The number of bits to use for each metadata field is configurable through macros 106in ``pw_log/config.h``. The field widths must sum to 32 bits. A field with zero 107bits allocated is excluded from the log metadata. 108 109.. doxygendefine:: PW_LOG_TOKENIZED_LEVEL_BITS 110.. doxygendefine:: PW_LOG_TOKENIZED_LINE_BITS 111.. doxygendefine:: PW_LOG_TOKENIZED_FLAG_BITS 112.. doxygendefine:: PW_LOG_TOKENIZED_MODULE_BITS 113 114Creating and reading Metadata payloads 115^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 116``pw_log_tokenized`` provides a C++ class to facilitate the creation and 117interpretation of packed log metadata payloads. 118 119.. doxygenclass:: pw::log_tokenized::GenericMetadata 120.. doxygentypedef:: pw::log_tokenized::Metadata 121 122The following example shows that a ``Metadata`` object can be created from a 123``uint32_t`` log metadata payload. 124 125.. code-block:: cpp 126 127 extern "C" void pw_log_tokenized_HandleLog( 128 uint32_t payload, 129 const uint8_t message[], 130 size_t size_bytes) { 131 pw::log_tokenized::Metadata metadata = payload; 132 // Check the log level to see if this log is a crash. 133 if (metadata.level() == PW_LOG_LEVEL_FATAL) { 134 HandleCrash(metadata, pw::ConstByteSpan( 135 reinterpret_cast<const std::byte*>(message), size_bytes)); 136 PW_UNREACHABLE; 137 } 138 // ... 139 } 140 141It's also possible to get a ``uint32_t`` representation of a ``Metadata`` 142object: 143 144.. code-block:: cpp 145 146 // Logs an explicitly created string token. 147 void LogToken(uint32_t token, int level, int line_number, int module) { 148 const uint32_t payload = 149 log_tokenized::Metadata( 150 level, module, PW_LOG_FLAGS, line_number) 151 .value(); 152 std::array<std::byte, sizeof(token)> token_buffer = 153 pw::bytes::CopyInOrder(endian::little, token); 154 155 pw_log_tokenized_HandleLog( 156 payload, 157 reinterpret_cast<const uint8_t*>(token_buffer.data()), 158 token_buffer.size()); 159 } 160 161The binary tokenized message may be encoded in the :ref:`prefixed Base64 format 162<module-pw_tokenizer-base64-format>` with the following function: 163 164.. doxygenfunction:: PrefixedBase64Encode(span<const std::byte>) 165 166Build targets 167------------- 168The GN build for ``pw_log_tokenized`` has two targets: ``pw_log_tokenized`` and 169``log_backend``. The ``pw_log_tokenized`` target provides the 170``pw_log_tokenized/log_tokenized.h`` header. The ``log_backend`` target 171implements the backend for the ``pw_log`` facade. ``pw_log_tokenized`` invokes 172the ``pw_log_tokenized:handler`` facade, which must be implemented by the user 173of ``pw_log_tokenized``. 174 175GCC has a bug resulting in section attributes of templated functions being 176ignored. This in turn means that log tokenization cannot work for templated 177functions, because the token database entries are lost at build time. 178For more information see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70435. 179If you are using GCC, the ``gcc_partially_tokenized`` target can be used as a 180backend for the ``pw_log`` facade instead which tokenizes as much as possible 181and uses the ``pw_log_string:handler`` for the rest using string logging. 182 183Python package 184============== 185``pw_log_tokenized`` includes a Python package for decoding tokenized logs. 186 187pw_log_tokenized 188---------------- 189.. automodule:: pw_log_tokenized 190 :members: 191 :undoc-members: 192