1.. _module-pw_log_tokenized: 2 3---------------- 4pw_log_tokenized 5---------------- 6The ``pw_log_tokenized`` module contains utilities for tokenized logging. It 7connects ``pw_log`` to ``pw_tokenizer`` and supports 8:ref:`module-pw_log-tokenized-args`. 9 10C++ backend 11=========== 12``pw_log_tokenized`` provides a backend for ``pw_log`` that tokenizes log 13messages with the ``pw_tokenizer`` module. The log level, 16-bit tokenized 14module name, and flags bits are passed through the payload argument. The macro 15eventually passes logs to the :c:func:`pw_log_tokenized_HandleLog` function, 16which must be implemented by the application. 17 18.. doxygenfunction:: pw_log_tokenized_HandleLog 19 20Example implementation: 21 22.. code-block:: cpp 23 24 extern "C" void pw_log_tokenized_HandleLog( 25 uint32_t payload, const uint8_t message[], size_t size) { 26 // The metadata object provides the log level, module token, and flags. 27 // These values can be recorded and used for runtime filtering. 28 pw::log_tokenized::Metadata metadata(payload); 29 30 if (metadata.level() < current_log_level) { 31 return; 32 } 33 34 if (metadata.flags() & HIGH_PRIORITY_LOG != 0) { 35 EmitHighPriorityLog(metadata.module(), message, size); 36 } else { 37 EmitLowPriorityLog(metadata.module(), message, size); 38 } 39 } 40 41See the documentation for :ref:`module-pw_tokenizer` for further details. 42 43Metadata in the format string 44----------------------------- 45With tokenized logging, the log format string is converted to a 32-bit token. 46Regardless of how long the format string is, it's always represented by a 32-bit 47token. Because of this, metadata can be packed into the tokenized string with 48no cost. 49 50``pw_log_tokenized`` uses a simple key-value format to encode metadata in a 51format string. Each field starts with the ``■`` (U+25A0 "Black Square") 52character, followed by the key name, the ``♦`` (U+2666 "Black Diamond Suit") 53character, and then the value. The string is encoded as UTF-8. Key names are 54comprised of alphanumeric ASCII characters and underscore and start with a 55letter. 56 57.. code-block:: 58 59 "■key1♦contents1■key2♦contents2■key3♦contents3" 60 61This format makes the message easily machine parseable and human readable. It is 62extremely unlikely to conflict with log message contents due to the characters 63used. 64 65``pw_log_tokenized`` uses three fields: ``msg``, ``module``, and ``file``. 66Implementations may add other fields, but they will be ignored by the 67``pw_log_tokenized`` tooling. 68 69.. code-block:: 70 71 "■msg♦Hyperdrive %d set to %f■module♦engine■file♦propulsion/hyper.cc" 72 73Using key-value pairs allows placing the fields in any order. 74``pw_log_tokenized`` places the message first. This is prefered when tokenizing 75C code because the tokenizer only hashes a fixed number of characters. If the 76file were first, the long path might take most of the hashed characters, 77increasing the odds of a collision with other strings in that file. In C++, all 78characters in the string are hashed, so the order is not important. 79 80Metadata in the tokenizer payload argument 81------------------------------------------- 82``pw_log_tokenized`` packs runtime-accessible metadata into a 32-bit integer 83which is passed as the "payload" argument for ``pw_log_tokenizer``'s global 84handler with payload facade. Packing this metadata into a single word rather 85than separate arguments reduces the code size significantly. 86 87Four items are packed into the payload argument: 88 89- Log level -- Used for runtime log filtering by level. 90- Line number -- Used to track where a log message originated. 91- Log flags -- Implementation-defined log flags. 92- Tokenized :c:macro:`PW_LOG_MODULE_NAME` -- Used for runtime log filtering by 93 module. 94 95Configuring metadata bit fields 96^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 97The number of bits to use for each metadata field is configurable through macros 98in ``pw_log/config.h``. The field widths must sum to 32 bits. A field with zero 99bits allocated is excluded from the log metadata. 100 101.. c:macro:: PW_LOG_TOKENIZED_LEVEL_BITS 102 103 Bits to allocate for the log level. Defaults to :c:macro:`PW_LOG_LEVEL_BITS` 104 (3). 105 106.. c:macro:: PW_LOG_TOKENIZED_LINE_BITS 107 108 Bits to allocate for the line number. Defaults to 11 (up to line 2047). If the 109 line number is too large to be represented by this field, line is reported as 110 0. 111 112 Including the line number can slightly increase code size. Without the line 113 number, the log metadata argument is the same for all logs with the same level 114 and flags. With the line number, each metadata value is unique and must be 115 encoded as a separate word in the binary. Systems with extreme space 116 constraints may exclude line numbers by setting this macro to 0. 117 118 It is possible to include line numbers in tokenized log format strings, but 119 that is discouraged because line numbers change whenever a file is edited. 120 Passing the line number with the metadata is a lightweight way to include it. 121 122.. c:macro:: PW_LOG_TOKENIZED_FLAG_BITS 123 124 Bits to use for implementation-defined flags. Defaults to 2. 125 126.. c:macro:: PW_LOG_TOKENIZED_MODULE_BITS 127 128 Bits to use for the tokenized version of :c:macro:`PW_LOG_MODULE_NAME`. 129 Defaults to 16, which gives a ~1% probability of a collision with 37 module 130 names. 131 132Creating and reading Metadata payloads 133^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 134``pw_log_tokenized`` provides a C++ class to facilitate the creation and 135interpretation of packed log metadata payloads. 136 137.. doxygenclass:: pw::log_tokenized::GenericMetadata 138.. doxygentypedef:: pw::log_tokenized::Metadata 139 140The following example shows that a ``Metadata`` object can be created from a 141``uint32_t`` log metadata payload. 142 143.. code-block:: cpp 144 145 extern "C" void pw_log_tokenized_HandleLog( 146 uint32_t payload, 147 const uint8_t message[], 148 size_t size_bytes) { 149 pw::log_tokenized::Metadata metadata = payload; 150 // Check the log level to see if this log is a crash. 151 if (metadata.level() == PW_LOG_LEVEL_FATAL) { 152 HandleCrash(metadata, pw::ConstByteSpan( 153 reinterpret_cast<const std::byte*>(message), size_bytes)); 154 PW_UNREACHABLE; 155 } 156 // ... 157 } 158 159It's also possible to get a ``uint32_t`` representation of a ``Metadata`` 160object: 161 162.. code-block:: cpp 163 164 // Logs an explicitly created string token. 165 void LogToken(uint32_t token, int level, int line_number, int module) { 166 const uint32_t payload = 167 log_tokenized::Metadata( 168 level, module, PW_LOG_FLAGS, line_number) 169 .value(); 170 std::array<std::byte, sizeof(token)> token_buffer = 171 pw::bytes::CopyInOrder(endian::little, token); 172 173 pw_log_tokenized_HandleLog( 174 payload, 175 reinterpret_cast<const uint8_t*>(token_buffer.data()), 176 token_buffer.size()); 177 } 178 179The binary tokenized message may be encoded in the :ref:`prefixed Base64 format 180<module-pw_tokenizer-base64-format>` with the following function: 181 182.. doxygenfunction:: PrefixedBase64Encode(span<const std::byte>) 183 184Build targets 185------------- 186The GN build for ``pw_log_tokenized`` has two targets: ``pw_log_tokenized`` and 187``log_backend``. The ``pw_log_tokenized`` target provides the 188``pw_log_tokenized/log_tokenized.h`` header. The ``log_backend`` target 189implements the backend for the ``pw_log`` facade. ``pw_log_tokenized`` invokes 190the ``pw_log_tokenized:handler`` facade, which must be implemented by the user 191of ``pw_log_tokenized``. 192 193GCC has a bug resulting in section attributes of templated functions being 194ignored. This in turn means that log tokenization cannot work for templated 195functions, because the token database entries are lost at build time. 196For more information see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70435. 197If you are using GCC, the ``gcc_partially_tokenized`` target can be used as a 198backend for the ``pw_log`` facade instead which tokenizes as much as possible 199and uses the ``pw_log_string:handler`` for the rest using string logging. 200 201Python package 202============== 203``pw_log_tokenized`` includes a Python package for decoding tokenized logs. 204 205pw_log_tokenized 206---------------- 207.. automodule:: pw_log_tokenized 208 :members: 209 :undoc-members: 210