1.. _module-pw_trace_tokenized: 2 3================== 4pw_trace_tokenized 5================== 6Pigweed's tracing module provides facilities for applications to trace 7information about the execution of their application. The module is split into 8two components: 9 101. The facade, provided elsewhere, which is only a macro interface layer 112. The backend (this module), is one implemention of the low level tracing. 12 13------ 14Status 15------ 16This module is currently in development, and is therefore still undergoing 17significant changes. 18 19Future work will: 20 211. Add a more complete API for how to retrieve data from ring_buffer. 222. Add a Python library to decode the trace data. 233. Add examples with sample output (especially for filtering and triggering). 244. Add tools to retrieve trace data. 255. Add more sinks, such as RTT. 266. Add support to more platforms. 277. Improve the locking behaviour and provide default trace locking 28 implementions. 29 30-------- 31Overview 32-------- 33The tokenized trace backend aims to be a reasonable tradeoff of trace features 34and event size for most applications. It works by encoding all compile time data 35for a trace event into a tokenized number. This provides a good amount of 36compression, while maintaining the full trace feature set. 37 38In addition the tokenized trace backend adds flexibility through callbacks, 39which allows the application to do things such as filtering trace_events and 40triggering tracing to turn on and off. This flexibility can help maximize the 41effectiveness of a limited trace buffer as well as be a valuable tool while 42debugging. 43 44 45Compatibility 46------------- 47Most of this module is compatible with C and C++, the only exception to this is 48the ``RegisterCallbackWhenCreated`` helper class. 49 50Dependencies 51------------ 52``pw_assert`` 53``pw_log`` 54``pw_preprocessor`` 55``pw_status`` 56``pw_tokenizer`` 57``pw_trace:facade`` 58``pw_varint`` 59 60--------- 61Macro API 62--------- 63All code should use the trace API facade directly. This backend fully 64implements all features of the tracing facade. 65 66 67Event Callbacks & Data Sinks 68---------------------------- 69The tokenized trace module adds both event callbacks and data sinks which 70provide hooks into tracing. 71 72The *event callbacks* are called when trace events occur, with the trace event 73data, before the event is encoded or sent to the sinks. The callbacks may 74modify the run-time fields of the trace event, i.e. ``trace_id``, 75``data_buffer`` and ``data_size``. Using the return flags, these callbacks can 76be used to adjust the trace behaviour at runtime in response to specific events. 77 78If requested (using ``called_on_every_event``) the callback will be called on 79every trace event regardless if tracing is currently enabled or not. Using this, 80the application can trigger tracing on or off when specific traces or patterns 81of traces are observed, or can selectively filter traces to preserve the trace 82buffer. 83 84The event callback is called in the context of the traced task. It must be 85ISR-safe to support tracing within ISRs. It must be lightweight to prevent 86performance issues in the trace tasks. 87 88The return flags ``pw_trace_TraceEventReturnFlags`` support the following 89behaviors: 90 91* ``PW_TRACE_EVENT_RETURN_FLAGS_SKIP_EVENT`` can be set true to skip this 92 sample. 93* ``PW_TRACE_EVENT_RETURN_FLAGS_DISABLE_AFTER_PROCESSING`` can be set true to 94 disable tracing after this sample. 95 96.. cpp:function:: pw_trace_TraceEventReturnFlags pw_trace_EventCallback( \ 97 void* user_data, \ 98 pw_trace_tokenized_TraceEvent* event) 99.. cpp:function:: pw_Status pw_trace_RegisterEventCallback( \ 100 pw_trace_EventCallback callback, \ 101 pw_trace_EventCallbackFlags flags, \ 102 void* user_data, \ 103 pw_trace_EventCallbackHandle* handle) 104.. cpp:function:: pw_Status pw_trace_UnregisterEventCallback( \ 105 pw_trace_EventCallbackHandle handle) 106 107 108The *data sinks* are called only for trace events which get processed (tracing 109is enabled, and the sample not skipped). The sink callback is called with the 110encoded bytes of the trace event, which can be used by the application to 111connect different data sinks. The callback is broken into three callbacks 112``pw_trace_SinkStartBlock``, ``pw_trace_SinkAddBytes``, and 113``pw_trace_SinkEndBlock``. ``Start`` is called with the size of the block, 114before any bytes are emitted and can be used if needed to allocate space. 115``AddBytes`` is then called multiple times with chunks of bytes. Finally ``End`` 116is called to allow any cleanup to be done by the sink if neccessary. Not all 117callbacks are required, it is acceptible to provide nullptr for any callbacks 118which you don't require. 119 120.. cpp:function:: void pw_trace_SinkStartBlock(void* user_data, size_t size) 121.. cpp:function:: void pw_trace_SinkAddBytes( \ 122 void* user_data, \ 123 const void* bytes, \ 124 size_t size) 125.. cpp:function:: void pw_trace_SinkEndBlock(void* user_data) 126.. cpp:function:: pw_Status pw_trace_RegisterSink( \ 127 pw_trace_SinkStartBlock start, \ 128 pw_trace_SinkAddBytes add_bytes, \ 129 pw_trace_SinkEndBlock end_block, \ 130 void* user_data, \ 131 pw_trace_SinkHandle* handle) 132.. cpp:function:: pw_Status pw_trace_UnregisterSink(pw_trace_SinkHandle handle) 133 134Trace Reference 135--------------- 136Some use-cases might involve referencing a specific trace event, for example 137to use it as a trigger or filtering. Since the trace events are tokenized, a 138macro is provided to generate the token to use as a reference. All the fields 139must match exactly to generate the correct trace reference. If the trace does 140not have a group, use ``PW_TRACE_GROUP_LABEL_DEFAULT``. 141 142.. cpp:function:: PW_TRACE_REF(event_type, module, label, flags, group) 143.. cpp:function:: PW_TRACE_REF_DATA( \ 144 event_type, module, label, flags, group, type) 145 146 147----------- 148Time source 149----------- 150Tracing requires the platform to provide the time source for tracing, this can 151be done in one of a few ways. 152 1531. Create a file with the default time functions, and provide as build variable 154 ``pw_trace_tokenized_time``, which will get pulled in as a dependency. 1552. Provide time functions elsewhere in project, and ensure they are included. 1563. Redefine the trace time macros to something else, other then the default 157 trace time functions. 158 159.. cpp:function:: PW_TRACE_TIME_TYPE pw_trace_GetTraceTime() 160.. cpp:function:: PW_TRACE_GET_TIME() 161.. cpp:function:: size_t pw_trace_GetTraceTimeTicksPerSecond() 162.. cpp:function:: PW_TRACE_GET_TIME_TICKS_PER_SECOND() 163 164 165------ 166Buffer 167------ 168The optional trace buffer adds a ring buffer which contains the encoded trace 169data. This is still a work in progress, in particular better methods for 170retrieving the data still need to be added. Currently there is an accessor for 171the underlying ring buffer object, but this is a short term solution. 172 173.. cpp:function:: void ClearBuffer() 174.. cpp:function:: pw::ring_buffer::PrefixedEntryRingBuffer* GetBuffer() 175 176The buffer has two configurable options: 177 1781. PW_TRACE_BUFFER_SIZE_BYTES: The total size of the ring buffer in bytes. 1792. PW_TRACE_BUFFER_MAX_BLOCK_SIZE_BYTES: The maximum single trace object size. 180 Including the token, time, and any attached data. Any trace object larger 181 then this will be dropped. 182 183.. cpp:function:: ConstByteSpan DeringAndViewRawBuffer() 184 185The DeringAndViewRawBuffer function can be used to get bulk access of the full 186deringed prefixed-ring-buffer data. This might be neccessary for large zero-copy 187bulk transfers. It is the caller's responsibility to disable tracing during 188access to the buffer. The data in the block is defined by the 189prefixed-ring-buffer format without any user-preamble. 190 191 192Added dependencies 193------------------ 194``pw_ring_buffer`` 195``pw_varint`` 196 197 198------- 199Logging 200------- 201The optional trace buffer logging adds support to dump trace buffers to the log. 202Buffers are converted to base64-encoding then split across log lines. Trace logs 203are surrounded by 'begin' and 'end' tags. 204 205Ex. Invoking PW_TRACE_INSTANT with 'test1' and 'test2', then calling this 206function would produce this in the output logs: 207 208.. code:: sh 209 210 [TRACE] begin 211 [TRACE] data: BWdDMRoABWj52YMB 212 [TRACE] end 213 214Added dependencies 215------------------ 216``pw_base64`` 217``pw_log`` 218``pw_ring_buffer`` 219``pw_string`` 220``pw_tokenizer`` 221``pw_varint`` 222 223-------------- 224Python decoder 225-------------- 226The python decoder can be used to convert the binary trace data into json data 227which can be viewed in chrome://tracing. 228 229``get_trace.py`` can be used for retrieveing trace data from devices which are 230using the trace_rpc_server. 231 232``trace_tokenized.py`` can be used to decode a binary file of trace data. 233 234-------- 235Examples 236-------- 237The examples all use `pw_trace` sample app to provide the trace data. Details 238for how to build, run, and decode the traces are included at the top of each 239example. This is early work, and is provided as an example of how different 240tracing concepts can look. 241 242Basic 243----- 244The basic example turns on tracing and dumps all trace output to a file provided 245on the command line. 246 247Trigger 248------- 249The trigger example demonstrates how a trace event can be used as a trigger to 250start and stop capturing a trace. The examples makes use of ``PW_TRACE_REF`` 251and ``PW_TRACE_REF_DATA`` to specify a start and stop event for the capture. 252This can be useful if the trace buffer is small and you wish to capture a 253specific series of events. 254 255Filter 256------ 257The filter example demonstrates how a callback can be used to filter which trace 258events get processed and saved. In this example all events from the processing 259task which don't have traceId equal to 3 are removed. Both the other task traces 260are not removed. This can be a useful feature while debugging as it limits the 261amount of events which get stored to the buffer, and only saves the events of 262interest. 263 264-------------------- 265Snapshot integration 266-------------------- 267Tokenized trace buffers can be captured to a ``pw.snapshot.Snapshot`` or 268``pw.trace.SnapshotTraceInfo`` proto in the ``trace_data`` field. The expected 269format is a de-ringed raw tokenized trace buffer, which can be retrieved via 270``pw::trace::DeringAndViewRawBuffer()``. 271 272``pw_trace_tokenized`` does not yet have Python tooling integration for 273interpretation of serialized snapshots with a populated ``trace_data`` field. 274