1# Trace configuration 2 3Unlike many always-on logging systems (e.g. Linux's rsyslog, Android's logcat), 4in Perfetto all tracing data sources are idle by default and record data only 5when instructed to do so. 6 7Data sources record data only when one (or more) tracing sessions are active. 8A tracing session is started by invoking the `perfetto` cmdline client and 9passing a config (see QuickStart guide for 10[Android](/docs/quickstart/android-tracing.md) or 11[Linux](/docs/quickstart/linux-tracing.md)). 12 13A simple trace config looks like this: 14 15```protobuf 16duration_ms: 10000 17 18buffers { 19 size_kb: 65536 20 fill_policy: RING_BUFFER 21} 22 23data_sources { 24 config { 25 name: "linux.ftrace" 26 target_buffer: 0 27 ftrace_config { 28 ftrace_events: "sched_switch" 29 ftrace_events: "sched_wakeup" 30 } 31 } 32} 33 34```` 35 36And is used as follows: 37 38```bash 39perfetto --txt -c config.pbtx -o trace_file.pftrace 40``` 41 42TIP: Some more complete examples of trace configs can be found in the repo in 43[`/test/configs/`](/test/configs/). 44 45## TraceConfig 46 47The TraceConfig is a protobuf message 48([reference docs](/docs/reference/trace-config-proto.autogen)) that defines: 49 501. The general behavior of the whole tracing system, e.g.: 51 * The max duration of the trace. 52 * The number of in-memory buffers and their size. 53 * The max size of the output trace file. 54 552. Which data sources to enable and their configuration, e.g.: 56 * For the [kernel tracing data source](/docs/data-sources/cpu-scheduling.md) 57 , which ftrace events to enable. 58 * For the [heap profiler](/docs/data-sources/native-heap-profiler.md), the 59 target process name and sampling rate. 60 61 See the _data sources_ section of the docs for details on how to 62 configure the data sources bundled with Perfetto. 63 643. The `{data source} x {buffer}` mappings: which buffer each data 65 source should write into (see [buffers section](#buffers) below). 66 67The tracing service (`traced`) acts as a configuration dispatcher: it receives 68a config from the `perfetto` cmdline client (or any other 69[Consumer](/docs/concepts/service-model.md#consumer)) and forwards parts of the 70config to the various [Producers](/docs/concepts/service-model.md#producer) 71connected. 72 73When a tracing session is started by a consumer, the tracing service will: 74 75* Read the outer section of the TraceConfig (e.g. `duration_ms`, `buffers`) and 76 use that to determine its own behavior. 77* Read the list of data sources in the `data_sources` section. For each data 78 source listed in the config, if a corresponding name (`"linux.ftrace"` in the 79 example below) was registered, the service will ask the producer process to 80 start that data source, passing it the raw bytes of the 81 [`DataSourceConfig` subsection][dss] verbatim to the data source (See 82 backward/forward compat section below). 83 84 85 86[dss]: /docs/reference/trace-config-proto.autogen#DataSourceConfig 87 88## Buffers 89 90The buffer sections define the number, size and policy of the in-memory buffers 91owned by the tracing service. It looks as follows: 92 93```protobuf 94// Buffer #0 95buffers { 96 size_kb: 4096 97 fill_policy: RING_BUFFER 98} 99 100// Buffer #1 101buffers { 102 size_kb: 8192 103 fill_policy: DISCARD 104} 105``` 106 107Each buffer has a fill policy which is either: 108 109* RING_BUFFER (default): the buffer behaves like a ring buffer and writes when 110 full will wrap over and replace the oldest trace data in the buffer. 111 112* DISCARD: the buffer stops accepting data once full. Further write attempts are 113 dropped. 114 115WARNING: DISCARD can have unexpected side-effect with data sources that commit 116data at the end of the trace. 117 118A trace config must define at least one buffer to be valid. In the simplest case 119all data sources will write their trace data into the same buffer. 120 121 While this is 122fine for most basic cases, it can be problematic in cases where different data 123sources write at significantly different rates. 124 125For instance, imagine a trace config that enables both: 126 1271. The kernel scheduler tracer. On a typical Android phone this records 128 ~10000 events/second, writing ~1 MB/s of trace data into the buffer. 129 1302. Memory stat polling. This data source writes the contents of /proc/meminfo 131 into the trace buffer and is configured to poll every 5 seconds, writing 132 ~100 KB per poll interval. 133 134If both data sources are configured to write into the same buffer and such 135buffer is set to 4MB, most traces will contain only one memory snapshot. There 136are very good chances that most traces won't contain any memory snapshot at all, 137even if the 2nd data sources was working perfectly. 138This is because during the 5 s. polling interval, the scheduler data source can 139end up filling the whole buffer, pushing the memory snapshot data out of the 140buffer. 141 142## Dynamic buffer mapping 143 144Data-source <> buffer mappings are dynamic in Perfetto. 145In the simplest case a tracing session can define only one buffer. By default, 146all data sources will record data into that one buffer. 147 148In cases like the example above, it might be preferable separating these data 149sources into different buffers. 150This can be achieved with the `target_buffer` field of the TraceConfig. 151 152 153 154Can be achieved with: 155 156```protobuf 157data_sources { 158 config { 159 name: "linux.ftrace" 160 target_buffer: 0 // <-- This goes into buffer 0. 161 ftrace_config { ... } 162 } 163} 164 165data_sources: { 166 config { 167 name: "linux.sys_stats" 168 target_buffer: 1 // <-- This goes into buffer 1. 169 sys_stats_config { ... } 170 } 171} 172 173data_sources: { 174 config { 175 name: "android.heapprofd" 176 target_buffer: 1 // <-- This goes into buffer 1 as well. 177 heapprofd_config { ... } 178 } 179} 180``` 181 182## PBTX vs binary format 183 184There are two ways to pass the trace config when using the `perfetto` cmdline 185client format: 186 187#### Text format 188 189It is the preferred format for human-driven workflows and exploration. It 190allows to pass directly the text file in the PBTX (ProtoBuf TeXtual 191representation) syntax, for the schema defined in the 192[trace_config.proto](/protos/perfetto/config/trace_config.proto) 193(see [reference docs](/docs/reference/trace-config-proto.autogen)) 194 195When using this mode pass the `--txt` flag to `perfetto` to indicate the config 196should be interpreted as a PBTX file: 197 198```bash 199perfetto -c /path/to/config.pbtx --txt -o trace_file.pftrace 200``` 201 202NOTE: The `--txt` option has been introduced only in Android 10 (Q). Older 203versions support only the binary format. 204 205WARNING: Do not use the text format for machine-to-machine interaction 206benchmark, scripts and tools) as it's more prone to breakages (e.g. if a field 207is renamed or an enum is turned into an integer) 208 209#### Binary format 210 211It is the preferred format for machine-to-machine (M2M) interaction. It involves 212passing the protobuf-encoded binary of the TraceConfig message. 213This can be obtained passing the PBTX in input to the protobuf's `protoc` 214compiler (which can be downloaded 215[here](https://github.com/protocolbuffers/protobuf/releases)). 216 217```bash 218cd ~/code/perfetto # external/perfetto in the Android tree. 219 220protoc --encode=perfetto.protos.TraceConfig \ 221 -I. protos/perfetto/config/perfetto_config.proto \ 222 < config.txpb \ 223 > config.bin 224``` 225 226and then passing it to perfetto as follows, without the `--txt` argument: 227 228```bash 229perfetto -c config.bin -o trace_file.pftrace 230``` 231 232## {#long-traces} Streaming long traces 233 234By default Perfetto keeps the full trace buffer(s) in memory and writes it into 235the destination file (the `-o` cmdline argument) only at the end of the tracing 236session. This is to reduce the perf-intrusiveness of the tracing system. 237This, however, limits the max size of the trace to the physical memory size of 238the device, which is often too limiting. 239 240In some cases (e.g., benchmarks, hard to repro cases) it is desirable to capture 241traces that are way larger than that, at the cost of extra I/O overhead. 242 243To achieve that, Perfetto allows to periodically write the trace buffers into 244the target file (or stdout) using the following TraceConfig fields: 245 246* `write_into_file (bool)`: 247When true periodically drains the trace buffers into the output 248file. When this option is enabled, the userspace buffers need to be just 249big enough to hold tracing data between two write periods. 250The buffer sizing depends on the activity of the device. 251The data rate of a typical trace is ~1-4 MB/s. So a 16MB in-memory buffer can 252hold for up write periods of ~4 seconds before starting to lose data. 253 254* `file_write_period_ms (uint32)`: 255Overrides the default drain period (5s). Shorter periods require a smaller 256userspace buffer but increase the performance intrusiveness of tracing. If 257the period given is less than 100ms, the tracing service will use a period 258of 100ms. 259 260* `max_file_size_bytes (uint64)`: 261If set, stops the tracing session after N bytes have been written. Used to 262cap the size of the trace. 263 264For a complete example of a working trace config in long-tracing mode see 265[`/test/configs/long_trace.cfg`](/test/configs/long_trace.cfg). 266 267Summary: to capture a long trace just set `write_into_file:true`, set a long 268 `duration_ms` and use an in-memory buffer size of 32MB or more. 269 270## Data-source specific config 271 272Alongside the trace-wide configuration parameters, the trace config also defines 273data-source-specific behaviors. At the proto schema level, this is defined in 274the `DataSourceConfig` section of `TraceConfig`: 275 276From [data_source_config.proto](/protos/perfetto/config/data_source_config.proto): 277 278```protobuf 279message TraceConfig { 280 ... 281 repeated DataSource data_sources = 2; // See below. 282} 283 284message DataSource { 285 optional protos.DataSourceConfig config = 1; // See below. 286 ... 287} 288 289message DataSourceConfig { 290 optional string name = 1; 291 ... 292 optional FtraceConfig ftrace_config = 100 [lazy = true]; 293 ... 294 optional AndroidPowerConfig android_power_config = 106 [lazy = true]; 295} 296``` 297 298Fields like `ftrace_config`, `android_power_config` are examples of data-source 299specific configs. The tracing service will completely ignore the contents of 300those fields and route the whole DataSourceConfig object to any data source 301registered with the same name. 302 303The `[lazy=true]` marker has a special implication in the 304[protozero](/docs/design-docs/protozero.md) code generator. Unlike standard 305nested messages, it generates raw accessors (e.g., 306`const std::string& ftrace_config_raw()` instead of 307`const protos::FtraceConfig& ftrace_config()`). This is to avoid injecting too 308many `#include` dependencies and avoiding binary size bloat in the code that 309implements data sources. 310 311#### A note on backwards/forward compatibility 312The tracing service will route the raw binary blob of the `DataSourceConfig` 313message to the data sources with a matching name, without attempting to decode 314and re-encode it. If the `DataSourceConfig` section of the trace config contains 315a new field that didn't exist at the time when the service was built, the 316service will still pass the `DataSourceConfig` through to the data source. 317This allows to introduced new data sources without needing the service to 318know anything about them upfront. 319 320TODO: we are aware of the fact that today extending the `DataSourceConfig` with 321a custom proto requires changing the `data_source_config.proto` in the Perfetto 322repo, which is unideal for external projects. The long-term plan is to reserve 323a range of fields for non-upstream extensions and provide generic templated 324accessors for client code. Until then, we accept patches upstream to introduce 325ad-hoc configurations for your own data sources. 326 327## Multi-process data sources 328 329Some data sources are singletons. E.g., in the case of scheduler tracing that 330Perfetto ships on Android, there is only data source for the whole system, 331owned by the `traced_probes` service. 332 333However, in the general case multiple processes can advertise the same data 334source. This is the case, for instance, when using the 335[Perfetto SDK](/docs/instrumentation/tracing-sdk.md) for userspace 336instrumentation. 337 338If this happens, when starting a tracing session that specifies that data 339source in the trace config, Perfetto by default will ask all processes that 340advertise that data source to start it. 341 342In some cases it might be desirable to further limit the enabling of the data 343source to a specific process (or set of processes). That is possible through the 344`producer_name_filter` and `producer_name_regex_filter`. 345 346NOTE: the typical Perfetto run-time model is: one process == one Perfetto 347 Producer; one Producer typically hosts multiple data sources. 348 349When those filters are set, the Perfetto tracing service will activate the data 350source only in the subset of producers matching the filter. 351 352Example: 353 354```protobuf 355buffers { 356 size_kb: 4096 357} 358 359data_sources { 360 config { 361 name: "track_event" 362 363 # Enable the data source only on Chrome and Chrome canary. 364 producer_name_filter: "com.android.chrome" 365 producer_name_filter: "com.google.chrome.canary" 366 } 367} 368``` 369 370## Triggers 371 372In nominal conditions, a tracing session has a lifecycle that simply matches the 373invocation of the `perfetto` cmdline client: trace data recording starts when 374the TraceConfig is passed to `perfetto` and ends when either the 375`TraceConfig.duration_ms` has elapsed, or when the cmdline client terminates. 376 377Perfetto supports an alternative mode of either starting or stopping the trace 378which is based on triggers. The overall idea is to declare in the trace config 379itself: 380 381* A set of triggers, which are just free-form strings. 382* Whether a given trigger should cause the trace to be started or stopped, and 383 the start/stop delay. 384 385Why using triggers? Why can't one just start perfetto or kill(SIGTERM) it when 386needed? The rationale of all this is the security model: in most Perfetto 387deployments (e.g., on Android) only privileged entities (e.g., adb shell) can 388configure/start/stop tracing. Apps are unprivileged in this sense and they 389cannot control tracing. 390 391Triggers offer a way to unprivileged apps to control, in a limited fashion, the 392lifecycle of a tracing session. The conceptual model is: 393 394* The privileged Consumer (see 395 [_Service model_](/docs/concepts/service-model.md)), i.e. the entity 396 that is normally authorized to start tracing (e.g., adb shell in Android), 397 declares upfront what are the possible trigger names for the trace and what 398 they will do. 399* Unprivileged entities (any random app process) can activate those triggers. 400 Unprivileged entities don't get a say on what the triggers will do, they only 401 communicate that an event happened. 402 403Triggers can be signaled via the cmdline util 404 405```bash 406/system/bin/trigger_perfetto "trigger_name" 407``` 408 409(or also by starting an independent trace session which uses only the 410`activate_triggers: "trigger_name"` field in the config) 411 412There are two types of triggers: 413 414#### Start triggers 415 416Start triggers allow activating a tracing session only after some significant 417event has happened. Passing a trace config that has `START_TRACING` trigger 418causes the tracing session to stay idle (i.e. not recording any data) until either 419the trigger is hit or the `duration_ms` timeout is hit. 420 421Example config: 422```protobuf 423// If no trigger is hit, the trace will end without having recorded any data 424// after 30s. 425duration_ms: 30000 426 427// If the "myapp_is_slow" is hit, the trace starts recording data and will be 428// stopped after 5s. 429trigger_config { 430 trigger_mode: START_TRACING 431 triggers { 432 name: "myapp_is_slow" 433 stop_delay_ms: 5000 434 } 435} 436 437// The rest of the config is as usual. 438buffers { ... } 439data_sources { ... } 440``` 441 442#### Stop triggers 443 444STOP_TRACING triggers allow to prematurely finalize a trace when the trigger is 445hit. In this mode the trace starts immediately when the `perfetto` client is 446invoked (like in nominal cases). The trigger acts as a premature finalization 447signal. 448 449This can be used to use perfetto in flight-recorder mode. By starting a trace 450with buffers configured in `RING_BUFFER` mode and `STOP_TRACING` triggers, 451the trace will be recorded in a loop and finalized when the culprit event is 452detected. This is key for events where the root cause is in the recent past 453(e.g., the app detects a slow scroll or a missing frame). 454 455Example config: 456```protobuf 457// If no trigger is hit, the trace will end after 30s. 458duration_ms: 30000 459 460// If the "missed_frame" is hit, the trace is stopped after 1s. 461trigger_config { 462 trigger_mode: STOP_TRACING 463 triggers { 464 name: "missed_frame" 465 stop_delay_ms: 1000 466 } 467} 468 469// The rest of the config is as usual. 470buffers { ... } 471data_sources { ... } 472``` 473 474## Other resources 475 476* [TraceConfig Reference](/docs/reference/trace-config-proto.autogen) 477* [Buffers and dataflow](/docs/concepts/buffers.md) 478