1# Trace Processor 2 3_The Trace Processor is a C++ library 4([/src/trace_processor](/src/trace_processor)) that ingests traces encoded in a 5wide variety of formats and exposes an SQL interface for querying trace events 6contained in a consistent set of tables. It also has other features including 7computation of summary metrics, annotating the trace with user-friendly 8descriptions and deriving new events from the contents of the trace._ 9 10 11 12## Quickstart 13 14The [quickstart](/docs/quickstart/trace-analysis.md) provides a quick overview 15on how to run SQL queries against traces using trace processor. 16 17## Introduction 18 19Events in a trace are optimized for fast, low-overhead recording. Therefore 20traces need significant data processing to extract meaningful information from 21them. This is compounded by the number of legacy formats which are still in use and 22need to be supported in trace analysis tools. 23 24The trace processor abstracts this complexity by parsing traces, extracting the 25data inside, and exposing it in a set of database tables which can be queried 26with SQL. 27 28Features of the trace processor include: 29 30* Execution of SQL queries on a custom, in-memory, columnar database backed by 31 the SQLite query engine. 32* Metrics subsystem which allows computation of summarized view of the trace 33 (e.g. CPU or memory usage of a process, time taken for app startup etc.). 34* Annotating events in the trace with user-friendly descriptions, providing 35 context and explanation of events to newer users. 36* Creation of new events derived from the contents of the trace. 37 38The formats supported by trace processor include: 39 40* Perfetto native protobuf format 41* Linux ftrace 42* Android systrace 43* Chrome JSON (including JSON embedding Android systrace text) 44* Fuchsia binary format 45* [Ninja](https://ninja-build.org/) logs (the build system) 46 47The trace processor is embedded in a wide variety of trace analysis tools, including: 48 49* [trace_processor](/docs/analysis/trace-processor.md), a standalone binary 50 providing a shell interface (and the reference embedder). 51* [Perfetto UI](https://ui.perfetto.dev), in the form of a WebAssembly module. 52* [Android Graphics Inspector](https://gpuinspector.dev/). 53* [Android Studio](https://developer.android.com/studio/). 54 55## Concepts 56 57The trace processor has some foundational terminology and concepts which are 58used in the rest of documentation. 59 60### Events 61 62In the most general sense, a trace is simply a collection of timestamped 63"events". Events can have associated metadata and context which allows them to 64be interpreted and analyzed. 65 66Events form the foundation of trace processor and are one of two types: slices 67and counters. 68 69#### Slices 70 71 72 73A slice refers to an interval of time with some data describing what was 74happening in that interval. Some example of slices include: 75 76* Scheduling slices for each CPU 77* Atrace slices on Android 78* Userspace slices from Chrome 79 80#### Counters 81 82 83 84A counter is a continuous value which varies over time. Some examples of 85counters include: 86 87* CPU frequency for each CPU core 88* RSS memory events - both from the kernel and polled from /proc/stats 89* atrace counter events from Android 90* Chrome counter events 91 92### Tracks 93 94A track is a named partition of events of the same type and the same associated 95context. For example: 96 97* Scheduling slices have one track for each CPU 98* Sync userspace slice have one track for each thread which emitted an event 99* Async userspace slices have one track for each “cookie” linking a set of async 100 events 101 102The most intuitive way to think of a track is to imagine how they would be drawn 103in a UI; if all the events are in a single row, they belong to the same track. 104For example, all the scheduling events for CPU 5 are on the same track: 105 106 107 108Tracks can be split into various types based on the type of event they contain 109and the context they are associated with. Examples include: 110 111* Global tracks are not associated to any context and contain slices 112* Thread tracks are associated to a single thread and contain slices 113* Counter tracks are not associated to any context and contain counters 114* CPU counter tracks are associated to a single CPU and contain counters 115 116### Thread and process identifiers 117 118The handling of threads and processes needs special care when considered in the 119context of tracing; identifiers for threads and processes (e.g. `pid`/`tgid` and 120`tid` in Android/macOS/Linux) can be reused by the operating system over the 121course of a trace. This means they cannot be relied upon as a unique identifier 122when querying tables in trace processor. 123 124To solve this problem, the trace processor uses `utid` (_unique_ tid) for 125threads and `upid` (_unique_ pid) for processes. All references to threads and 126processes (e.g. in CPU scheduling data, thread tracks) uses `utid` and `upid` 127instead of the system identifiers. 128 129## Object-oriented tables 130 131Modeling an object with many types is a common problem in trace processor. For 132example, tracks can come in many varieties (thread tracks, process tracks, 133counter tracks etc). Each type has a piece of data associated to it unique to 134that type; for example, thread tracks have a `utid` of the thread, counter 135tracks have the `unit` of the counter. 136 137To solve this problem in object-oriented languages, a `Track` class could be 138created and inheritance used for all subclasses (e.g. `ThreadTrack` and 139`CounterTrack` being subclasses of `Track`, `ProcessCounterTrack` being a 140subclass of `CounterTrack` etc). 141 142 143 144In trace processor, this "object-oriented" approach is replicated by having 145different tables for each type of object. For example, we have a `track` table 146as the "root" of the hierarchy with the `thread_track` and `counter_track` 147tables "inheriting from" the `track` table. 148 149NOTE: [The appendix below](#appendix-table-inheritance) gives the exact rules 150for inheritance between tables for interested readers. 151 152Inheritance between the tables works in the natural way (i.e. how it works in 153OO languages) and is best summarized by a diagram. 154 155 156 157NOTE: For an up-to-date of how tables currently inherit from each other as well 158as a comprehensive reference of all the column and how they are inherited see 159the [SQL tables](/docs/analysis/sql-tables.autogen) reference page. 160 161## Writing Queries 162 163### Context using tracks 164 165A common question when querying tables in trace processor is: "how do I obtain 166the process or thread for a slice?". Phrased more generally, the question is 167"how do I get the context for an event?". 168 169In trace processor, any context associated with all events on a track is found 170on the associated `track` tables. 171 172For example, to obtain the `utid` of any thread which emitted a `measure` slice 173 174```sql 175SELECT utid 176FROM slice 177JOIN thread_track ON thread_track.id = slice.track_id 178WHERE slice.name = 'measure' 179``` 180 181Similarly, to obtain the `upid`s of any process which has a `mem.swap` counter 182greater than 1000 183 184```sql 185SELECT upid 186FROM counter 187JOIN process_counter_track ON process_counter_track.id = slice.track_id 188WHERE process_counter_track.name = 'mem.swap' AND value > 1000 189``` 190 191If the source and type of the event is known beforehand (which is generally the 192case), the following can be used to find the `track` table to join with 193 194| Event type | Associated with | Track table | Constraint in WHERE clause | 195| :--------- | ------------------ | --------------------- | -------------------------- | 196| slice | N/A (global scope) | track | `type = 'track'` | 197| slice | thread | thread_track | N/A | 198| slice | process | process_track | N/A | 199| counter | N/A (global scope) | counter_track | `type = 'counter_track'` | 200| counter | thread | thread_counter_track | N/A | 201| counter | process | process_counter_track | N/A | 202| counter | cpu | cpu_counter_track | N/A | 203 204On the other hand, sometimes the source is not known. In this case, joining with 205the `track `table and looking up the `type` column will give the exact track 206table to join with. 207 208For example, to find the type of track for `measure` events, the following query 209could be used. 210 211```sql 212SELECT type 213FROM slice 214JOIN track ON track.id = slice.track_id 215WHERE slice.name = 'measure' 216``` 217 218### Thread and process tables 219 220While obtaining `utid`s and `upid`s are a step in the right direction, generally 221users want the original `tid`, `pid`, and process/thread names. 222 223The `thread` and `process` tables map `utid`s and `upid`s to threads and 224processes respectively. For example, to lookup the thread with `utid` 10 225 226```sql 227SELECT tid, name 228FROM thread 229WHERE utid = 10 230``` 231 232The `thread` and `process` tables can also be joined with the associated track 233tables directly to jump directly from the slice or counter to the information 234about processes and threads. 235 236For example, to get a list of all the threads which emitted a `measure` slice 237 238```sql 239SELECT thread.name AS thread_name 240FROM slice 241JOIN thread_track ON slice.track_id = thread_track.id 242JOIN thread USING(utid) 243WHERE slice.name = 'measure' 244GROUP BY thread_name 245``` 246 247## Metrics 248 249TIP: To see how to add to add a new metric to trace processor, see the checklist 250[here](/docs/contributing/common-tasks.md#new-metric). 251 252The metrics subsystem is a significant part of trace processor and thus is 253documented on its own [page](/docs/analysis/metrics.md). 254 255## Annotations 256 257TIP: To see how to add to add a new annotation to trace processor, see the 258checklist [here](/docs/contributing/common-tasks.md#new-annotation). 259 260Annotations attach a human-readable description to a slice in the trace. This 261can include information like the source of a slice, why a slice is important and 262links to documentation where the viewer can learn more about the slice. 263In essence, descriptions act as if an expert was telling the user what the slice 264means. 265 266For example, consider the `inflate` slice which occurs during view inflation in 267Android. We can add the following description and link: 268 269**Description**: Constructing a View hierarchy from pre-processed XML via 270LayoutInflater#layout. This includes constructing all of the View objects in the 271hierarchy, and applying styled attributes. 272 273## Creating derived events 274 275TIP: To see how to add to add a new annotation to trace processor, see the 276 checklist [here](/docs/contributing/common-tasks.md#new-annotation). 277 278This feature allows creation of new events (slices and counters) from the data 279in the trace. These events can then be displayed in the UI tracks as if they 280were part of the trace itself. 281 282This is useful as often the data in the trace is very low-level. While low 283level information is important for experts to perform deep debugging, often 284users are just looking for a high level overview without needing to consider 285events from multiple locations. 286 287For example, an app startup in Android spans multiple components including 288`ActivityManager`, `system_server`, and the newly created app process derived 289from `zygote`. Most users do not need this level of detail; they are only 290interested in a single slice spanning the entire startup. 291 292Creating derived events is tied very closely to 293[metrics subsystem](/docs/analysis/metrics.md); often SQL-based metrics need to 294create higher-level abstractions from raw events as intermediate artifacts. 295 296From previous example, the 297[startup metric](/src/trace_processor/metrics/android/android_startup.sql) 298creates the exact `launching` slice we want to display in the UI. 299 300The other benefit of aligning the two is that changes in metrics are 301automatically kept in sync with what the user sees in the UI. 302 303## Alerts 304 305Alerts are used to draw the attention of the user to interesting parts of the 306trace; this are usually warnings or errors about anomalies which occurred in the 307trace. 308 309Currently, alerts are not implemented in the trace processor but the API to 310create derived events was designed with them in mind. We plan on adding another 311column `alert_type` (name to be finalized) to the annotations table which can 312have the value `warning`, `error` or `null`. Depending on this value, the 313Perfetto UI will flag these events to the user. 314 315NOTE: we do not plan on supporting case where alerts need to be added to 316 existing events. Instead, new events should be created using annotations 317 and alerts added on these instead; this is because the trace processor 318 storage is monotonic-append-only. 319 320## Appendix: table inheritance 321 322Concretely, the rules for inheritance between tables works are as follows: 323 324* Every row in a table has an `id` which is unique for a hierarchy of tables. 325 * For example, every `track` will have an `id` which is unique among all 326 tracks (regardless of the type of track) 327* If a table C inherits from P, each row in C will also be in P _with the same 328 id_ 329 * This allows for ids to act as "pointers" to rows; lookups by id can be 330 performed on any table which has that row 331 * For example, every `process_counter_track` row will have a matching row in 332 `counter_track` which will itself have matching rows in `track` 333* If a table C with columns `A` and `B` inherits from P with column `A`, `A` 334 will have the same data in both C and P 335 * For example, suppose 336 * `process_counter_track` has columns `name`, `unit` and `upid` 337 * `counter_track` has `name` and `unit` 338 * `track` has `name` 339 * Every row in `process_counter_track` will have the same `name` for the row 340 with the same id in `track` and `counter_track` 341 * Similarly, every row in `process_counter_track` will have both the same 342 `name ` and `unit` for the row with the same id in `counter_track` 343* Every row in a table has a `type` column. This specifies the _most specific_ 344 table this row belongs to. 345 * This allows _dynamic casting_ of a row to its most specific type 346 * For example, for if a row in the `track` is actually a 347 `process_counter_track`, it's type column will be `process_counter_track`. 348
