1# Quickstart: SQL-based analysis and trace-based metrics 2 3_This quickstart explains how to use `trace_processor` as well as its Python API to 4programmatically query the trace contents through SQL and compute trace-based metrics._ 5 6## Trace Processor 7 8TraceProcessor is a multi-format trace importing and query engine based on 9SQLite. It comes both as a C++ library and as a standalone executable: 10`trace_processor_shell` (or just `trace_processor`). 11 12### Setup 13 14```bash 15# Download prebuilts (Linux and Mac only) 16curl -LO https://get.perfetto.dev/trace_processor 17chmod +x ./trace_processor 18 19# Start the interactive shell 20./trace_processor trace.perfetto-trace 21 22# Start a local trace processor instance to replace wasm module in the UI 23./trace_processor trace.perfetto-trace --http 24``` 25 26NOTE: In HTTP mode the trace will be loaded into the `trace_processor` and 27 the UI will connect and issue queries over TCP. This can allow 28 arbitrary sized traces to be loaded since there are no memory 29 constraints, unlike the WASM module. In addition, this can improve 30 performance in the UI as it issues SQL queries. 31 32See [Trace Processor docs](/docs/analysis/trace-processor.md) for the full 33TraceProcessor guide. 34 35### Sample queries 36 37For more exhaustive examples see the _SQL_ section of the various _Data sources_ 38docs. 39 40#### Slices 41 42Slices are stackable events which have name and span some duration of time. 43 44 45 46``` 47> SELECT ts, dur, name FROM slice 48ts dur name 49-------------------- -------------------- --------------------------- 50 261187017446933 358594 eglSwapBuffersWithDamageKHR 51 261187017518340 357 onMessageReceived 52 261187020825163 9948 queueBuffer 53 261187021345235 642 bufferLoad 54 261187121345235 153 query 55 ... 56``` 57 58#### Counters 59 60Counters are events with a value which changes over time. 61 62 63 64``` 65> SELECT ts, value FROM counter 66ts value 67-------------------- -------------------- 68 261187012149954 1454.000000 69 261187012399172 4232.000000 70 261187012447402 14304.000000 71 261187012535839 15490.000000 72 261187012590890 17490.000000 73 261187012590890 16590.000000 74... 75``` 76 77#### Scheduler slices 78 79Scheduler slices indicate which thread was scheduled on which CPU at which time. 80 81 82 83``` 84> SELECT ts, dur, cpu, utid FROM sched 85ts dur cpu utid 86-------------------- -------------------- -------------------- -------------------- 87 261187012170489 267188 0 390 88 261187012170995 247153 1 767 89 261187012418183 12812 2 2790 90 261187012421099 220000 6 683 91 261187012430995 72396 7 2791 92... 93``` 94 95### Trace-based metrics 96 97Trace Processor offers also a higher-level query interface that allows to run 98pre-baked queries, herein called "metrics". Metrics are generally curated by 99domain experts, often the same people who add the instrumentation points in the 100first place, and output structured JSON/Protobuf/text. 101Metrics allow to get a summarized view of the trace without having to type any 102SQL or having to load the trace in the UI. 103 104The metrics` schema files live in the 105[/protos/perfetto/metrics](/protos/perfetto/metrics/) directory. 106The corresponding SQL queries live in 107[/src/trace_processor/metrics](/src/trace_processor/metrics/). 108 109#### Run a single metric 110 111Let's run the [`android_cpu`](/protos/perfetto/metrics/android/cpu_metric.proto) 112metric. This metrics computes the total CPU time and the total cycles 113(CPU frequency * time spent running at that frequency) for each process in the 114trace, breaking it down by CPU (_core_) number. 115 116```protobuf 117./trace_processor --run-metrics android_cpu trace.perfetto-trace 118 119android_cpu { 120 process_info { 121 name: "/system/bin/init" 122 threads { 123 name: "init" 124 core { 125 id: 1 126 metrics { 127 mcycles: 1 128 runtime_ns: 570365 129 min_freq_khz: 1900800 130 max_freq_khz: 1900800 131 avg_freq_khz: 1902017 132 } 133 } 134 core { 135 id: 3 136 metrics { 137 mcycles: 0 138 runtime_ns: 366406 139 min_freq_khz: 1900800 140 max_freq_khz: 1900800 141 avg_freq_khz: 1902908 142 } 143 } 144 ... 145 } 146 ... 147 } 148 process_info { 149 name: "/system/bin/logd" 150 threads { 151 name: "logd.writer" 152 core { 153 id: 0 154 metrics { 155 mcycles: 8 156 runtime_ns: 33842357 157 min_freq_khz: 595200 158 max_freq_khz: 1900800 159 avg_freq_khz: 1891825 160 } 161 } 162 core { 163 id: 1 164 metrics { 165 mcycles: 9 166 runtime_ns: 36019300 167 min_freq_khz: 1171200 168 max_freq_khz: 1900800 169 avg_freq_khz: 1887969 170 } 171 } 172 ... 173 } 174 ... 175 } 176 ... 177} 178``` 179 180#### Running multiple metrics 181 182Multiple metrics can be flagged using comma separators to the `--run-metrics` 183flag. This will output a text proto with the combined result of running both 184metrics. 185 186```protobuf 187$ ./trace_processor --run-metrics android_mem,android_cpu trace.perfetto-trace 188 189android_mem { 190 process_metrics { 191 process_name: ".dataservices" 192 total_counters { 193 anon_rss { 194 min: 19451904 195 max: 19890176 196 avg: 19837548.157829277 197 } 198 file_rss { 199 min: 25804800 200 max: 25829376 201 avg: 25827909.957489081 202 } 203 swap { 204 min: 9289728 205 max: 9728000 206 avg: 9342355.8421707246 207 } 208 anon_and_swap { 209 min: 29179904 210 max: 29179904 211 avg: 29179904 212 } 213 } 214 ... 215 } 216 ... 217} 218android_cpu { 219 process_info { 220 name: "/system/bin/init" 221 threads { 222 name: "init" 223 core { 224 id: 1 225 metrics { 226 mcycles: 1 227 runtime_ns: 570365 228 min_freq_khz: 1900800 229 max_freq_khz: 1900800 230 avg_freq_khz: 1902017 231 } 232 } 233 ... 234 } 235 ... 236 } 237 ... 238} 239``` 240 241#### JSON and binary output 242 243The trace processor also supports binary protobuf and JSON as alternative output 244formats. This is useful when the intended reader is an offline tool. 245 246Both single and multiple metrics are supported as with proto text output. 247 248``` 249./trace_processor --run-metrics android_mem --metrics-output=binary trace.perfetto-trace 250<binary protobuf output> 251 252./trace_processor --run-metrics android_mem,android_cpu --metrics-output=json trace.perfetto-trace 253{ 254 "android_mem": { 255 "process_metrics": [ 256 { 257 "process_name": ".dataservices", 258 "total_counters": { 259 "anon_rss": { 260 "min": 19451904.000000, 261 "max": 19890176.000000, 262 "avg": 19837548.157829 263 }, 264 "file_rss": { 265 "min": 25804800.000000, 266 "max": 25829376.000000, 267 "avg": 25827909.957489 268 }, 269 "swap": { 270 "min": 9289728.000000, 271 "max": 9728000.000000, 272 "avg": 9342355.842171 273 }, 274 "anon_and_swap": { 275 "min": 29179904.000000, 276 "max": 29179904.000000, 277 "avg": 29179904.000000 278 } 279 }, 280 ... 281 }, 282 ... 283 ] 284 } 285 "android_cpu": { 286 "process_info": [ 287 { 288 "name": "\/system\/bin\/init", 289 "threads": [ 290 { 291 "name": "init", 292 "core": [ 293 { 294 "id": 1, 295 "metrics": { 296 "mcycles": 1, 297 "runtime_ns": 570365, 298 "min_freq_khz": 1900800, 299 "max_freq_khz": 1900800, 300 "avg_freq_khz": 1902017 301 } 302 }, 303 ... 304 ] 305 ... 306 } 307 ... 308 ] 309 ... 310 }, 311 ... 312 ] 313 ... 314 } 315} 316``` 317 318## Python API 319 320The API can be run without requiring the `trace_processor` binary to be 321downloaded or installed. 322 323### Setup 324``` 325$ pip install perfetto 326``` 327NOTE: The API is only compatible with Python3. 328 329### Example functions 330See the Python API section of 331[Trace Processor (SQL)](/docs/analysis/trace-processor.md) to get 332more details on all available functions. 333 334#### Query 335```python 336from perfetto.trace_processor import TraceProcessor 337tp = TraceProcessor(file_path='trace.perfetto-trace') 338 339qr_it = tp.query('SELECT name FROM slice') 340for row in qr_it: 341 print(row.name) 342``` 343**Output** 344``` 345eglSwapBuffersWithDamageKHR 346onMessageReceived 347queueBuffer 348bufferLoad 349query 350... 351``` 352#### Query as Pandas DataFrame 353```python 354from perfetto.trace_processor import TraceProcessor 355tp = TraceProcessor(file_path='trace.perfetto-trace') 356 357qr_it = tp.query('SELECT ts, name FROM slice') 358qr_df = qr_it.as_pandas_dataframe() 359print(qr_df.to_string()) 360``` 361**Output** 362``` 363ts name 364-------------------- --------------------------- 365 261187017446933 eglSwapBuffersWithDamageKHR 366 261187017518340 onMessageReceived 367 261187020825163 queueBuffer 368 261187021345235 bufferLoad 369 261187121345235 query 370 ... 371``` 372#### Metric 373```python 374from perfetto.trace_processor import TraceProcessor 375tp = TraceProcessor(file_path='trace.perfetto-trace') 376 377cpu_metrics = tp.metric(['android_cpu']) 378print(cpu_metrics) 379``` 380**Output** 381``` 382metrics { 383 android_cpu { 384 process_info { 385 name: "/system/bin/init" 386 threads { 387 name: "init" 388 core { 389 id: 1 390 metrics { 391 mcycles: 1 392 runtime_ns: 570365 393 min_freq_khz: 1900800 394 max_freq_khz: 1900800 395 avg_freq_khz: 1902017 396 } 397 } 398 core { 399 id: 3 400 metrics { 401 mcycles: 0 402 runtime_ns: 366406 403 min_freq_khz: 1900800 404 max_freq_khz: 1900800 405 avg_freq_khz: 1902908 406 } 407 } 408 ... 409 } 410 ... 411 } 412 ... 413 } 414} 415``` 416 417## Next steps 418 419There are several options for exploring more of the trace analysis features 420Perfetto provides: 421 422* The [trace conversion quickstart](/docs/quickstart/traceconv.md) gives an 423 overview on how to convert Perfetto traces to legacy formats to integrate with 424 existing tooling. 425* The [Trace Processor documentation](/docs/analysis/trace-processor.md) gives 426 more information about how to work with trace processor including details on 427 how to write queries and how tables in trace processor are organized. 428* The [metrics documentation](/docs/analysis/metrics.md) gives a more in-depth 429 look into metrics including a short walkthrough on how to build an 430 experimental metric from scratch. 431* The [SQL table reference](/docs/analysis/sql-tables.autogen) gives a 432 comprehensive guide to the all the available tables in trace processor. 433* The [common tasks](/docs/contributing/common-tasks.md) page gives a list of 434 steps on how new metrics can be added to the trace processor. 435
