1# Common tasks 2 3The checklists below show how to achieve some common tasks in the codebase. 4 5## Add a new ftrace event 6 71. Find the `format` file for your event. The location of the file depends where `tracefs` is mounted but can often be found at `/sys/kernel/debug/tracing/events/EVENT_GROUP/EVENT_NAME/format`. 82. Copy the format file into the codebase at `src/traced/probes/ftrace/test/data/synthetic/events/EVENT_GROUP/EVENT_NAME/format`. 93. Add the event to [src/tools/ftrace_proto_gen/event_list](/src/tools/ftrace_proto_gen/event_list). 104. Run `tools/run_ftrace_proto_gen`. This will update `protos/perfetto/trace/ftrace/ftrace_event.proto` and `protos/perfetto/trace/ftrace/GROUP_NAME.proto`. 115. Run `tools/gen_all out/YOUR_BUILD_DIRECTORY`. This will update `src/traced/probes/ftrace/event_info.cc` and `protos/perfetto/trace/perfetto_trace.proto`. 126. If special handling in `trace_processor` is desired update [src/trace_processor/importers/ftrace/ftrace_parser.cc](/src/trace_processor/importers/ftrace/ftrace_parser.cc) to parse the event. 137. Upload and land your change as normal. 14 15Here is an [example change](https://android-review.googlesource.com/c/platform/external/perfetto/+/1290645) which added the `ion/ion_stat` event. 16 17## Contribute to SQL standard library 18 191. Add or edit an SQL file inside `perfetto/src/trace_processor/stdlib/`. This SQL file will be a new standard library module. 202. For a new file inside an existing package add the file to the corresponding `BUILD.gn`. 213. For a new package (subdirectory of `/stdlib/`), the package name (directory name) has to be added to the list in `/stdlib/BUILD.gn`. 22 23Files inside the standard library have to be formatted in a very specific way, as its structure is used to generate documentation. There are presubmit checks, but they are not infallible. 24 25- Running the file cannot generate any data. There can be only `CREATE PERFETTO {FUNCTION|TABLE|VIEW|MACRO}` statements inside. 26- The name of each standard library object needs to start with `{module_name}_` or be prefixed with an underscore(`_`) for internal objects. 27 The names must only contain lower and upper case letters and underscores. When a module is included (using the `INCLUDE PERFETTO MODULE`) the internal objects should not be treated as an API. 28- Every table or view should have [a schema](/docs/analysis/perfetto-sql-syntax.md#tableview-schema). 29 30### Documentation 31 32- Every non internal object, as well as its function arguments and columns in its schema have to be prefixed with an SQL comment documenting it. 33- Any text is going to be parsed as markdown, so usage of markdown functionality (code, links, lists) is encouraged. 34Whitespaces in anything apart from descriptions are ignored, so comments can be formatted neatly. 35If the line with description exceeds 80 chars, description can be continued in following lines. 36 - **Table/view**: each has to have schema, object description and a comment above each column's definition in the schema. 37 - Description is any text in the comment above `CREATE PERFETTO {TABLE,VIEW}` statement. 38 - Column's comment is the text immediately above column definition in the schema. 39 - **Scalar Functions**: each has to have a function description and description of return value in this order. 40 - Function description is any text in the comment above `CREATE PERFETTO FUNCTION` statement. 41 - For each argument there has to be a comment line immediately above argument definition. 42 - Return comment should immediately precede `RETURNS`. 43 - **Table Functions**: each has to have a function description, list of arguments (names, types, description) and list of columns. 44 - Function description is any text in the comment above `CREATE PERFETTO FUNCTION` statement. 45 - For each argument there has to be a comment line immediately above argument definition. 46 - For each column there has to be a comment line immediately above column definition. 47 48NOTE: Break lines outside of import description will be ignored. 49 50Example of properly formatted view in module `android`: 51```sql 52-- Count Binder transactions per process. 53CREATE PERFETTO VIEW android_binder_metrics_by_process( 54 -- Name of the process that started the binder transaction. 55 process_name STRING, 56 -- PID of the process that started the binder transaction. 57 pid INT, 58-- Name of the slice with binder transaction. 59 slice_name STRING, 60-- Number of binder transactions in process in slice. 61 event_count INT 62) AS 63SELECT 64 process.name AS process_name, 65 process.pid AS pid, 66 slice.name AS slice_name, 67 COUNT(*) AS event_count 68FROM slice 69JOIN thread_track ON slice.track_id = thread_track.id 70JOIN thread ON thread.utid = thread_track.utid 71JOIN process ON thread.upid = process.upid 72WHERE 73 slice.name GLOB 'binder*' 74GROUP BY 75 process_name, 76 slice_name; 77``` 78 79Example of table function in module `android`: 80```sql 81-- Given a launch id and GLOB for a slice name, returns columns for matching slices. 82CREATE PERFETTO FUNCTION ANDROID_SLICES_FOR_LAUNCH_AND_SLICE_NAME( 83 -- Id of launch. 84 launch_id INT, 85 -- Name of slice with launch. 86 slice_name STRING 87) 88RETURNS TABLE( 89 -- Name of slice with launch. 90 slice_name STRING, 91 -- Timestamp of slice start. 92 slice_ts INT, 93 -- Duration of slice. 94 slice_dur INT, 95 -- Name of thread with slice. 96 thread_name STRING, 97 -- Arg set id. 98 arg_set_id INT 99) 100AS 101SELECT 102 slice_name, 103 slice_ts, 104 slice_dur, 105 thread_name, 106 arg_set_id 107FROM thread_slices_for_all_launches 108WHERE launch_id = $launch_id AND slice_name GLOB $slice_name; 109``` 110 111 112## {#new-metric} Add a new trace-based metric 113 1141. Create the proto file containing the metric in the [protos/perfetto/metrics](/protos/perfetto/metrics) folder. The appropriate` BUILD.gn` file should be updated as well. 1152. Import the proto in [protos/perfetto/metrics/metrics.proto](/protos/perfetto/metrics/metrics.proto) and add a field for the new message. 1163. Run `tools/gen_all out/YOUR_BUILD_DIRECTORY`. This will update the generated headers containing the descriptors for the proto. 117 * *Note: this step has to be performed any time any metric-related proto is modified.* 118 * If you don't see anything inside the `out/` directory you might have to 119 rerun `tools/setup_all_configs.py`. 1204. Add a new SQL file for the metric to [src/trace_processor/metrics](/src/trace_processor/metrics). The appropriate `BUILD.gn` file should be updated as well. 121 * To learn how to write new metrics, see the [trace-based metrics documentation](/docs/analysis/metrics.md). 1225. Build all targets in your out directory with `tools/ninja -C out/YOUR_BUILD_DIRECTORY`. 1236. Add a new diff test for the metric. This can be done by adding files to 124the `tests.*.py` files in a proper [test/trace_processor](/test/trace_processor) subfolder. 1251. Run the newly added test with `tools/diff_test_trace_processor.py <path to trace processor binary>`. 1262. Upload and land your change as normal. 127 128Here is an [example change](https://android-review.googlesource.com/c/platform/external/perfetto/+/1290643) which added the `time_in_state` metric. 129 130## Add a new trace processor table 131 1321. Create the new table in the appropriate header file in [src/trace_processor/tables](/src/trace_processor/tables) by copying one of the existing macro definitions. 133 * Make sure to understand whether a root or derived table is needed and copy the appropriate one. For more information see the [trace processor](/docs/analysis/trace-processor.md) documentation. 1342. Register the table with the trace processor in the constructor for the [TraceProcessorImpl class](/src/trace_processor/trace_processor_impl.cc). 1353. If also implementing ingestion of events into the table: 136 1. Modify the appropriate parser class in [src/trace_processor/importers](/src/trace_processor/importers) and add the code to add rows to the newly added table. 137 2. Add a new diff test for the added parsing code and table. 138 3. Run the newly added test with `tools/diff_test_trace_processor.py <path to trace processor shell binary>`. 1394. Upload and land your change as normal. 140 141## Adding new derived events 142 143As derived events depend on metrics, the initial steps are same as that of developing a metric (see above). 144 145NOTE: the metric can be just an empty proto message during prototyping or if no summarization is necessary. However, generally if an event is important enough to display in the UI, it should also be tracked in benchmarks as a metric. 146 147To extend a metric with annotations: 148 1491. Create a new table or view with the name `<metric name>_event`. 150 * For example, for the [`android_startup`]() metric, we create a view named `android_startup_event`. 151 * Note that the trailing `_event` suffix in the table name is important. 152 * The schema required for this table is given below. 1532. List your metric in the `initialiseHelperViews` method of `trace_controller.ts`. 1543. Upload and land your change as normal. 155 156The schema of the `<metric name>_event` table/view is as follows: 157 158| Name | Type | Presence | Meaning | 159| :----------- | -------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | 160| `track_type` | `string` | Mandatory | 'slice' for slices, 'counter' for counters | 161| `track_name` | `string` | Mandatory | Name of the track to display in the UI. Also the track identifier i.e. all events with same `track_name` appear on the same track. | 162| `ts` | `int64` | Mandatory | The timestamp of the event (slice or counter) | 163| `dur` | `int64` | Mandatory for slice, NULL for counter | The duration of the slice | 164| `slice_name` | `string` | Mandatory for slice, NULL for counter | The name of the slice | 165| `value` | `double` | Mandatory for counter, NULL for slice | The value of the counter | 166| `group_name` | `string` | Optional | Name of the track group under which the track appears. All tracks with the same `group_name` are placed under the same group by that name. Tracks that lack this field or have NULL value in this field are displayed without any grouping. | 167 168#### Known issues: 169 170* Nested slices within the same track are not supported. We plan to support this 171 once we have a concrete usecase. 172* Tracks are always created in the global scope. We plan to extend this to 173 threads and processes in the near future with additional contexts added as 174 necessary. 175* Instant events are currently not supported in the UI but this will be 176 implemented in the near future. In trace processor, instants are always `0` 177 duration slices with special rendering on the UI side. 178* There is no way to tie newly added events back to the source events in the 179 trace which were used to generate them. This is not currently a priority but 180 something we may add in the future. 181 182 183## Update `TRACE_PROCESSOR_CURRENT_API_VERSION` 184 185Generally you do not have to worry about version skew between the UI 186and the `trace_processor` since they are built together at the same 187commit. However version skew can occur when using the `--httpd` mode 188which allows a native `trace_processor` instance to be used with the UI. 189 190A common case is when the UI is more recent than `trace_processor` 191and depends on a new table definition. With older versions of 192`trace_processor` in `--httpd` mode the UI crashes attempting to query 193a non-existant table. To avoid this we use a version number. If the 194version number `trace_processor` reports is older than the one the UI 195was built with we prompt the user to update. 196 1971. Go to `protos/perfetto/trace_processor/trace_processor.proto` 1982. Increment `TRACE_PROCESSOR_CURRENT_API_VERSION` 1993. Add a comment explaining what has changed. 200