Lines Matching +full:layer +full:- +full:buffer +full:- +full:offset
1 .. SPDX-License-Identifier: GPL-2.0
4 Perf ring buffer
11 2. Ring buffer implementation
13 2.2 Ring buffer for different tracing modes
15 2.2.2 Per-thread mode
16 2.2.3 Per-CPU mode
18 2.3 Accessing buffer
19 2.3.1 Producer-consumer model
21 2.3.3 Writing samples into buffer
22 2.3.4 Reading samples from buffer
25 3. The mechanism of AUX ring buffer
34 The ring buffer is a fundamental mechanism for data transfer. perf uses
36 kind of ring buffer which is so called auxiliary (AUX) ring buffer also
40 The ring buffer implementation is critical but it's also a very
42 space use the ring buffer to exchange data and stores data into data
43 file, thus the ring buffer needs to transfer data with high throughput;
44 on the other hand, the ring buffer management should avoid significant
47 This documentation dives into the details for perf ring buffer with two
48 parts: firstly it explains the perf ring buffer implementation, then the
49 second part discusses the AUX ring buffer mechanism.
51 2. Ring buffer implementation
55 -------------------
57 That said, a typical ring buffer is managed by a head pointer and a tail
63 +---------------------------+
65 +---------------------------+
66 `-> Tail `-> Head
70 Figure 1. Ring buffer
72 Perf uses the same way to manage its ring buffer. In the implementation
74 pages, the control structure and then the ring buffer itself. The page
76 in continuous virtual addresses simplifies locating the ring buffer
81 kernel starts to fill records into the ring buffer, it updates the head
83 the buffer. On the other side, when the user page is a writable mapping,
85 data from the ring buffer. Yet another case is for the user page's
86 read-only mapping, which is to be addressed in the section
91 user page ring buffer
92 +---------+---------+ +---------------------------------------+
94 +---------+---------+ +---------------------------------------+
95 ` `----------------^ ^
96 `----------------------------------------------|
100 Figure 2. Perf ring buffer
102 When using the ``perf record`` tool, we can specify the ring buffer size
103 with option ``-m`` or ``--mmap-pages=``, the given size will be rounded up
106 to VMA area until the perf tool accesses the buffer from the user space.
107 In other words, at the first time accesses the buffer's page from user
113 2.2 Ring buffer for different tracing modes
114 -------------------------------------------
118 modes and how the ring buffer meets requirements for them. At last we
135 evsel::cpus::map[] = { 0 .. _SC_NPROCESSORS_ONLN-1 }
140 buffers. As shown below, the perf tool allocates individual ring buffer
145 the *T1* thread and stored in the ring buffer associated with the CPU on
151 +----+ +-----------+ +----+
153 +----+--------------+-----------+----------+----+-------->
156 +-----------------------------------------------------+
157 | Ring buffer 0 |
158 +-----------------------------------------------------+
161 +-----+
163 -----+-----+--------------------------------------------->
166 +-----------------------------------------------------+
167 | Ring buffer 1 |
168 +-----------------------------------------------------+
171 +----+ +-------+
173 --------------------------+----+--------+-------+-------->
176 +-----------------------------------------------------+
177 | Ring buffer 2 |
178 +-----------------------------------------------------+
181 +--------------+
183 -----------+--------------+------------------------------>
186 +-----------------------------------------------------+
187 | Ring buffer 3 |
188 +-----------------------------------------------------+
193 Figure 3. Ring buffer for default mode
195 2.2.2 Per-thread mode
198 By specifying option ``--per-thread`` in perf command, e.g.
202 perf record --per-thread test_program
207 evsel::cpus::map[0] = { -1 }
211 In this mode, a single ring buffer is allocated for the profiled thread;
221 +----+ +-----------+ +----+
223 +----+--------------+-----------+----------+----+-------->
226 | +-----+ |
228 --|--+-----+----------------------------------|---------->
231 | | +----+ +---+ |
233 --|-----|-----------------+----+--------+---+-|---------->
236 | | +--------------+ | |
238 --|-----|--+--------------+-|-----------------|---------->
241 +-----------------------------------------------------+
242 | Ring buffer |
243 +-----------------------------------------------------+
248 Figure 4. Ring buffer for per-thread mode
250 When perf runs in per-thread mode, a ring buffer is allocated for the
251 profiled thread *T1*. The ring buffer is dedicated for thread *T1*, if the
253 buffer; when the thread is sleeping, all associated events will be
254 disabled, thus no trace data will be recorded into the ring buffer.
256 2.2.3 Per-CPU mode
259 The option ``-C`` is used to collect samples on the list of CPUs, for
260 example the below perf command receives option ``-C 0,2``::
262 perf record -C 0,2 test_program
268 evsel::threads::map[] = { -1 }
273 running on CPU1 and CPU3, since the ring buffer is absent for them, any
275 options for per-thread mode and per-CPU mode, e.g. the options ``–C 0,2`` and
282 +----+ +-----------+ +----+
284 +----+--------------+-----------+----------+----+-------->
287 +-----------------------------------------------------+
288 | Ring buffer 0 |
289 +-----------------------------------------------------+
292 +-----+
294 -----+-----+--------------------------------------------->
297 +----+ +-------+
299 --------------------------+----+--------+-------+-------->
302 +-----------------------------------------------------+
303 | Ring buffer 1 |
304 +-----------------------------------------------------+
307 +--------------+
309 -----------+--------------+------------------------------>
314 Figure 5. Ring buffer for per-CPU mode
322 perf record -a test_program
324 Similar to the per-CPU mode, the perf event doesn't bind to any PID, and
327 evsel::cpus::map[] = { 0 .. _SC_NPROCESSORS_ONLN-1 }
328 evsel::threads::map[] = { -1 }
331 In the system wide mode, every CPU has its own ring buffer, all threads
333 the ring buffer belonging to the CPU which the events occurred on.
338 +----+ +-----------+ +----+
340 +----+--------------+-----------+----------+----+-------->
343 +-----------------------------------------------------+
344 | Ring buffer 0 |
345 +-----------------------------------------------------+
348 +-----+
350 -----+-----+--------------------------------------------->
353 +-----------------------------------------------------+
354 | Ring buffer 1 |
355 +-----------------------------------------------------+
358 +----+ +-------+
360 --------------------------+----+--------+-------+-------->
363 +-----------------------------------------------------+
364 | Ring buffer 2 |
365 +-----------------------------------------------------+
368 +--------------+
370 -----------+--------------+------------------------------>
373 +-----------------------------------------------------+
374 | Ring buffer 3 |
375 +-----------------------------------------------------+
380 Figure 6. Ring buffer for system wide mode
382 2.3 Accessing buffer
383 --------------------
385 Based on the understanding of how the ring buffer is allocated in
386 various modes, this section explains access the ring buffer.
388 2.3.1 Producer-consumer model
392 into the ring buffer; the perf command in user space consumes the
393 samples by reading out data from the ring buffer and finally saves the
394 data into the file for post analysis. It’s a typical producer-consumer
395 model for using the ring buffer.
399 space, the kernel event core layer introduces a watermark, which is
401 the ring buffer, and if the used buffer exceeds the watermark, the
402 kernel wakes up the perf process to read samples from the ring buffer.
408 Polling / `--------------| Ring buffer
409 v v ;---------------------v
410 +----------------+ +---------+---------+ +-------------------+
412 +----------------+ +---------+---------+ +-------------------+
413 ^ ^ `------------------------^
415 +-----------------------------+
416 | Kernel event core layer |
417 +-----------------------------+
421 Figure 7. Writing and reading the ring buffer
423 When the kernel event core layer notifies the user space, because
424 multiple events might share the same ring buffer for recording samples,
425 the core layer iterates every event associated with the ring buffer and
430 one by one, if it finds any ring buffer containing samples it will read
432 perf process is able to run on any CPU, this leads to the ring buffer
440 Linux kernel supports two write directions for the ring buffer: forward and
442 buffer, the backward writing stores data from the end of the ring buffer with
445 Additionally, the tool can map buffers in either read-write mode or read-only
448 The ring buffer in the read-write mode is mapped with the property
455 Alternatively, in the read-only mode, only the kernel keeps to update
461 combinations to support buffer types: the non-overwrite buffer and the
462 overwritable buffer.
464 .. list-table::
466 :header-rows: 1
468 * - Mapping mode
469 - Forward
470 - Backward
471 * - read-write
472 - Non-overwrite ring buffer
473 - Not used
474 * - read-only
475 - Not used
476 - Overwritable ring buffer
478 The non-overwrite ring buffer uses the read-write mapping with forward
479 writing. It starts to save data from the beginning of the ring buffer
480 and wrap around when overflow, which is used with the read-write mode in
481 the normal ring buffer. When the consumer doesn't keep up with the
484 when it finds a space in the ring buffer.
486 The overwritable ring buffer uses the backward writing with the
487 read-only mode. It saves the data from the end of the ring buffer and
489 knows where it starts to read and until the end of the ring buffer, thus
495 2.3.3 Writing samples into buffer argument
498 When a sample is taken and saved into the ring buffer, the kernel
500 info for writing ring buffer which is stored in the structure
502 the ring buffer and updates the head pointer in the user page so the
506 tracking the information related to the buffer. The advantages of it is
507 that it enables concurrent writing to the buffer by different events.
514 2.3.4 Reading samples from buffer argument
518 structure to handle the head and tail of the buffer. It also uses
519 ``perf_mmap`` structure to keep track of a context for the ring buffer, this
520 context includes information about the buffer's starting and ending
522 circular buffer pointer even for an overflow.
525 the recorded data from the ring buffer, and then updates the buffer's
534 ordering, this means it’s possible to access the ring buffer and the
536 sequence for memory accessing perf ring buffer, memory barriers are
542 if (LOAD ->data_tail) { LOAD ->data_head
546 STORE ->data_head STORE ->data_tail
555 buffer;
557 (D) pairs with (A). (D) separates the ring buffer data reading from
575 Some architectures support one-way permeable barrier with load-acquire
576 and store-release operations, these barriers are more relaxed with less
580 If an architecture doesn’t support load-acquire and store-release in its
587 3. The mechanism of AUX ring buffer
591 buffer. In the first part it will discuss the connection between the
592 AUX ring buffer and the regular ring buffer, then the second part will
593 examine how the AUX ring buffer co-works with the regular ring buffer,
594 as well as the additional features introduced by the AUX ring buffer for
598 ---------------------------------------------------------
600 Generally, the AUX ring buffer is an auxiliary for the regular ring
601 buffer. The regular ring buffer is primarily used to store the event
603 union ``perf_event``; the AUX ring buffer is for recording the hardware
606 The general use and advantage of the AUX ring buffer is that it is
608 regular profile samples that write to the regular ring buffer cause an
610 using interrupts would be overwhelming for the regular ring buffer
611 mechanism. Having an AUX buffer allows for a region of memory more
614 The AUX ring buffer reuses the same algorithm with the regular ring
615 buffer for the buffer management. The control structure
617 for the head and tail pointers of the AUX ring buffer.
619 During the initialisation phase, besides the mmap()-ed regular ring
620 buffer, the perf tool invokes a second syscall in the
621 ``auxtrace_mmap__mmap()`` function for the mmap of the AUX buffer with
622 non-zero file offset; ``rb_alloc_aux()`` in the kernel allocates pages
625 regular ring buffer.
630 perf record -a -e cycles -e cs_etm/@tmc_etr0/ -- sleep 2
634 into the regular ring buffer while the CoreSight's AUX trace data is
635 stored in the AUX ring buffer.
637 As a result, we can see the regular ring buffer and the AUX ring buffer
639 ring buffer and the AUX ring buffer per CPU-wise, which is the same as
642 in the system. For per-thread mode, the perf tool allocates only one
643 regular ring buffer and one AUX ring buffer for the whole session. For
644 the per-CPU mode, the perf allocates two kinds of ring buffers for
645 selected CPUs specified by the option ``-C``.
655 +----+ +-----------+ +----+
657 +----+--------------+-----------+----------+----+-------->
660 +-----------------------------------------------------+
661 | Ring buffer 0 |
662 +-----------------------------------------------------+
665 +-----------------------------------------------------+
666 | AUX Ring buffer 0 |
667 +-----------------------------------------------------+
670 +-----+
672 -----+-----+--------------------------------------------->
675 +-----------------------------------------------------+
676 | Ring buffer 1 |
677 +-----------------------------------------------------+
680 +-----------------------------------------------------+
681 | AUX Ring buffer 1 |
682 +-----------------------------------------------------+
685 +----+ +-------+
687 --------------------------+----+--------+-------+-------->
690 +-----------------------------------------------------+
691 | Ring buffer 2 |
692 +-----------------------------------------------------+
695 +-----------------------------------------------------+
696 | AUX Ring buffer 2 |
697 +-----------------------------------------------------+
700 +--------------+
702 -----------+--------------+------------------------------>
705 +-----------------------------------------------------+
706 | Ring buffer 3 |
707 +-----------------------------------------------------+
710 +-----------------------------------------------------+
711 | AUX Ring buffer 3 |
712 +-----------------------------------------------------+
717 Figure 8. AUX ring buffer for system wide mode
720 --------------
723 regular ring buffer, ``perf_aux_output_begin()`` and ``perf_aux_output_end()``
724 serve for the AUX ring buffer for processing the hardware trace data.
726 Once the hardware trace data is stored into the AUX ring buffer, the PMU
728 Similar to the regular ring buffer, the AUX ring buffer needs to apply
730 :ref:`memory_synchronization`. Since the AUX ring buffer is managed by the
738 - It fills an event ``PERF_RECORD_AUX`` into the regular ring buffer, this
740 chunk of hardware trace data has been stored into the AUX ring buffer;
742 - Since the hardware trace driver has stored new trace data into the AUX
743 ring buffer, the argument *size* indicates how many bytes have been
745 header pointer ``perf_buffer::aux_head`` to reflect the latest buffer usage.
754 buffer to the perf data file, it synthesizes a ``PERF_RECORD_AUXTRACE``
756 which portion of data in the AUX ring buffer is saved. Afterwards, the perf
762 -----------------
764 Perf supports snapshot mode for AUX ring buffer, in this mode, users
769 perf record -e cs_etm/@tmc_etr0/u -S -a program &
772 kill -USR2 $PERFPID
778 - Before a snapshot is taken, the AUX ring buffer acts in free run mode.
782 - Once the perf tool receives the *USR2* signal, it triggers the callback
784 tracing. The kernel driver then populates the AUX ring buffer with the
786 regular ring buffer;
788 - Then perf tool takes a snapshot, ``record__read_auxtrace_snapshot()``
789 reads out the hardware trace data from the AUX ring buffer and saves it
792 - After the snapshot is finished, ``auxtrace_record::snapshot_finish()``
797 because the AUX ring buffer can overflow in free run mode, the tail
800 of whether the AUX ring buffer has been wrapped around or not, at the
801 end it fixes up the AUX buffer's head which are used to calculate the
804 As we know, the buffers' deployment can be per-thread mode, per-CPU
814 +------------------------+
815 | AUX Ring buffer 0 | <- aux_head
816 +------------------------+
818 +--------------------------------+
819 | AUX Ring buffer 1 | <- aux_head
820 +--------------------------------+
822 +--------------------------------------------+
823 | AUX Ring buffer 2 | <- aux_head
824 +--------------------------------------------+
826 +---------------------------------------+
827 | AUX Ring buffer 3 | <- aux_head
828 +---------------------------------------+