1.. SPDX-License-Identifier: GPL-2.0 2.. include:: <isonum.txt> 3 4=========================================== 5User Interface for Resource Control feature 6=========================================== 7 8:Copyright: |copy| 2016 Intel Corporation 9:Authors: - Fenghua Yu <fenghua.yu@intel.com> 10 - Tony Luck <tony.luck@intel.com> 11 - Vikas Shivappa <vikas.shivappa@intel.com> 12 13 14Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT). 15AMD refers to this feature as AMD Platform Quality of Service(AMD QoS). 16 17This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo 18flag bits: 19 20============================================= ================================ 21RDT (Resource Director Technology) Allocation "rdt_a" 22CAT (Cache Allocation Technology) "cat_l3", "cat_l2" 23CDP (Code and Data Prioritization) "cdp_l3", "cdp_l2" 24CQM (Cache QoS Monitoring) "cqm_llc", "cqm_occup_llc" 25MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local" 26MBA (Memory Bandwidth Allocation) "mba" 27============================================= ================================ 28 29To use the feature mount the file system:: 30 31 # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps]] /sys/fs/resctrl 32 33mount options are: 34 35"cdp": 36 Enable code/data prioritization in L3 cache allocations. 37"cdpl2": 38 Enable code/data prioritization in L2 cache allocations. 39"mba_MBps": 40 Enable the MBA Software Controller(mba_sc) to specify MBA 41 bandwidth in MBps 42 43L2 and L3 CDP are controlled separately. 44 45RDT features are orthogonal. A particular system may support only 46monitoring, only control, or both monitoring and control. Cache 47pseudo-locking is a unique way of using cache control to "pin" or 48"lock" data in the cache. Details can be found in 49"Cache Pseudo-Locking". 50 51 52The mount succeeds if either of allocation or monitoring is present, but 53only those files and directories supported by the system will be created. 54For more details on the behavior of the interface during monitoring 55and allocation, see the "Resource alloc and monitor groups" section. 56 57Info directory 58============== 59 60The 'info' directory contains information about the enabled 61resources. Each resource has its own subdirectory. The subdirectory 62names reflect the resource names. 63 64Each subdirectory contains the following files with respect to 65allocation: 66 67Cache resource(L3/L2) subdirectory contains the following files 68related to allocation: 69 70"num_closids": 71 The number of CLOSIDs which are valid for this 72 resource. The kernel uses the smallest number of 73 CLOSIDs of all enabled resources as limit. 74"cbm_mask": 75 The bitmask which is valid for this resource. 76 This mask is equivalent to 100%. 77"min_cbm_bits": 78 The minimum number of consecutive bits which 79 must be set when writing a mask. 80 81"shareable_bits": 82 Bitmask of shareable resource with other executing 83 entities (e.g. I/O). User can use this when 84 setting up exclusive cache partitions. Note that 85 some platforms support devices that have their 86 own settings for cache use which can over-ride 87 these bits. 88"bit_usage": 89 Annotated capacity bitmasks showing how all 90 instances of the resource are used. The legend is: 91 92 "0": 93 Corresponding region is unused. When the system's 94 resources have been allocated and a "0" is found 95 in "bit_usage" it is a sign that resources are 96 wasted. 97 98 "H": 99 Corresponding region is used by hardware only 100 but available for software use. If a resource 101 has bits set in "shareable_bits" but not all 102 of these bits appear in the resource groups' 103 schematas then the bits appearing in 104 "shareable_bits" but no resource group will 105 be marked as "H". 106 "X": 107 Corresponding region is available for sharing and 108 used by hardware and software. These are the 109 bits that appear in "shareable_bits" as 110 well as a resource group's allocation. 111 "S": 112 Corresponding region is used by software 113 and available for sharing. 114 "E": 115 Corresponding region is used exclusively by 116 one resource group. No sharing allowed. 117 "P": 118 Corresponding region is pseudo-locked. No 119 sharing allowed. 120 121Memory bandwidth(MB) subdirectory contains the following files 122with respect to allocation: 123 124"min_bandwidth": 125 The minimum memory bandwidth percentage which 126 user can request. 127 128"bandwidth_gran": 129 The granularity in which the memory bandwidth 130 percentage is allocated. The allocated 131 b/w percentage is rounded off to the next 132 control step available on the hardware. The 133 available bandwidth control steps are: 134 min_bandwidth + N * bandwidth_gran. 135 136"delay_linear": 137 Indicates if the delay scale is linear or 138 non-linear. This field is purely informational 139 only. 140 141If RDT monitoring is available there will be an "L3_MON" directory 142with the following files: 143 144"num_rmids": 145 The number of RMIDs available. This is the 146 upper bound for how many "CTRL_MON" + "MON" 147 groups can be created. 148 149"mon_features": 150 Lists the monitoring events if 151 monitoring is enabled for the resource. 152 153"max_threshold_occupancy": 154 Read/write file provides the largest value (in 155 bytes) at which a previously used LLC_occupancy 156 counter can be considered for re-use. 157 158Finally, in the top level of the "info" directory there is a file 159named "last_cmd_status". This is reset with every "command" issued 160via the file system (making new directories or writing to any of the 161control files). If the command was successful, it will read as "ok". 162If the command failed, it will provide more information that can be 163conveyed in the error returns from file operations. E.g. 164:: 165 166 # echo L3:0=f7 > schemata 167 bash: echo: write error: Invalid argument 168 # cat info/last_cmd_status 169 mask f7 has non-consecutive 1-bits 170 171Resource alloc and monitor groups 172================================= 173 174Resource groups are represented as directories in the resctrl file 175system. The default group is the root directory which, immediately 176after mounting, owns all the tasks and cpus in the system and can make 177full use of all resources. 178 179On a system with RDT control features additional directories can be 180created in the root directory that specify different amounts of each 181resource (see "schemata" below). The root and these additional top level 182directories are referred to as "CTRL_MON" groups below. 183 184On a system with RDT monitoring the root directory and other top level 185directories contain a directory named "mon_groups" in which additional 186directories can be created to monitor subsets of tasks in the CTRL_MON 187group that is their ancestor. These are called "MON" groups in the rest 188of this document. 189 190Removing a directory will move all tasks and cpus owned by the group it 191represents to the parent. Removing one of the created CTRL_MON groups 192will automatically remove all MON groups below it. 193 194All groups contain the following files: 195 196"tasks": 197 Reading this file shows the list of all tasks that belong to 198 this group. Writing a task id to the file will add a task to the 199 group. If the group is a CTRL_MON group the task is removed from 200 whichever previous CTRL_MON group owned the task and also from 201 any MON group that owned the task. If the group is a MON group, 202 then the task must already belong to the CTRL_MON parent of this 203 group. The task is removed from any previous MON group. 204 205 206"cpus": 207 Reading this file shows a bitmask of the logical CPUs owned by 208 this group. Writing a mask to this file will add and remove 209 CPUs to/from this group. As with the tasks file a hierarchy is 210 maintained where MON groups may only include CPUs owned by the 211 parent CTRL_MON group. 212 When the resource group is in pseudo-locked mode this file will 213 only be readable, reflecting the CPUs associated with the 214 pseudo-locked region. 215 216 217"cpus_list": 218 Just like "cpus", only using ranges of CPUs instead of bitmasks. 219 220 221When control is enabled all CTRL_MON groups will also contain: 222 223"schemata": 224 A list of all the resources available to this group. 225 Each resource has its own line and format - see below for details. 226 227"size": 228 Mirrors the display of the "schemata" file to display the size in 229 bytes of each allocation instead of the bits representing the 230 allocation. 231 232"mode": 233 The "mode" of the resource group dictates the sharing of its 234 allocations. A "shareable" resource group allows sharing of its 235 allocations while an "exclusive" resource group does not. A 236 cache pseudo-locked region is created by first writing 237 "pseudo-locksetup" to the "mode" file before writing the cache 238 pseudo-locked region's schemata to the resource group's "schemata" 239 file. On successful pseudo-locked region creation the mode will 240 automatically change to "pseudo-locked". 241 242When monitoring is enabled all MON groups will also contain: 243 244"mon_data": 245 This contains a set of files organized by L3 domain and by 246 RDT event. E.g. on a system with two L3 domains there will 247 be subdirectories "mon_L3_00" and "mon_L3_01". Each of these 248 directories have one file per event (e.g. "llc_occupancy", 249 "mbm_total_bytes", and "mbm_local_bytes"). In a MON group these 250 files provide a read out of the current value of the event for 251 all tasks in the group. In CTRL_MON groups these files provide 252 the sum for all tasks in the CTRL_MON group and all tasks in 253 MON groups. Please see example section for more details on usage. 254 255Resource allocation rules 256------------------------- 257 258When a task is running the following rules define which resources are 259available to it: 260 2611) If the task is a member of a non-default group, then the schemata 262 for that group is used. 263 2642) Else if the task belongs to the default group, but is running on a 265 CPU that is assigned to some specific group, then the schemata for the 266 CPU's group is used. 267 2683) Otherwise the schemata for the default group is used. 269 270Resource monitoring rules 271------------------------- 2721) If a task is a member of a MON group, or non-default CTRL_MON group 273 then RDT events for the task will be reported in that group. 274 2752) If a task is a member of the default CTRL_MON group, but is running 276 on a CPU that is assigned to some specific group, then the RDT events 277 for the task will be reported in that group. 278 2793) Otherwise RDT events for the task will be reported in the root level 280 "mon_data" group. 281 282 283Notes on cache occupancy monitoring and control 284=============================================== 285When moving a task from one group to another you should remember that 286this only affects *new* cache allocations by the task. E.g. you may have 287a task in a monitor group showing 3 MB of cache occupancy. If you move 288to a new group and immediately check the occupancy of the old and new 289groups you will likely see that the old group is still showing 3 MB and 290the new group zero. When the task accesses locations still in cache from 291before the move, the h/w does not update any counters. On a busy system 292you will likely see the occupancy in the old group go down as cache lines 293are evicted and re-used while the occupancy in the new group rises as 294the task accesses memory and loads into the cache are counted based on 295membership in the new group. 296 297The same applies to cache allocation control. Moving a task to a group 298with a smaller cache partition will not evict any cache lines. The 299process may continue to use them from the old partition. 300 301Hardware uses CLOSid(Class of service ID) and an RMID(Resource monitoring ID) 302to identify a control group and a monitoring group respectively. Each of 303the resource groups are mapped to these IDs based on the kind of group. The 304number of CLOSid and RMID are limited by the hardware and hence the creation of 305a "CTRL_MON" directory may fail if we run out of either CLOSID or RMID 306and creation of "MON" group may fail if we run out of RMIDs. 307 308max_threshold_occupancy - generic concepts 309------------------------------------------ 310 311Note that an RMID once freed may not be immediately available for use as 312the RMID is still tagged the cache lines of the previous user of RMID. 313Hence such RMIDs are placed on limbo list and checked back if the cache 314occupancy has gone down. If there is a time when system has a lot of 315limbo RMIDs but which are not ready to be used, user may see an -EBUSY 316during mkdir. 317 318max_threshold_occupancy is a user configurable value to determine the 319occupancy at which an RMID can be freed. 320 321Schemata files - general concepts 322--------------------------------- 323Each line in the file describes one resource. The line starts with 324the name of the resource, followed by specific values to be applied 325in each of the instances of that resource on the system. 326 327Cache IDs 328--------- 329On current generation systems there is one L3 cache per socket and L2 330caches are generally just shared by the hyperthreads on a core, but this 331isn't an architectural requirement. We could have multiple separate L3 332caches on a socket, multiple cores could share an L2 cache. So instead 333of using "socket" or "core" to define the set of logical cpus sharing 334a resource we use a "Cache ID". At a given cache level this will be a 335unique number across the whole system (but it isn't guaranteed to be a 336contiguous sequence, there may be gaps). To find the ID for each logical 337CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id 338 339Cache Bit Masks (CBM) 340--------------------- 341For cache resources we describe the portion of the cache that is available 342for allocation using a bitmask. The maximum value of the mask is defined 343by each cpu model (and may be different for different cache levels). It 344is found using CPUID, but is also provided in the "info" directory of 345the resctrl file system in "info/{resource}/cbm_mask". Intel hardware 346requires that these masks have all the '1' bits in a contiguous block. So 3470x3, 0x6 and 0xC are legal 4-bit masks with two bits set, but 0x5, 0x9 348and 0xA are not. On a system with a 20-bit mask each bit represents 5% 349of the capacity of the cache. You could partition the cache into four 350equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000. 351 352Memory bandwidth Allocation and monitoring 353========================================== 354 355For Memory bandwidth resource, by default the user controls the resource 356by indicating the percentage of total memory bandwidth. 357 358The minimum bandwidth percentage value for each cpu model is predefined 359and can be looked up through "info/MB/min_bandwidth". The bandwidth 360granularity that is allocated is also dependent on the cpu model and can 361be looked up at "info/MB/bandwidth_gran". The available bandwidth 362control steps are: min_bw + N * bw_gran. Intermediate values are rounded 363to the next control step available on the hardware. 364 365The bandwidth throttling is a core specific mechanism on some of Intel 366SKUs. Using a high bandwidth and a low bandwidth setting on two threads 367sharing a core will result in both threads being throttled to use the 368low bandwidth. The fact that Memory bandwidth allocation(MBA) is a core 369specific mechanism where as memory bandwidth monitoring(MBM) is done at 370the package level may lead to confusion when users try to apply control 371via the MBA and then monitor the bandwidth to see if the controls are 372effective. Below are such scenarios: 373 3741. User may *not* see increase in actual bandwidth when percentage 375 values are increased: 376 377This can occur when aggregate L2 external bandwidth is more than L3 378external bandwidth. Consider an SKL SKU with 24 cores on a package and 379where L2 external is 10GBps (hence aggregate L2 external bandwidth is 380240GBps) and L3 external bandwidth is 100GBps. Now a workload with '20 381threads, having 50% bandwidth, each consuming 5GBps' consumes the max L3 382bandwidth of 100GBps although the percentage value specified is only 50% 383<< 100%. Hence increasing the bandwidth percentage will not yield any 384more bandwidth. This is because although the L2 external bandwidth still 385has capacity, the L3 external bandwidth is fully used. Also note that 386this would be dependent on number of cores the benchmark is run on. 387 3882. Same bandwidth percentage may mean different actual bandwidth 389 depending on # of threads: 390 391For the same SKU in #1, a 'single thread, with 10% bandwidth' and '4 392thread, with 10% bandwidth' can consume upto 10GBps and 40GBps although 393they have same percentage bandwidth of 10%. This is simply because as 394threads start using more cores in an rdtgroup, the actual bandwidth may 395increase or vary although user specified bandwidth percentage is same. 396 397In order to mitigate this and make the interface more user friendly, 398resctrl added support for specifying the bandwidth in MBps as well. The 399kernel underneath would use a software feedback mechanism or a "Software 400Controller(mba_sc)" which reads the actual bandwidth using MBM counters 401and adjust the memory bandwidth percentages to ensure:: 402 403 "actual bandwidth < user specified bandwidth". 404 405By default, the schemata would take the bandwidth percentage values 406where as user can switch to the "MBA software controller" mode using 407a mount option 'mba_MBps'. The schemata format is specified in the below 408sections. 409 410L3 schemata file details (code and data prioritization disabled) 411---------------------------------------------------------------- 412With CDP disabled the L3 schemata format is:: 413 414 L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 415 416L3 schemata file details (CDP enabled via mount option to resctrl) 417------------------------------------------------------------------ 418When CDP is enabled L3 control is split into two separate resources 419so you can specify independent masks for code and data like this:: 420 421 L3DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 422 L3CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 423 424L2 schemata file details 425------------------------ 426CDP is supported at L2 using the 'cdpl2' mount option. The schemata 427format is either:: 428 429 L2:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 430 431or 432 433 L2DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 434 L2CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 435 436 437Memory bandwidth Allocation (default mode) 438------------------------------------------ 439 440Memory b/w domain is L3 cache. 441:: 442 443 MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;... 444 445Memory bandwidth Allocation specified in MBps 446--------------------------------------------- 447 448Memory bandwidth domain is L3 cache. 449:: 450 451 MB:<cache_id0>=bw_MBps0;<cache_id1>=bw_MBps1;... 452 453Reading/writing the schemata file 454--------------------------------- 455Reading the schemata file will show the state of all resources 456on all domains. When writing you only need to specify those values 457which you wish to change. E.g. 458:: 459 460 # cat schemata 461 L3DATA:0=fffff;1=fffff;2=fffff;3=fffff 462 L3CODE:0=fffff;1=fffff;2=fffff;3=fffff 463 # echo "L3DATA:2=3c0;" > schemata 464 # cat schemata 465 L3DATA:0=fffff;1=fffff;2=3c0;3=fffff 466 L3CODE:0=fffff;1=fffff;2=fffff;3=fffff 467 468Cache Pseudo-Locking 469==================== 470CAT enables a user to specify the amount of cache space that an 471application can fill. Cache pseudo-locking builds on the fact that a 472CPU can still read and write data pre-allocated outside its current 473allocated area on a cache hit. With cache pseudo-locking, data can be 474preloaded into a reserved portion of cache that no application can 475fill, and from that point on will only serve cache hits. The cache 476pseudo-locked memory is made accessible to user space where an 477application can map it into its virtual address space and thus have 478a region of memory with reduced average read latency. 479 480The creation of a cache pseudo-locked region is triggered by a request 481from the user to do so that is accompanied by a schemata of the region 482to be pseudo-locked. The cache pseudo-locked region is created as follows: 483 484- Create a CAT allocation CLOSNEW with a CBM matching the schemata 485 from the user of the cache region that will contain the pseudo-locked 486 memory. This region must not overlap with any current CAT allocation/CLOS 487 on the system and no future overlap with this cache region is allowed 488 while the pseudo-locked region exists. 489- Create a contiguous region of memory of the same size as the cache 490 region. 491- Flush the cache, disable hardware prefetchers, disable preemption. 492- Make CLOSNEW the active CLOS and touch the allocated memory to load 493 it into the cache. 494- Set the previous CLOS as active. 495- At this point the closid CLOSNEW can be released - the cache 496 pseudo-locked region is protected as long as its CBM does not appear in 497 any CAT allocation. Even though the cache pseudo-locked region will from 498 this point on not appear in any CBM of any CLOS an application running with 499 any CLOS will be able to access the memory in the pseudo-locked region since 500 the region continues to serve cache hits. 501- The contiguous region of memory loaded into the cache is exposed to 502 user-space as a character device. 503 504Cache pseudo-locking increases the probability that data will remain 505in the cache via carefully configuring the CAT feature and controlling 506application behavior. There is no guarantee that data is placed in 507cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict 508“locked” data from cache. Power management C-states may shrink or 509power off cache. Deeper C-states will automatically be restricted on 510pseudo-locked region creation. 511 512It is required that an application using a pseudo-locked region runs 513with affinity to the cores (or a subset of the cores) associated 514with the cache on which the pseudo-locked region resides. A sanity check 515within the code will not allow an application to map pseudo-locked memory 516unless it runs with affinity to cores associated with the cache on which the 517pseudo-locked region resides. The sanity check is only done during the 518initial mmap() handling, there is no enforcement afterwards and the 519application self needs to ensure it remains affine to the correct cores. 520 521Pseudo-locking is accomplished in two stages: 522 5231) During the first stage the system administrator allocates a portion 524 of cache that should be dedicated to pseudo-locking. At this time an 525 equivalent portion of memory is allocated, loaded into allocated 526 cache portion, and exposed as a character device. 5272) During the second stage a user-space application maps (mmap()) the 528 pseudo-locked memory into its address space. 529 530Cache Pseudo-Locking Interface 531------------------------------ 532A pseudo-locked region is created using the resctrl interface as follows: 533 5341) Create a new resource group by creating a new directory in /sys/fs/resctrl. 5352) Change the new resource group's mode to "pseudo-locksetup" by writing 536 "pseudo-locksetup" to the "mode" file. 5373) Write the schemata of the pseudo-locked region to the "schemata" file. All 538 bits within the schemata should be "unused" according to the "bit_usage" 539 file. 540 541On successful pseudo-locked region creation the "mode" file will contain 542"pseudo-locked" and a new character device with the same name as the resource 543group will exist in /dev/pseudo_lock. This character device can be mmap()'ed 544by user space in order to obtain access to the pseudo-locked memory region. 545 546An example of cache pseudo-locked region creation and usage can be found below. 547 548Cache Pseudo-Locking Debugging Interface 549---------------------------------------- 550The pseudo-locking debugging interface is enabled by default (if 551CONFIG_DEBUG_FS is enabled) and can be found in /sys/kernel/debug/resctrl. 552 553There is no explicit way for the kernel to test if a provided memory 554location is present in the cache. The pseudo-locking debugging interface uses 555the tracing infrastructure to provide two ways to measure cache residency of 556the pseudo-locked region: 557 5581) Memory access latency using the pseudo_lock_mem_latency tracepoint. Data 559 from these measurements are best visualized using a hist trigger (see 560 example below). In this test the pseudo-locked region is traversed at 561 a stride of 32 bytes while hardware prefetchers and preemption 562 are disabled. This also provides a substitute visualization of cache 563 hits and misses. 5642) Cache hit and miss measurements using model specific precision counters if 565 available. Depending on the levels of cache on the system the pseudo_lock_l2 566 and pseudo_lock_l3 tracepoints are available. 567 568When a pseudo-locked region is created a new debugfs directory is created for 569it in debugfs as /sys/kernel/debug/resctrl/<newdir>. A single 570write-only file, pseudo_lock_measure, is present in this directory. The 571measurement of the pseudo-locked region depends on the number written to this 572debugfs file: 573 5741: 575 writing "1" to the pseudo_lock_measure file will trigger the latency 576 measurement captured in the pseudo_lock_mem_latency tracepoint. See 577 example below. 5782: 579 writing "2" to the pseudo_lock_measure file will trigger the L2 cache 580 residency (cache hits and misses) measurement captured in the 581 pseudo_lock_l2 tracepoint. See example below. 5823: 583 writing "3" to the pseudo_lock_measure file will trigger the L3 cache 584 residency (cache hits and misses) measurement captured in the 585 pseudo_lock_l3 tracepoint. 586 587All measurements are recorded with the tracing infrastructure. This requires 588the relevant tracepoints to be enabled before the measurement is triggered. 589 590Example of latency debugging interface 591~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 592In this example a pseudo-locked region named "newlock" was created. Here is 593how we can measure the latency in cycles of reading from this region and 594visualize this data with a histogram that is available if CONFIG_HIST_TRIGGERS 595is set:: 596 597 # :> /sys/kernel/debug/tracing/trace 598 # echo 'hist:keys=latency' > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/trigger 599 # echo 1 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/enable 600 # echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure 601 # echo 0 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/enable 602 # cat /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/hist 603 604 # event histogram 605 # 606 # trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active] 607 # 608 609 { latency: 456 } hitcount: 1 610 { latency: 50 } hitcount: 83 611 { latency: 36 } hitcount: 96 612 { latency: 44 } hitcount: 174 613 { latency: 48 } hitcount: 195 614 { latency: 46 } hitcount: 262 615 { latency: 42 } hitcount: 693 616 { latency: 40 } hitcount: 3204 617 { latency: 38 } hitcount: 3484 618 619 Totals: 620 Hits: 8192 621 Entries: 9 622 Dropped: 0 623 624Example of cache hits/misses debugging 625~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 626In this example a pseudo-locked region named "newlock" was created on the L2 627cache of a platform. Here is how we can obtain details of the cache hits 628and misses using the platform's precision counters. 629:: 630 631 # :> /sys/kernel/debug/tracing/trace 632 # echo 1 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_l2/enable 633 # echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure 634 # echo 0 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_l2/enable 635 # cat /sys/kernel/debug/tracing/trace 636 637 # tracer: nop 638 # 639 # _-----=> irqs-off 640 # / _----=> need-resched 641 # | / _---=> hardirq/softirq 642 # || / _--=> preempt-depth 643 # ||| / delay 644 # TASK-PID CPU# |||| TIMESTAMP FUNCTION 645 # | | | |||| | | 646 pseudo_lock_mea-1672 [002] .... 3132.860500: pseudo_lock_l2: hits=4097 miss=0 647 648 649Examples for RDT allocation usage 650~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 651 6521) Example 1 653 654On a two socket machine (one L3 cache per socket) with just four bits 655for cache bit masks, minimum b/w of 10% with a memory bandwidth 656granularity of 10%. 657:: 658 659 # mount -t resctrl resctrl /sys/fs/resctrl 660 # cd /sys/fs/resctrl 661 # mkdir p0 p1 662 # echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata 663 # echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata 664 665The default resource group is unmodified, so we have access to all parts 666of all caches (its schemata file reads "L3:0=f;1=f"). 667 668Tasks that are under the control of group "p0" may only allocate from the 669"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. 670Tasks in group "p1" use the "lower" 50% of cache on both sockets. 671 672Similarly, tasks that are under the control of group "p0" may use a 673maximum memory b/w of 50% on socket0 and 50% on socket 1. 674Tasks in group "p1" may also use 50% memory b/w on both sockets. 675Note that unlike cache masks, memory b/w cannot specify whether these 676allocations can overlap or not. The allocations specifies the maximum 677b/w that the group may be able to use and the system admin can configure 678the b/w accordingly. 679 680If resctrl is using the software controller (mba_sc) then user can enter the 681max b/w in MB rather than the percentage values. 682:: 683 684 # echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata 685 # echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata 686 687In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w 688of 1024MB where as on socket 1 they would use 500MB. 689 6902) Example 2 691 692Again two sockets, but this time with a more realistic 20-bit mask. 693 694Two real time tasks pid=1234 running on processor 0 and pid=5678 running on 695processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy 696neighbors, each of the two real-time tasks exclusively occupies one quarter 697of L3 cache on socket 0. 698:: 699 700 # mount -t resctrl resctrl /sys/fs/resctrl 701 # cd /sys/fs/resctrl 702 703First we reset the schemata for the default group so that the "upper" 70450% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by 705ordinary tasks:: 706 707 # echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata 708 709Next we make a resource group for our first real time task and give 710it access to the "top" 25% of the cache on socket 0. 711:: 712 713 # mkdir p0 714 # echo "L3:0=f8000;1=fffff" > p0/schemata 715 716Finally we move our first real time task into this resource group. We 717also use taskset(1) to ensure the task always runs on a dedicated CPU 718on socket 0. Most uses of resource groups will also constrain which 719processors tasks run on. 720:: 721 722 # echo 1234 > p0/tasks 723 # taskset -cp 1 1234 724 725Ditto for the second real time task (with the remaining 25% of cache):: 726 727 # mkdir p1 728 # echo "L3:0=7c00;1=fffff" > p1/schemata 729 # echo 5678 > p1/tasks 730 # taskset -cp 2 5678 731 732For the same 2 socket system with memory b/w resource and CAT L3 the 733schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is 73410): 735 736For our first real time task this would request 20% memory b/w on socket 0. 737:: 738 739 # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata 740 741For our second real time task this would request an other 20% memory b/w 742on socket 0. 743:: 744 745 # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata 746 7473) Example 3 748 749A single socket system which has real-time tasks running on core 4-7 and 750non real-time workload assigned to core 0-3. The real-time tasks share text 751and data, so a per task association is not required and due to interaction 752with the kernel it's desired that the kernel on these cores shares L3 with 753the tasks. 754:: 755 756 # mount -t resctrl resctrl /sys/fs/resctrl 757 # cd /sys/fs/resctrl 758 759First we reset the schemata for the default group so that the "upper" 76050% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0 761cannot be used by ordinary tasks:: 762 763 # echo "L3:0=3ff\nMB:0=50" > schemata 764 765Next we make a resource group for our real time cores and give it access 766to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on 767socket 0. 768:: 769 770 # mkdir p0 771 # echo "L3:0=ffc00\nMB:0=50" > p0/schemata 772 773Finally we move core 4-7 over to the new group and make sure that the 774kernel and the tasks running there get 50% of the cache. They should 775also get 50% of memory bandwidth assuming that the cores 4-7 are SMT 776siblings and only the real time threads are scheduled on the cores 4-7. 777:: 778 779 # echo F0 > p0/cpus 780 7814) Example 4 782 783The resource groups in previous examples were all in the default "shareable" 784mode allowing sharing of their cache allocations. If one resource group 785configures a cache allocation then nothing prevents another resource group 786to overlap with that allocation. 787 788In this example a new exclusive resource group will be created on a L2 CAT 789system with two L2 cache instances that can be configured with an 8-bit 790capacity bitmask. The new exclusive resource group will be configured to use 79125% of each cache instance. 792:: 793 794 # mount -t resctrl resctrl /sys/fs/resctrl/ 795 # cd /sys/fs/resctrl 796 797First, we observe that the default group is configured to allocate to all L2 798cache:: 799 800 # cat schemata 801 L2:0=ff;1=ff 802 803We could attempt to create the new resource group at this point, but it will 804fail because of the overlap with the schemata of the default group:: 805 806 # mkdir p0 807 # echo 'L2:0=0x3;1=0x3' > p0/schemata 808 # cat p0/mode 809 shareable 810 # echo exclusive > p0/mode 811 -sh: echo: write error: Invalid argument 812 # cat info/last_cmd_status 813 schemata overlaps 814 815To ensure that there is no overlap with another resource group the default 816resource group's schemata has to change, making it possible for the new 817resource group to become exclusive. 818:: 819 820 # echo 'L2:0=0xfc;1=0xfc' > schemata 821 # echo exclusive > p0/mode 822 # grep . p0/* 823 p0/cpus:0 824 p0/mode:exclusive 825 p0/schemata:L2:0=03;1=03 826 p0/size:L2:0=262144;1=262144 827 828A new resource group will on creation not overlap with an exclusive resource 829group:: 830 831 # mkdir p1 832 # grep . p1/* 833 p1/cpus:0 834 p1/mode:shareable 835 p1/schemata:L2:0=fc;1=fc 836 p1/size:L2:0=786432;1=786432 837 838The bit_usage will reflect how the cache is used:: 839 840 # cat info/L2/bit_usage 841 0=SSSSSSEE;1=SSSSSSEE 842 843A resource group cannot be forced to overlap with an exclusive resource group:: 844 845 # echo 'L2:0=0x1;1=0x1' > p1/schemata 846 -sh: echo: write error: Invalid argument 847 # cat info/last_cmd_status 848 overlaps with exclusive group 849 850Example of Cache Pseudo-Locking 851~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 852Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked 853region is exposed at /dev/pseudo_lock/newlock that can be provided to 854application for argument to mmap(). 855:: 856 857 # mount -t resctrl resctrl /sys/fs/resctrl/ 858 # cd /sys/fs/resctrl 859 860Ensure that there are bits available that can be pseudo-locked, since only 861unused bits can be pseudo-locked the bits to be pseudo-locked needs to be 862removed from the default resource group's schemata:: 863 864 # cat info/L2/bit_usage 865 0=SSSSSSSS;1=SSSSSSSS 866 # echo 'L2:1=0xfc' > schemata 867 # cat info/L2/bit_usage 868 0=SSSSSSSS;1=SSSSSS00 869 870Create a new resource group that will be associated with the pseudo-locked 871region, indicate that it will be used for a pseudo-locked region, and 872configure the requested pseudo-locked region capacity bitmask:: 873 874 # mkdir newlock 875 # echo pseudo-locksetup > newlock/mode 876 # echo 'L2:1=0x3' > newlock/schemata 877 878On success the resource group's mode will change to pseudo-locked, the 879bit_usage will reflect the pseudo-locked region, and the character device 880exposing the pseudo-locked region will exist:: 881 882 # cat newlock/mode 883 pseudo-locked 884 # cat info/L2/bit_usage 885 0=SSSSSSSS;1=SSSSSSPP 886 # ls -l /dev/pseudo_lock/newlock 887 crw------- 1 root root 243, 0 Apr 3 05:01 /dev/pseudo_lock/newlock 888 889:: 890 891 /* 892 * Example code to access one page of pseudo-locked cache region 893 * from user space. 894 */ 895 #define _GNU_SOURCE 896 #include <fcntl.h> 897 #include <sched.h> 898 #include <stdio.h> 899 #include <stdlib.h> 900 #include <unistd.h> 901 #include <sys/mman.h> 902 903 /* 904 * It is required that the application runs with affinity to only 905 * cores associated with the pseudo-locked region. Here the cpu 906 * is hardcoded for convenience of example. 907 */ 908 static int cpuid = 2; 909 910 int main(int argc, char *argv[]) 911 { 912 cpu_set_t cpuset; 913 long page_size; 914 void *mapping; 915 int dev_fd; 916 int ret; 917 918 page_size = sysconf(_SC_PAGESIZE); 919 920 CPU_ZERO(&cpuset); 921 CPU_SET(cpuid, &cpuset); 922 ret = sched_setaffinity(0, sizeof(cpuset), &cpuset); 923 if (ret < 0) { 924 perror("sched_setaffinity"); 925 exit(EXIT_FAILURE); 926 } 927 928 dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR); 929 if (dev_fd < 0) { 930 perror("open"); 931 exit(EXIT_FAILURE); 932 } 933 934 mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, 935 dev_fd, 0); 936 if (mapping == MAP_FAILED) { 937 perror("mmap"); 938 close(dev_fd); 939 exit(EXIT_FAILURE); 940 } 941 942 /* Application interacts with pseudo-locked memory @mapping */ 943 944 ret = munmap(mapping, page_size); 945 if (ret < 0) { 946 perror("munmap"); 947 close(dev_fd); 948 exit(EXIT_FAILURE); 949 } 950 951 close(dev_fd); 952 exit(EXIT_SUCCESS); 953 } 954 955Locking between applications 956---------------------------- 957 958Certain operations on the resctrl filesystem, composed of read/writes 959to/from multiple files, must be atomic. 960 961As an example, the allocation of an exclusive reservation of L3 cache 962involves: 963 964 1. Read the cbmmasks from each directory or the per-resource "bit_usage" 965 2. Find a contiguous set of bits in the global CBM bitmask that is clear 966 in any of the directory cbmmasks 967 3. Create a new directory 968 4. Set the bits found in step 2 to the new directory "schemata" file 969 970If two applications attempt to allocate space concurrently then they can 971end up allocating the same bits so the reservations are shared instead of 972exclusive. 973 974To coordinate atomic operations on the resctrlfs and to avoid the problem 975above, the following locking procedure is recommended: 976 977Locking is based on flock, which is available in libc and also as a shell 978script command 979 980Write lock: 981 982 A) Take flock(LOCK_EX) on /sys/fs/resctrl 983 B) Read/write the directory structure. 984 C) funlock 985 986Read lock: 987 988 A) Take flock(LOCK_SH) on /sys/fs/resctrl 989 B) If success read the directory structure. 990 C) funlock 991 992Example with bash:: 993 994 # Atomically read directory structure 995 $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl 996 997 # Read directory contents and create new subdirectory 998 999 $ cat create-dir.sh 1000 find /sys/fs/resctrl/ > output.txt 1001 mask = function-of(output.txt) 1002 mkdir /sys/fs/resctrl/newres/ 1003 echo mask > /sys/fs/resctrl/newres/schemata 1004 1005 $ flock /sys/fs/resctrl/ ./create-dir.sh 1006 1007Example with C:: 1008 1009 /* 1010 * Example code do take advisory locks 1011 * before accessing resctrl filesystem 1012 */ 1013 #include <sys/file.h> 1014 #include <stdlib.h> 1015 1016 void resctrl_take_shared_lock(int fd) 1017 { 1018 int ret; 1019 1020 /* take shared lock on resctrl filesystem */ 1021 ret = flock(fd, LOCK_SH); 1022 if (ret) { 1023 perror("flock"); 1024 exit(-1); 1025 } 1026 } 1027 1028 void resctrl_take_exclusive_lock(int fd) 1029 { 1030 int ret; 1031 1032 /* release lock on resctrl filesystem */ 1033 ret = flock(fd, LOCK_EX); 1034 if (ret) { 1035 perror("flock"); 1036 exit(-1); 1037 } 1038 } 1039 1040 void resctrl_release_lock(int fd) 1041 { 1042 int ret; 1043 1044 /* take shared lock on resctrl filesystem */ 1045 ret = flock(fd, LOCK_UN); 1046 if (ret) { 1047 perror("flock"); 1048 exit(-1); 1049 } 1050 } 1051 1052 void main(void) 1053 { 1054 int fd, ret; 1055 1056 fd = open("/sys/fs/resctrl", O_DIRECTORY); 1057 if (fd == -1) { 1058 perror("open"); 1059 exit(-1); 1060 } 1061 resctrl_take_shared_lock(fd); 1062 /* code to read directory contents */ 1063 resctrl_release_lock(fd); 1064 1065 resctrl_take_exclusive_lock(fd); 1066 /* code to read and write directory contents */ 1067 resctrl_release_lock(fd); 1068 } 1069 1070Examples for RDT Monitoring along with allocation usage 1071======================================================= 1072Reading monitored data 1073---------------------- 1074Reading an event file (for ex: mon_data/mon_L3_00/llc_occupancy) would 1075show the current snapshot of LLC occupancy of the corresponding MON 1076group or CTRL_MON group. 1077 1078 1079Example 1 (Monitor CTRL_MON group and subset of tasks in CTRL_MON group) 1080------------------------------------------------------------------------ 1081On a two socket machine (one L3 cache per socket) with just four bits 1082for cache bit masks:: 1083 1084 # mount -t resctrl resctrl /sys/fs/resctrl 1085 # cd /sys/fs/resctrl 1086 # mkdir p0 p1 1087 # echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata 1088 # echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata 1089 # echo 5678 > p1/tasks 1090 # echo 5679 > p1/tasks 1091 1092The default resource group is unmodified, so we have access to all parts 1093of all caches (its schemata file reads "L3:0=f;1=f"). 1094 1095Tasks that are under the control of group "p0" may only allocate from the 1096"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. 1097Tasks in group "p1" use the "lower" 50% of cache on both sockets. 1098 1099Create monitor groups and assign a subset of tasks to each monitor group. 1100:: 1101 1102 # cd /sys/fs/resctrl/p1/mon_groups 1103 # mkdir m11 m12 1104 # echo 5678 > m11/tasks 1105 # echo 5679 > m12/tasks 1106 1107fetch data (data shown in bytes) 1108:: 1109 1110 # cat m11/mon_data/mon_L3_00/llc_occupancy 1111 16234000 1112 # cat m11/mon_data/mon_L3_01/llc_occupancy 1113 14789000 1114 # cat m12/mon_data/mon_L3_00/llc_occupancy 1115 16789000 1116 1117The parent ctrl_mon group shows the aggregated data. 1118:: 1119 1120 # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy 1121 31234000 1122 1123Example 2 (Monitor a task from its creation) 1124-------------------------------------------- 1125On a two socket machine (one L3 cache per socket):: 1126 1127 # mount -t resctrl resctrl /sys/fs/resctrl 1128 # cd /sys/fs/resctrl 1129 # mkdir p0 p1 1130 1131An RMID is allocated to the group once its created and hence the <cmd> 1132below is monitored from its creation. 1133:: 1134 1135 # echo $$ > /sys/fs/resctrl/p1/tasks 1136 # <cmd> 1137 1138Fetch the data:: 1139 1140 # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy 1141 31789000 1142 1143Example 3 (Monitor without CAT support or before creating CAT groups) 1144--------------------------------------------------------------------- 1145 1146Assume a system like HSW has only CQM and no CAT support. In this case 1147the resctrl will still mount but cannot create CTRL_MON directories. 1148But user can create different MON groups within the root group thereby 1149able to monitor all tasks including kernel threads. 1150 1151This can also be used to profile jobs cache size footprint before being 1152able to allocate them to different allocation groups. 1153:: 1154 1155 # mount -t resctrl resctrl /sys/fs/resctrl 1156 # cd /sys/fs/resctrl 1157 # mkdir mon_groups/m01 1158 # mkdir mon_groups/m02 1159 1160 # echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks 1161 # echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks 1162 1163Monitor the groups separately and also get per domain data. From the 1164below its apparent that the tasks are mostly doing work on 1165domain(socket) 0. 1166:: 1167 1168 # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy 1169 31234000 1170 # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy 1171 34555 1172 # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy 1173 31234000 1174 # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy 1175 32789 1176 1177 1178Example 4 (Monitor real time tasks) 1179----------------------------------- 1180 1181A single socket system which has real time tasks running on cores 4-7 1182and non real time tasks on other cpus. We want to monitor the cache 1183occupancy of the real time threads on these cores. 1184:: 1185 1186 # mount -t resctrl resctrl /sys/fs/resctrl 1187 # cd /sys/fs/resctrl 1188 # mkdir p1 1189 1190Move the cpus 4-7 over to p1:: 1191 1192 # echo f0 > p1/cpus 1193 1194View the llc occupancy snapshot:: 1195 1196 # cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy 1197 11234000 1198