1* Texas Instruments CPSW ethernet driver 2 3Multiqueue & CBS & MQPRIO 4===================================================================== 5===================================================================== 6 7The cpsw has 3 CBS shapers for each external ports. This document 8describes MQPRIO and CBS Qdisc offload configuration for cpsw driver 9based on examples. It potentially can be used in audio video bridging 10(AVB) and time sensitive networking (TSN). 11 12The following examples were tested on AM572x EVM and BBB boards. 13 14Test setup 15========== 16 17Under consideration two examples with AM572x EVM running cpsw driver 18in dual_emac mode. 19 20Several prerequisites: 21- TX queues must be rated starting from txq0 that has highest priority 22- Traffic classes are used starting from 0, that has highest priority 23- CBS shapers should be used with rated queues 24- The bandwidth for CBS shapers has to be set a little bit more then 25 potential incoming rate, thus, rate of all incoming tx queues has 26 to be a little less 27- Real rates can differ, due to discreetness 28- Map skb-priority to txq is not enough, also skb-priority to l2 prio 29 map has to be created with ip or vconfig tool 30- Any l2/socket prio (0 - 7) for classes can be used, but for 31 simplicity default values are used: 3 and 2 32- only 2 classes tested: A and B, but checked and can work with more, 33 maximum allowed 4, but only for 3 rate can be set. 34 35Test setup for examples 36======================= 37 +-------------------------------+ 38 |--+ | 39 | | Workstation0 | 40 |E | MAC 18:03:73:66:87:42 | 41+-----------------------------+ +--|t | | 42| | 1 | E | | |h |./tsn_listener -d \ | 43| Target board: | 0 | t |--+ |0 | 18:03:73:66:87:42 -i eth0 \| 44| AM572x EVM | 0 | h | | | -s 1500 | 45| | 0 | 0 | |--+ | 46| Only 2 classes: |Mb +---| +-------------------------------+ 47| class A, class B | | 48| | +---| +-------------------------------+ 49| | 1 | E | |--+ | 50| | 0 | t | | | Workstation1 | 51| | 0 | h |--+ |E | MAC 20:cf:30:85:7d:fd | 52| |Mb | 1 | +--|t | | 53+-----------------------------+ |h |./tsn_listener -d \ | 54 |0 | 20:cf:30:85:7d:fd -i eth0 \| 55 | | -s 1500 | 56 |--+ | 57 +-------------------------------+ 58 59********************************************************************* 60********************************************************************* 61********************************************************************* 62Example 1: One port tx AVB configuration scheme for target board 63---------------------------------------------------------------------- 64(prints and scheme for AM572x evm, applicable for single port boards) 65 66tc - traffic class 67txq - transmit queue 68p - priority 69f - fifo (cpsw fifo) 70S - shaper configured 71 72+------------------------------------------------------------------+ u 73| +---------------+ +---------------+ +------+ +------+ | s 74| | | | | | | | | | e 75| | App 1 | | App 2 | | Apps | | Apps | | r 76| | Class A | | Class B | | Rest | | Rest | | 77| | Eth0 | | Eth0 | | Eth0 | | Eth1 | | s 78| | VLAN100 | | VLAN100 | | | | | | | | p 79| | 40 Mb/s | | 20 Mb/s | | | | | | | | a 80| | SO_PRIORITY=3 | | SO_PRIORITY=2 | | | | | | | | c 81| | | | | | | | | | | | | | e 82| +---|-----------+ +---|-----------+ +---|--+ +---|--+ | 83+-----|------------------|------------------|--------|-------------+ 84 +-+ +------------+ | | 85 | | +-----------------+ +--+ 86 | | | | 87+---|-------|-------------|-----------------------|----------------+ 88| +----+ +----+ +----+ +----+ +----+ | 89| | p3 | | p2 | | p1 | | p0 | | p0 | | k 90| \ / \ / \ / \ / \ / | e 91| \ / \ / \ / \ / \ / | r 92| \/ \/ \/ \/ \/ | n 93| | | | | | e 94| | | +-----+ | | l 95| | | | | | 96| +----+ +----+ +----+ +----+ | s 97| |tc0 | |tc1 | |tc2 | |tc0 | | p 98| \ / \ / \ / \ / | a 99| \ / \ / \ / \ / | c 100| \/ \/ \/ \/ | e 101| | | +-----+ | | 102| | | | | | | 103| | | | | | | 104| | | | | | | 105| +----+ +----+ +----+ +----+ +----+ | 106| |txq0| |txq1| |txq2| |txq3| |txq4| | 107| \ / \ / \ / \ / \ / | 108| \ / \ / \ / \ / \ / | 109| \/ \/ \/ \/ \/ | 110| +-|------|------|------|--+ +--|--------------+ | 111| | | | | | | Eth0.100 | | Eth1 | | 112+---|------|------|------|------------------------|----------------+ 113 | | | | | 114 p p p p | 115 3 2 0-1, 4-7 <- L2 priority | 116 | | | | | 117 | | | | | 118+---|------|------|------|------------------------|----------------+ 119| | | | | |----------+ | 120| +----+ +----+ +----+ +----+ +----+ | 121| |dma7| |dma6| |dma5| |dma4| |dma3| | 122| \ / \ / \ / \ / \ / | c 123| \S / \S / \ / \ / \ / | p 124| \/ \/ \/ \/ \/ | s 125| | | | +----- | | w 126| | | | | | | 127| | | | | | | d 128| +----+ +----+ +----+p p+----+ | r 129| | | | | | |o o| | | i 130| | f3 | | f2 | | f0 |r r| f0 | | v 131| |tc0 | |tc1 | |tc2 |t t|tc0 | | e 132| \CBS / \CBS / \CBS /1 2\CBS / | r 133| \S / \S / \ / \ / | 134| \/ \/ \/ \/ | 135+------------------------------------------------------------------+ 136========================================Eth==========================> 137 1381) 139// Add 4 tx queues, for interface Eth0, and 1 tx queue for Eth1 140$ ethtool -L eth0 rx 1 tx 5 141rx unmodified, ignoring 142 1432) 144// Check if num of queues is set correctly: 145$ ethtool -l eth0 146Channel parameters for eth0: 147Pre-set maximums: 148RX: 8 149TX: 8 150Other: 0 151Combined: 0 152Current hardware settings: 153RX: 1 154TX: 5 155Other: 0 156Combined: 0 157 1583) 159// TX queues must be rated starting from 0, so set bws for tx0 and tx1 160// Set rates 40 and 20 Mb/s appropriately. 161// Pay attention, real speed can differ a bit due to discreetness. 162// Leave last 2 tx queues not rated. 163$ echo 40 > /sys/class/net/eth0/queues/tx-0/tx_maxrate 164$ echo 20 > /sys/class/net/eth0/queues/tx-1/tx_maxrate 165 1664) 167// Check maximum rate of tx (cpdma) queues: 168$ cat /sys/class/net/eth0/queues/tx-*/tx_maxrate 16940 17020 1710 1720 1730 174 1755) 176// Map skb->priority to traffic class: 177// 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2 178// Map traffic class to transmit queue: 179// tc0 -> txq0, tc1 -> txq1, tc2 -> (txq2, txq3) 180$ tc qdisc replace dev eth0 handle 100: parent root mqprio num_tc 3 \ 181map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@0 1@1 2@2 hw 1 182 1835a) 184// As two interface sharing same set of tx queues, assign all traffic 185// coming to interface Eth1 to separate queue in order to not mix it 186// with traffic from interface Eth0, so use separate txq to send 187// packets to Eth1, so all prio -> tc0 and tc0 -> txq4 188// Here hw 0, so here still default configuration for eth1 in hw 189$ tc qdisc replace dev eth1 handle 100: parent root mqprio num_tc 1 \ 190map 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 queues 1@4 hw 0 191 1926) 193// Check classes settings 194$ tc -g class show dev eth0 195+---(100:ffe2) mqprio 196| +---(100:3) mqprio 197| +---(100:4) mqprio 198| 199+---(100:ffe1) mqprio 200| +---(100:2) mqprio 201| 202+---(100:ffe0) mqprio 203 +---(100:1) mqprio 204 205$ tc -g class show dev eth1 206+---(100:ffe0) mqprio 207 +---(100:5) mqprio 208 2097) 210// Set rate for class A - 41 Mbit (tc0, txq0) using CBS Qdisc 211// Set it +1 Mb for reserve (important!) 212// here only idle slope is important, others arg are ignored 213// Pay attention, real speed can differ a bit due to discreetness 214$ tc qdisc add dev eth0 parent 100:1 cbs locredit -1438 \ 215hicredit 62 sendslope -959000 idleslope 41000 offload 1 216net eth0: set FIFO3 bw = 50 217 2188) 219// Set rate for class B - 21 Mbit (tc1, txq1) using CBS Qdisc: 220// Set it +1 Mb for reserve (important!) 221$ tc qdisc add dev eth0 parent 100:2 cbs locredit -1468 \ 222hicredit 65 sendslope -979000 idleslope 21000 offload 1 223net eth0: set FIFO2 bw = 30 224 2259) 226// Create vlan 100 to map sk->priority to vlan qos 227$ ip link add link eth0 name eth0.100 type vlan id 100 2288021q: 802.1Q VLAN Support v1.8 2298021q: adding VLAN 0 to HW filter on device eth0 2308021q: adding VLAN 0 to HW filter on device eth1 231net eth0: Adding vlanid 100 to vlan filter 232 23310) 234// Map skb->priority to L2 prio, 1 to 1 235$ ip link set eth0.100 type vlan \ 236egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 237 23811) 239// Check egress map for vlan 100 240$ cat /proc/net/vlan/eth0.100 241[...] 242INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0 243EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 244 24512) 246// Run your appropriate tools with socket option "SO_PRIORITY" 247// to 3 for class A and/or to 2 for class B 248// (I took at https://www.spinics.net/lists/netdev/msg460869.html) 249./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p3 -s 1500& 250./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p2 -s 1500& 251 25213) 253// run your listener on workstation (should be in same vlan) 254// (I took at https://www.spinics.net/lists/netdev/msg460869.html) 255./tsn_listener -d 18:03:73:66:87:42 -i enp5s0 -s 1500 256Receiving data rate: 39012 kbps 257Receiving data rate: 39012 kbps 258Receiving data rate: 39012 kbps 259Receiving data rate: 39012 kbps 260Receiving data rate: 39012 kbps 261Receiving data rate: 39012 kbps 262Receiving data rate: 39012 kbps 263Receiving data rate: 39012 kbps 264Receiving data rate: 39012 kbps 265Receiving data rate: 39012 kbps 266Receiving data rate: 39012 kbps 267Receiving data rate: 39012 kbps 268Receiving data rate: 39000 kbps 269 27014) 271// Restore default configuration if needed 272$ ip link del eth0.100 273$ tc qdisc del dev eth1 root 274$ tc qdisc del dev eth0 root 275net eth0: Prev FIFO2 is shaped 276net eth0: set FIFO3 bw = 0 277net eth0: set FIFO2 bw = 0 278$ ethtool -L eth0 rx 1 tx 1 279 280********************************************************************* 281********************************************************************* 282********************************************************************* 283Example 2: Two port tx AVB configuration scheme for target board 284---------------------------------------------------------------------- 285(prints and scheme for AM572x evm, for dual emac boards only) 286 287+------------------------------------------------------------------+ u 288| +----------+ +----------+ +------+ +----------+ +----------+ | s 289| | | | | | | | | | | | e 290| | App 1 | | App 2 | | Apps | | App 3 | | App 4 | | r 291| | Class A | | Class B | | Rest | | Class B | | Class A | | 292| | Eth0 | | Eth0 | | | | | Eth1 | | Eth1 | | s 293| | VLAN100 | | VLAN100 | | | | | VLAN100 | | VLAN100 | | p 294| | 40 Mb/s | | 20 Mb/s | | | | | 10 Mb/s | | 30 Mb/s | | a 295| | SO_PRI=3 | | SO_PRI=2 | | | | | SO_PRI=3 | | SO_PRI=2 | | c 296| | | | | | | | | | | | | | | | | e 297| +---|------+ +---|------+ +---|--+ +---|------+ +---|------+ | 298+-----|-------------|-------------|---------|-------------|--------+ 299 +-+ +-------+ | +----------+ +----+ 300 | | +-------+------+ | | 301 | | | | | | 302+---|-------|-------------|--------------|-------------|-------|---+ 303| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | 304| | p3 | | p2 | | p1 | | p0 | | p0 | | p1 | | p2 | | p3 | | k 305| \ / \ / \ / \ / \ / \ / \ / \ / | e 306| \ / \ / \ / \ / \ / \ / \ / \ / | r 307| \/ \/ \/ \/ \/ \/ \/ \/ | n 308| | | | | | | | e 309| | | +----+ +----+ | | | l 310| | | | | | | | 311| +----+ +----+ +----+ +----+ +----+ +----+ | s 312| |tc0 | |tc1 | |tc2 | |tc2 | |tc1 | |tc0 | | p 313| \ / \ / \ / \ / \ / \ / | a 314| \ / \ / \ / \ / \ / \ / | c 315| \/ \/ \/ \/ \/ \/ | e 316| | | +-----+ +-----+ | | | 317| | | | | | | | | | 318| | | | | | | | | | 319| | | | | E E | | | | | 320| +----+ +----+ +----+ +----+ t t +----+ +----+ +----+ +----+ | 321| |txq0| |txq1| |txq4| |txq5| h h |txq6| |txq7| |txq3| |txq2| | 322| \ / \ / \ / \ / 0 1 \ / \ / \ / \ / | 323| \ / \ / \ / \ / . . \ / \ / \ / \ / | 324| \/ \/ \/ \/ 1 1 \/ \/ \/ \/ | 325| +-|------|------|------|--+ 0 0 +-|------|------|------|--+ | 326| | | | | | | 0 0 | | | | | | | 327+---|------|------|------|---------------|------|------|------|----+ 328 | | | | | | | | 329 p p p p p p p p 330 3 2 0-1, 4-7 <-L2 pri-> 0-1, 4-7 2 3 331 | | | | | | | | 332 | | | | | | | | 333+---|------|------|------|---------------|------|------|------|----+ 334| | | | | | | | | | 335| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | 336| |dma7| |dma6| |dma3| |dma2| |dma1| |dma0| |dma4| |dma5| | 337| \ / \ / \ / \ / \ / \ / \ / \ / | c 338| \S / \S / \ / \ / \ / \ / \S / \S / | p 339| \/ \/ \/ \/ \/ \/ \/ \/ | s 340| | | | +----- | | | | | w 341| | | | | +----+ | | | | 342| | | | | | | | | | d 343| +----+ +----+ +----+p p+----+ +----+ +----+ | r 344| | | | | | |o o| | | | | | | i 345| | f3 | | f2 | | f0 |r CPSW r| f3 | | f2 | | f0 | | v 346| |tc0 | |tc1 | |tc2 |t t|tc0 | |tc1 | |tc2 | | e 347| \CBS / \CBS / \CBS /1 2\CBS / \CBS / \CBS / | r 348| \S / \S / \ / \S / \S / \ / | 349| \/ \/ \/ \/ \/ \/ | 350+------------------------------------------------------------------+ 351========================================Eth==========================> 352 3531) 354// Add 8 tx queues, for interface Eth0, but they are common, so are accessed 355// by two interfaces Eth0 and Eth1. 356$ ethtool -L eth1 rx 1 tx 8 357rx unmodified, ignoring 358 3592) 360// Check if num of queues is set correctly: 361$ ethtool -l eth0 362Channel parameters for eth0: 363Pre-set maximums: 364RX: 8 365TX: 8 366Other: 0 367Combined: 0 368Current hardware settings: 369RX: 1 370TX: 8 371Other: 0 372Combined: 0 373 3743) 375// TX queues must be rated starting from 0, so set bws for tx0 and tx1 for Eth0 376// and for tx2 and tx3 for Eth1. That is, rates 40 and 20 Mb/s appropriately 377// for Eth0 and 30 and 10 Mb/s for Eth1. 378// Real speed can differ a bit due to discreetness 379// Leave last 4 tx queues as not rated 380$ echo 40 > /sys/class/net/eth0/queues/tx-0/tx_maxrate 381$ echo 20 > /sys/class/net/eth0/queues/tx-1/tx_maxrate 382$ echo 30 > /sys/class/net/eth1/queues/tx-2/tx_maxrate 383$ echo 10 > /sys/class/net/eth1/queues/tx-3/tx_maxrate 384 3854) 386// Check maximum rate of tx (cpdma) queues: 387$ cat /sys/class/net/eth0/queues/tx-*/tx_maxrate 38840 38920 39030 39110 3920 3930 3940 3950 396 3975) 398// Map skb->priority to traffic class for Eth0: 399// 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2 400// Map traffic class to transmit queue: 401// tc0 -> txq0, tc1 -> txq1, tc2 -> (txq4, txq5) 402$ tc qdisc replace dev eth0 handle 100: parent root mqprio num_tc 3 \ 403map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@0 1@1 2@4 hw 1 404 4056) 406// Check classes settings 407$ tc -g class show dev eth0 408+---(100:ffe2) mqprio 409| +---(100:5) mqprio 410| +---(100:6) mqprio 411| 412+---(100:ffe1) mqprio 413| +---(100:2) mqprio 414| 415+---(100:ffe0) mqprio 416 +---(100:1) mqprio 417 4187) 419// Set rate for class A - 41 Mbit (tc0, txq0) using CBS Qdisc for Eth0 420// here only idle slope is important, others ignored 421// Real speed can differ a bit due to discreetness 422$ tc qdisc add dev eth0 parent 100:1 cbs locredit -1470 \ 423hicredit 62 sendslope -959000 idleslope 41000 offload 1 424net eth0: set FIFO3 bw = 50 425 4268) 427// Set rate for class B - 21 Mbit (tc1, txq1) using CBS Qdisc for Eth0 428$ tc qdisc add dev eth0 parent 100:2 cbs locredit -1470 \ 429hicredit 65 sendslope -979000 idleslope 21000 offload 1 430net eth0: set FIFO2 bw = 30 431 4329) 433// Create vlan 100 to map sk->priority to vlan qos for Eth0 434$ ip link add link eth0 name eth0.100 type vlan id 100 435net eth0: Adding vlanid 100 to vlan filter 436 43710) 438// Map skb->priority to L2 prio for Eth0.100, one to one 439$ ip link set eth0.100 type vlan \ 440egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 441 44211) 443// Check egress map for vlan 100 444$ cat /proc/net/vlan/eth0.100 445[...] 446INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0 447EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 448 44912) 450// Map skb->priority to traffic class for Eth1: 451// 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2 452// Map traffic class to transmit queue: 453// tc0 -> txq2, tc1 -> txq3, tc2 -> (txq6, txq7) 454$ tc qdisc replace dev eth1 handle 100: parent root mqprio num_tc 3 \ 455map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@2 1@3 2@6 hw 1 456 45713) 458// Check classes settings 459$ tc -g class show dev eth1 460+---(100:ffe2) mqprio 461| +---(100:7) mqprio 462| +---(100:8) mqprio 463| 464+---(100:ffe1) mqprio 465| +---(100:4) mqprio 466| 467+---(100:ffe0) mqprio 468 +---(100:3) mqprio 469 47014) 471// Set rate for class A - 31 Mbit (tc0, txq2) using CBS Qdisc for Eth1 472// here only idle slope is important, others ignored, but calculated 473// for interface speed - 100Mb for eth1 port. 474// Set it +1 Mb for reserve (important!) 475$ tc qdisc add dev eth1 parent 100:3 cbs locredit -1035 \ 476hicredit 465 sendslope -69000 idleslope 31000 offload 1 477net eth1: set FIFO3 bw = 31 478 47915) 480// Set rate for class B - 11 Mbit (tc1, txq3) using CBS Qdisc for Eth1 481// Set it +1 Mb for reserve (important!) 482$ tc qdisc add dev eth1 parent 100:4 cbs locredit -1335 \ 483hicredit 405 sendslope -89000 idleslope 11000 offload 1 484net eth1: set FIFO2 bw = 11 485 48616) 487// Create vlan 100 to map sk->priority to vlan qos for Eth1 488$ ip link add link eth1 name eth1.100 type vlan id 100 489net eth1: Adding vlanid 100 to vlan filter 490 49117) 492// Map skb->priority to L2 prio for Eth1.100, one to one 493$ ip link set eth1.100 type vlan \ 494egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 495 49618) 497// Check egress map for vlan 100 498$ cat /proc/net/vlan/eth1.100 499[...] 500INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0 501EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 502 50319) 504// Run appropriate tools with socket option "SO_PRIORITY" to 3 505// for class A and to 2 for class B. For both interfaces 506./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p2 -s 1500& 507./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p3 -s 1500& 508./tsn_talker -d 20:cf:30:85:7d:fd -i eth1.100 -p2 -s 1500& 509./tsn_talker -d 20:cf:30:85:7d:fd -i eth1.100 -p3 -s 1500& 510 51120) 512// run your listener on workstation (should be in same vlan) 513// (I took at https://www.spinics.net/lists/netdev/msg460869.html) 514./tsn_listener -d 18:03:73:66:87:42 -i enp5s0 -s 1500 515Receiving data rate: 39012 kbps 516Receiving data rate: 39012 kbps 517Receiving data rate: 39012 kbps 518Receiving data rate: 39012 kbps 519Receiving data rate: 39012 kbps 520Receiving data rate: 39012 kbps 521Receiving data rate: 39012 kbps 522Receiving data rate: 39012 kbps 523Receiving data rate: 39012 kbps 524Receiving data rate: 39012 kbps 525Receiving data rate: 39012 kbps 526Receiving data rate: 39012 kbps 527Receiving data rate: 39000 kbps 528 52921) 530// Restore default configuration if needed 531$ ip link del eth1.100 532$ ip link del eth0.100 533$ tc qdisc del dev eth1 root 534net eth1: Prev FIFO2 is shaped 535net eth1: set FIFO3 bw = 0 536net eth1: set FIFO2 bw = 0 537$ tc qdisc del dev eth0 root 538net eth0: Prev FIFO2 is shaped 539net eth0: set FIFO3 bw = 0 540net eth0: set FIFO2 bw = 0 541$ ethtool -L eth0 rx 1 tx 1 542