1Copyright (C) 2015 Freescale Semiconductor Inc. 2 3DPAA2 (Data Path Acceleration Architecture Gen2) 4------------------------------------------------ 5 6This document provides an overview of the Freescale DPAA2 architecture 7and how it is integrated into the Linux kernel. 8 9Contents summary 10 -DPAA2 overview 11 -Overview of DPAA2 objects 12 -DPAA2 Linux driver architecture overview 13 -bus driver 14 -DPRC driver 15 -allocator 16 -DPIO driver 17 -Ethernet 18 -MAC 19 20DPAA2 Overview 21-------------- 22 23DPAA2 is a hardware architecture designed for high-speeed network 24packet processing. DPAA2 consists of sophisticated mechanisms for 25processing Ethernet packets, queue management, buffer management, 26autonomous L2 switching, virtual Ethernet bridging, and accelerator 27(e.g. crypto) sharing. 28 29A DPAA2 hardware component called the Management Complex (or MC) manages the 30DPAA2 hardware resources. The MC provides an object-based abstraction for 31software drivers to use the DPAA2 hardware. 32 33The MC uses DPAA2 hardware resources such as queues, buffer pools, and 34network ports to create functional objects/devices such as network 35interfaces, an L2 switch, or accelerator instances. 36 37The MC provides memory-mapped I/O command interfaces (MC portals) 38which DPAA2 software drivers use to operate on DPAA2 objects: 39 40The diagram below shows an overview of the DPAA2 resource management 41architecture: 42 43 +--------------------------------------+ 44 | OS | 45 | DPAA2 drivers | 46 | | | 47 +-----------------------------|--------+ 48 | 49 | (create,discover,connect 50 | config,use,destroy) 51 | 52 DPAA2 | 53 +------------------------| mc portal |-+ 54 | | | 55 | +- - - - - - - - - - - - -V- - -+ | 56 | | | | 57 | | Management Complex (MC) | | 58 | | | | 59 | +- - - - - - - - - - - - - - - -+ | 60 | | 61 | Hardware Hardware | 62 | Resources Objects | 63 | --------- ------- | 64 | -queues -DPRC | 65 | -buffer pools -DPMCP | 66 | -Eth MACs/ports -DPIO | 67 | -network interface -DPNI | 68 | profiles -DPMAC | 69 | -queue portals -DPBP | 70 | -MC portals ... | 71 | ... | 72 | | 73 +--------------------------------------+ 74 75The MC mediates operations such as create, discover, 76connect, configuration, and destroy. Fast-path operations 77on data, such as packet transmit/receive, are not mediated by 78the MC and are done directly using memory mapped regions in 79DPIO objects. 80 81Overview of DPAA2 Objects 82------------------------- 83The section provides a brief overview of some key DPAA2 objects. 84A simple scenario is described illustrating the objects involved 85in creating a network interfaces. 86 87-DPRC (Datapath Resource Container) 88 89 A DPRC is a container object that holds all the other 90 types of DPAA2 objects. In the example diagram below there 91 are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) 92 in the container. 93 94 +---------------------------------------------------------+ 95 | DPRC | 96 | | 97 | +-------+ +-------+ +-------+ +-------+ +-------+ | 98 | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | 99 | +-------+ +-------+ +-------+ +---+---+ +---+---+ | 100 | | DPMCP | | DPIO | | 101 | +-------+ +-------+ | 102 | | DPMCP | | 103 | +-------+ | 104 | | 105 +---------------------------------------------------------+ 106 107 From the point of view of an OS, a DPRC behaves similar to a plug and 108 play bus, like PCI. DPRC commands can be used to enumerate the contents 109 of the DPRC, discover the hardware objects present (including mappable 110 regions and interrupts). 111 112 DPRC.1 (bus) 113 | 114 +--+--------+-------+-------+-------+ 115 | | | | | 116 DPMCP.1 DPIO.1 DPBP.1 DPNI.1 DPMAC.1 117 DPMCP.2 DPIO.2 118 DPMCP.3 119 120 Hardware objects can be created and destroyed dynamically, providing 121 the ability to hot plug/unplug objects in and out of the DPRC. 122 123 A DPRC has a mappable MMIO region (an MC portal) that can be used 124 to send MC commands. It has an interrupt for status events (like 125 hotplug). 126 127 All objects in a container share the same hardware "isolation context". 128 This means that with respect to an IOMMU the isolation granularity 129 is at the DPRC (container) level, not at the individual object 130 level. 131 132 DPRCs can be defined statically and populated with objects 133 via a config file passed to the MC when firmware starts 134 it. 135 136-DPAA2 Objects for an Ethernet Network Interface 137 138 A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX 139 queuing mechanisms, configuration mechanisms, buffer management, 140 physical ports, and interrupts. DPAA2 uses a more granular approach 141 utilizing multiple hardware objects. Each object provides specialized 142 functions. Groups of these objects are used by software to provide 143 Ethernet network interface functionality. This approach provides 144 efficient use of finite hardware resources, flexibility, and 145 performance advantages. 146 147 The diagram below shows the objects needed for a simple 148 network interface configuration on a system with 2 CPUs. 149 150 +---+---+ +---+---+ 151 CPU0 CPU1 152 +---+---+ +---+---+ 153 | | 154 +---+---+ +---+---+ 155 DPIO DPIO 156 +---+---+ +---+---+ 157 \ / 158 \ / 159 \ / 160 +---+---+ 161 DPNI --- DPBP,DPMCP 162 +---+---+ 163 | 164 | 165 +---+---+ 166 DPMAC 167 +---+---+ 168 | 169 port/PHY 170 171 Below the objects are described. For each object a brief description 172 is provided along with a summary of the kinds of operations the object 173 supports and a summary of key resources of the object (MMIO regions 174 and IRQs). 175 176 -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a 177 hardware device that connects to an Ethernet PHY and allows 178 physical transmission and reception of Ethernet frames. 179 -MMIO regions: none 180 -IRQs: DPNI link change 181 -commands: set link up/down, link config, get stats, 182 IRQ config, enable, reset 183 184 -DPNI (Datapath Network Interface): contains TX/RX queues, 185 network interface configuration, and RX buffer pool configuration 186 mechanisms. The TX/RX queues are in memory and are identified by 187 queue number. 188 -MMIO regions: none 189 -IRQs: link state 190 -commands: port config, offload config, queue config, 191 parse/classify config, IRQ config, enable, reset 192 193 -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue 194 packets and do hardware buffer pool management operations. The DPAA2 195 architecture separates the mechanism to access queues (the DPIO object) 196 from the queues themselves. The DPIO provides an MMIO interface to 197 enqueue/dequeue packets. To enqueue something a descriptor is written 198 to the DPIO MMIO region, which includes the target queue number. 199 There will typically be one DPIO assigned to each CPU. This allows all 200 CPUs to simultaneously perform enqueue/dequeued operations. DPIOs are 201 expected to be shared by different DPAA2 drivers. 202 -MMIO regions: queue operations, buffer management 203 -IRQs: data availability, congestion notification, buffer 204 pool depletion 205 -commands: IRQ config, enable, reset 206 207 -DPBP (Datapath Buffer Pool): represents a hardware buffer 208 pool. 209 -MMIO regions: none 210 -IRQs: none 211 -commands: enable, reset 212 213 -DPMCP (Datapath MC Portal): provides an MC command portal. 214 Used by drivers to send commands to the MC to manage 215 objects. 216 -MMIO regions: MC command portal 217 -IRQs: command completion 218 -commands: IRQ config, enable, reset 219 220 Object Connections 221 ------------------ 222 Some objects have explicit relationships that must 223 be configured: 224 225 -DPNI <--> DPMAC 226 -DPNI <--> DPNI 227 -DPNI <--> L2-switch-port 228 A DPNI must be connected to something such as a DPMAC, 229 another DPNI, or L2 switch port. The DPNI connection 230 is made via a DPRC command. 231 232 +-------+ +-------+ 233 | DPNI | | DPMAC | 234 +---+---+ +---+---+ 235 | | 236 +==========+ 237 238 -DPNI <--> DPBP 239 A network interface requires a 'buffer pool' (DPBP 240 object) which provides a list of pointers to memory 241 where received Ethernet data is to be copied. The 242 Ethernet driver configures the DPBPs associated with 243 the network interface. 244 245 Interrupts 246 ---------- 247 All interrupts generated by DPAA2 objects are message 248 interrupts. At the hardware level message interrupts 249 generated by devices will normally have 3 components-- 250 1) a non-spoofable 'device-id' expressed on the hardware 251 bus, 2) an address, 3) a data value. 252 253 In the case of DPAA2 devices/objects, all objects in the 254 same container/DPRC share the same 'device-id'. 255 For ARM-based SoC this is the same as the stream ID. 256 257 258DPAA2 Linux Driver Overview 259--------------------------- 260 261This section provides an overview of the Linux kernel drivers for 262DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" 263drivers and 2) functional object drivers (such as Ethernet). 264 265As described previously, a DPRC is a container that holds the other 266types of DPAA2 objects. It is functionally similar to a plug-and-play 267bus controller. 268 269Each object in the DPRC is a Linux "device" and is bound to a driver. 270The diagram below shows the Linux drivers involved in a networking 271scenario and the objects bound to each driver. A brief description 272of each driver follows. 273 274 +------------+ 275 | OS Network | 276 | Stack | 277 +------------+ +------------+ 278 | Allocator |. . . . . . . | Ethernet | 279 |(DPMCP,DPBP)| | (DPNI) | 280 +-.----------+ +---+---+----+ 281 . . ^ | 282 . . <data avail, | |<enqueue, 283 . . tx confirm> | | dequeue> 284 +-------------+ . | | 285 | DPRC driver | . +---+---V----+ +---------+ 286 | (DPRC) | . . . . . .| DPIO driver| | MAC | 287 +----------+--+ | (DPIO) | | (DPMAC) | 288 | +------+-----+ +-----+---+ 289 |<dev add/remove> | | 290 | | | 291 +----+--------------+ | +--+---+ 292 | MC-bus driver | | | PHY | 293 | | | |driver| 294 | /soc/fsl-mc | | +--+---+ 295 +-------------------+ | | 296 | | 297 ================================ HARDWARE =========|=================|====== 298 DPIO | 299 | | 300 DPNI---DPBP | 301 | | 302 DPMAC | 303 | | 304 PHY ---------------+ 305 ===================================================|======================== 306 307A brief description of each driver is provided below. 308 309 MC-bus driver 310 ------------- 311 The MC-bus driver is a platform driver and is probed from a 312 node in the device tree (compatible "fsl,qoriq-mc") passed in by boot 313 firmware. It is responsible for bootstrapping the DPAA2 kernel 314 infrastructure. 315 Key functions include: 316 -registering a new bus type named "fsl-mc" with the kernel, 317 and implementing bus call-backs (e.g. match/uevent/dev_groups) 318 -implementing APIs for DPAA2 driver registration and for device 319 add/remove 320 -creates an MSI IRQ domain 321 -doing a 'device add' to expose the 'root' DPRC, in turn triggering 322 a bind of the root DPRC to the DPRC driver 323 The binding for the MC-bus device-tree node can be consulted here: 324 Documentation/devicetree/bindings/misc/fsl,qoriq-mc.txt 325 326 DPRC driver 327 ----------- 328 The DPRC driver is bound to DPRC objects and does runtime management 329 of a bus instance. It performs the initial bus scan of the DPRC 330 and handles interrupts for container events such as hot plug by 331 re-scanning the DPRC. 332 333 Allocator 334 ---------- 335 Certain objects such as DPMCP and DPBP are generic and fungible, 336 and are intended to be used by other drivers. For example, 337 the DPAA2 Ethernet driver needs: 338 -DPMCPs to send MC commands, to configure network interfaces 339 -DPBPs for network buffer pools 340 341 The allocator driver registers for these allocatable object types 342 and those objects are bound to the allocator when the bus is probed. 343 The allocator maintains a pool of objects that are available for 344 allocation by other DPAA2 drivers. 345 346 DPIO driver 347 ----------- 348 The DPIO driver is bound to DPIO objects and provides services that allow 349 other drivers such as the Ethernet driver to enqueue and dequeue data for 350 their respective objects. 351 Key services include: 352 -data availability notifications 353 -hardware queuing operations (enqueue and dequeue of data) 354 -hardware buffer pool management 355 356 To transmit a packet the Ethernet driver puts data on a queue and 357 invokes a DPIO API. For receive, the Ethernet driver registers 358 a data availability notification callback. To dequeue a packet 359 a DPIO API is used. 360 361 There is typically one DPIO object per physical CPU for optimum 362 performance, allowing different CPUs to simultaneously enqueue 363 and dequeue data. 364 365 The DPIO driver operates on behalf of all DPAA2 drivers 366 active in the kernel-- Ethernet, crypto, compression, 367 etc. 368 369 Ethernet 370 -------- 371 The Ethernet driver is bound to a DPNI and implements the kernel 372 interfaces needed to connect the DPAA2 network interface to 373 the network stack. 374 375 Each DPNI corresponds to a Linux network interface. 376 377 MAC driver 378 ---------- 379 An Ethernet PHY is an off-chip, board specific component and is managed 380 by the appropriate PHY driver via an mdio bus. The MAC driver 381 plays a role of being a proxy between the PHY driver and the 382 MC. It does this proxy via the MC commands to a DPMAC object. 383 If the PHY driver signals a link change, the MAC driver notifies 384 the MC via a DPMAC command. If a network interface is brought 385 up or down, the MC notifies the DPMAC driver via an interrupt and 386 the driver can take appropriate action. 387