1Copyright (C) 2015 Freescale Semiconductor Inc. 2 3DPAA2 (Data Path Acceleration Architecture Gen2) 4------------------------------------------------ 5 6This document provides an overview of the Freescale DPAA2 architecture 7and how it is integrated into the Linux kernel. 8 9Contents summary 10 -DPAA2 overview 11 -Overview of DPAA2 objects 12 -DPAA2 Linux driver architecture overview 13 -bus driver 14 -dprc driver 15 -allocator 16 -dpio driver 17 -Ethernet 18 -mac 19 20DPAA2 Overview 21-------------- 22 23DPAA2 is a hardware architecture designed for high-speeed network 24packet processing. DPAA2 consists of sophisticated mechanisms for 25processing Ethernet packets, queue management, buffer management, 26autonomous L2 switching, virtual Ethernet bridging, and accelerator 27(e.g. crypto) sharing. 28 29A DPAA2 hardware component called the Management Complex (or MC) manages the 30DPAA2 hardware resources. The MC provides an object-based abstraction for 31software drivers to use the DPAA2 hardware. 32 33The MC uses DPAA2 hardware resources such as queues, buffer pools, and 34network ports to create functional objects/devices such as network 35interfaces, an L2 switch, or accelerator instances. 36 37The MC provides memory-mapped I/O command interfaces (MC portals) 38which DPAA2 software drivers use to operate on DPAA2 objects: 39 40 +--------------------------------------+ 41 | OS | 42 | DPAA2 drivers | 43 | | | 44 +-----------------------------|--------+ 45 | 46 | (create,discover,connect 47 | config,use,destroy) 48 | 49 DPAA2 | 50 +------------------------| mc portal |-+ 51 | | | 52 | +- - - - - - - - - - - - -V- - -+ | 53 | | | | 54 | | Management Complex (MC) | | 55 | | | | 56 | +- - - - - - - - - - - - - - - -+ | 57 | | 58 | Hardware Hardware | 59 | Resources Objects | 60 | --------- ------- | 61 | -queues -DPRC | 62 | -buffer pools -DPMCP | 63 | -Eth MACs/ports -DPIO | 64 | -network interface -DPNI | 65 | profiles -DPMAC | 66 | -queue portals -DPBP | 67 | -MC portals ... | 68 | ... | 69 | | 70 +--------------------------------------+ 71 72The MC mediates operations such as create, discover, 73connect, configuration, and destroy. Fast-path operations 74on data, such as packet transmit/receive, are not mediated by 75the MC and are done directly using memory mapped regions in 76DPIO objects. 77 78Overview of DPAA2 Objects 79------------------------- 80The section provides a brief overview of some key objects 81in the DPAA2 hardware. A simple scenario is described illustrating 82the objects involved in creating a network interfaces. 83 84-DPRC (Datapath Resource Container) 85 86 A DPRC is an container object that holds all the other 87 types of DPAA2 objects. In the example diagram below there 88 are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) 89 in the container. 90 91 +---------------------------------------------------------+ 92 | DPRC | 93 | | 94 | +-------+ +-------+ +-------+ +-------+ +-------+ | 95 | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | 96 | +-------+ +-------+ +-------+ +---+---+ +---+---+ | 97 | | DPMCP | | DPIO | | 98 | +-------+ +-------+ | 99 | | DPMCP | | 100 | +-------+ | 101 | | 102 +---------------------------------------------------------+ 103 104 From the point of view of an OS, a DPRC is bus-like. Like 105 a plug-and-play bus, such as PCI, DPRC commands can be used to 106 enumerate the contents of the DPRC, discover the hardware 107 objects present (including mappable regions and interrupts). 108 109 dprc.1 (bus) 110 | 111 +--+--------+-------+-------+-------+ 112 | | | | | 113 dpmcp.1 dpio.1 dpbp.1 dpni.1 dpmac.1 114 dpmcp.2 dpio.2 115 dpmcp.3 116 117 Hardware objects can be created and destroyed dynamically, providing 118 the ability to hot plug/unplug objects in and out of the DPRC. 119 120 A DPRC has a mappable mmio region (an MC portal) that can be used 121 to send MC commands. It has an interrupt for status events (like 122 hotplug). 123 124 All objects in a container share the same hardware "isolation context". 125 This means that with respect to an IOMMU the isolation granularity 126 is at the DPRC (container) level, not at the individual object 127 level. 128 129 DPRCs can be defined statically and populated with objects 130 via a config file passed to the MC when firmware starts 131 it. There is also a Linux user space tool called "restool" 132 that can be used to create/destroy containers and objects 133 dynamically. 134 135-DPAA2 Objects for an Ethernet Network Interface 136 137 A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX 138 queuing mechanisms, configuration mechanisms, buffer management, 139 physical ports, and interrupts. DPAA2 uses a more granular approach 140 utilizing multiple hardware objects. Each object has specialized 141 functions, and are used together by software to provide Ethernet network 142 interface functionality. This approach provides efficient use of finite 143 hardware resources, flexibility, and performance advantages. 144 145 The diagram below shows the objects needed for a simple 146 network interface configuration on a system with 2 CPUs. 147 148 +---+---+ +---+---+ 149 CPU0 CPU1 150 +---+---+ +---+---+ 151 | | 152 +---+---+ +---+---+ 153 DPIO DPIO 154 +---+---+ +---+---+ 155 \ / 156 \ / 157 \ / 158 +---+---+ 159 DPNI --- DPBP,DPMCP 160 +---+---+ 161 | 162 | 163 +---+---+ 164 DPMAC 165 +---+---+ 166 | 167 port/PHY 168 169 Below the objects are described. For each object a brief description 170 is provided along with a summary of the kinds of operations the object 171 supports and a summary of key resources of the object (mmio regions 172 and irqs). 173 174 -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a 175 hardware device that connects to an Ethernet PHY and allows 176 physical transmission and reception of Ethernet frames. 177 -mmio regions: none 178 -irqs: dpni link change 179 -commands: set link up/down, link config, get stats, 180 irq config, enable, reset 181 182 -DPNI (Datapath Network Interface): contains TX/RX queues, 183 network interface configuration, and rx buffer pool configuration 184 mechanisms. 185 -mmio regions: none 186 -irqs: link state 187 -commands: port config, offload config, queue config, 188 parse/classify config, irq config, enable, reset 189 190 -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue 191 packets and do hardware buffer pool management operations. For 192 optimum performance there is typically DPIO per CPU. This allows 193 each CPU to perform simultaneous enqueue/dequeue operations. 194 -mmio regions: queue operations, buffer mgmt 195 -irqs: data availability, congestion notification, buffer 196 pool depletion 197 -commands: irq config, enable, reset 198 199 -DPBP (Datapath Buffer Pool): represents a hardware buffer 200 pool. 201 -mmio regions: none 202 -irqs: none 203 -commands: enable, reset 204 205 -DPMCP (Datapath MC Portal): provides an MC command portal. 206 Used by drivers to send commands to the MC to manage 207 objects. 208 -mmio regions: MC command portal 209 -irqs: command completion 210 -commands: irq config, enable, reset 211 212 Object Connections 213 ------------------ 214 Some objects have explicit relationships that must 215 be configured: 216 217 -DPNI <--> DPMAC 218 -DPNI <--> DPNI 219 -DPNI <--> L2-switch-port 220 A DPNI must be connected to something such as a DPMAC, 221 another DPNI, or L2 switch port. The DPNI connection 222 is made via a DPRC command. 223 224 +-------+ +-------+ 225 | DPNI | | DPMAC | 226 +---+---+ +---+---+ 227 | | 228 +==========+ 229 230 -DPNI <--> DPBP 231 A network interface requires a 'buffer pool' (DPBP 232 object) which provides a list of pointers to memory 233 where received Ethernet data is to be copied. The 234 Ethernet driver configures the DPBPs associated with 235 the network interface. 236 237 Interrupts 238 ---------- 239 All interrupts generated by DPAA2 objects are message 240 interrupts. At the hardware level message interrupts 241 generated by devices will normally have 3 components-- 242 1) a non-spoofable 'device-id' expressed on the hardware 243 bus, 2) an address, 3) a data value. 244 245 In the case of DPAA2 devices/objects, all objects in the 246 same container/DPRC share the same 'device-id'. 247 For ARM-based SoC this is the same as the stream ID. 248 249 250DPAA2 Linux Driver Overview 251--------------------------- 252 253This section provides an overview of the Linux kernel drivers for 254DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" 255drivers and 2) functional object drivers (such as Ethernet). 256 257As described previously, a DPRC is a container that holds the other 258types of DPAA2 objects. It is functionally similar to a plug-and-play 259bus controller. 260 261Each object in the DPRC is a Linux "device" and is bound to a driver. 262The diagram below shows the Linux drivers involved in a networking 263scenario and the objects bound to each driver. A brief description 264of each driver follows. 265 266 +------------+ 267 | OS Network | 268 | Stack | 269 +------------+ +------------+ 270 | Allocator |. . . . . . . | Ethernet | 271 |(dpmcp,dpbp)| | (dpni) | 272 +-.----------+ +---+---+----+ 273 . . ^ | 274 . . <data avail, | |<enqueue, 275 . . tx confirm> | | dequeue> 276 +-------------+ . | | 277 | DPRC driver | . +---+---V----+ +---------+ 278 | (dprc) | . . . . . .| DPIO driver| | MAC | 279 +----------+--+ | (dpio) | | (dpmac) | 280 | +------+-----+ +-----+---+ 281 |<dev add/remove> | | 282 | | | 283 +----+--------------+ | +--+---+ 284 | mc-bus driver | | | PHY | 285 | | | |driver| 286 | /fsl-mc@80c000000 | | +--+---+ 287 +-------------------+ | | 288 | | 289 ================================ HARDWARE =========|=================|====== 290 DPIO | 291 | | 292 DPNI---DPBP | 293 | | 294 DPMAC | 295 | | 296 PHY ---------------+ 297 ===================================================|======================== 298 299A brief description of each driver is provided below. 300 301 mc-bus driver 302 ------------- 303 The mc-bus driver is a platform driver and is probed from an 304 "/fsl-mc@xxxx" node in the device tree passed in by boot firmware. 305 It is responsible for bootstrapping the DPAA2 kernel infrastructure. 306 Key functions include: 307 -registering a new bus type named "fsl-mc" with the kernel, 308 and implementing bus call-backs (e.g. match/uevent/dev_groups) 309 -implemeting APIs for DPAA2 driver registration and for device 310 add/remove 311 -creates an MSI irq domain 312 -do a device add of the 'root' DPRC device, which is needed 313 to bootstrap things 314 315 DPRC driver 316 ----------- 317 The dprc-driver is bound DPRC objects and does runtime management 318 of a bus instance. It performs the initial bus scan of the DPRC 319 and handles interrupts for container events such as hot plug. 320 321 Allocator 322 ---------- 323 Certain objects such as DPMCP and DPBP are generic and fungible, 324 and are intended to be used by other drivers. For example, 325 the DPAA2 Ethernet driver needs: 326 -DPMCPs to send MC commands, to configure network interfaces 327 -DPBPs for network buffer pools 328 329 The allocator driver registers for these allocatable object types 330 and those objects are bound to the allocator when the bus is probed. 331 The allocator maintains a pool of objects that are available for 332 allocation by other DPAA2 drivers. 333 334 DPIO driver 335 ----------- 336 The DPIO driver is bound to DPIO objects and provides services that allow 337 other drivers such as the Ethernet driver to receive and transmit data. 338 Key services include: 339 -data availability notifications 340 -hardware queuing operations (enqueue and dequeue of data) 341 -hardware buffer pool management 342 343 There is typically one DPIO object per physical CPU for optimum 344 performance, allowing each CPU to simultaneously enqueue 345 and dequeue data. 346 347 The DPIO driver operates on behalf of all DPAA2 drivers 348 active in the kernel-- Ethernet, crypto, compression, 349 etc. 350 351 Ethernet 352 -------- 353 The Ethernet driver is bound to a DPNI and implements the kernel 354 interfaces needed to connect the DPAA2 network interface to 355 the network stack. 356 357 Each DPNI corresponds to a Linux network interface. 358 359 MAC driver 360 ---------- 361 An Ethernet PHY is an off-chip, board specific component and is managed 362 by the appropriate PHY driver via an mdio bus. The MAC driver 363 plays a role of being a proxy between the PHY driver and the 364 MC. It does this proxy via the MC commands to a DPMAC object. 365