1Copyright (C) 2015 Freescale Semiconductor Inc.
2
3DPAA2 (Data Path Acceleration Architecture Gen2)
4------------------------------------------------
5
6This document provides an overview of the Freescale DPAA2 architecture
7and how it is integrated into the Linux kernel.
8
9Contents summary
10 -DPAA2 overview
11 -Overview of DPAA2 objects
12 -DPAA2 Linux driver architecture overview
13 -bus driver
14 -DPRC driver
15 -allocator
16 -DPIO driver
17 -Ethernet
18 -MAC
19
20DPAA2 Overview
21--------------
22
23DPAA2 is a hardware architecture designed for high-speeed network
24packet processing. DPAA2 consists of sophisticated mechanisms for
25processing Ethernet packets, queue management, buffer management,
26autonomous L2 switching, virtual Ethernet bridging, and accelerator
27(e.g. crypto) sharing.
28
29A DPAA2 hardware component called the Management Complex (or MC) manages the
30DPAA2 hardware resources. The MC provides an object-based abstraction for
31software drivers to use the DPAA2 hardware.
32
33The MC uses DPAA2 hardware resources such as queues, buffer pools, and
34network ports to create functional objects/devices such as network
35interfaces, an L2 switch, or accelerator instances.
36
37The MC provides memory-mapped I/O command interfaces (MC portals)
38which DPAA2 software drivers use to operate on DPAA2 objects:
39
40The diagram below shows an overview of the DPAA2 resource management
41architecture:
42
43 +--------------------------------------+
44 | OS |
45 | DPAA2 drivers |
46 | | |
47 +-----------------------------|--------+
48 |
49 | (create,discover,connect
50 | config,use,destroy)
51 |
52 DPAA2 |
53 +------------------------| mc portal |-+
54 | | |
55 | +- - - - - - - - - - - - -V- - -+ |
56 | | | |
57 | | Management Complex (MC) | |
58 | | | |
59 | +- - - - - - - - - - - - - - - -+ |
60 | |
61 | Hardware Hardware |
62 | Resources Objects |
63 | --------- ------- |
64 | -queues -DPRC |
65 | -buffer pools -DPMCP |
66 | -Eth MACs/ports -DPIO |
67 | -network interface -DPNI |
68 | profiles -DPMAC |
69 | -queue portals -DPBP |
70 | -MC portals ... |
71 | ... |
72 | |
73 +--------------------------------------+
74
75The MC mediates operations such as create, discover,
76connect, configuration, and destroy. Fast-path operations
77on data, such as packet transmit/receive, are not mediated by
78the MC and are done directly using memory mapped regions in
79DPIO objects.
80
81Overview of DPAA2 Objects
82-------------------------
83The section provides a brief overview of some key DPAA2 objects.
84A simple scenario is described illustrating the objects involved
85in creating a network interfaces.
86
87-DPRC (Datapath Resource Container)
88
89 A DPRC is a container object that holds all the other
90 types of DPAA2 objects. In the example diagram below there
91 are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC)
92 in the container.
93
94 +---------------------------------------------------------+
95 | DPRC |
96 | |
97 | +-------+ +-------+ +-------+ +-------+ +-------+ |
98 | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | |
99 | +-------+ +-------+ +-------+ +---+---+ +---+---+ |
100 | | DPMCP | | DPIO | |
101 | +-------+ +-------+ |
102 | | DPMCP | |
103 | +-------+ |
104 | |
105 +---------------------------------------------------------+
106
107 From the point of view of an OS, a DPRC behaves similar to a plug and
108 play bus, like PCI. DPRC commands can be used to enumerate the contents
109 of the DPRC, discover the hardware objects present (including mappable
110 regions and interrupts).
111
112 DPRC.1 (bus)
113 |
114 +--+--------+-------+-------+-------+
115 | | | | |
116 DPMCP.1 DPIO.1 DPBP.1 DPNI.1 DPMAC.1
117 DPMCP.2 DPIO.2
118 DPMCP.3
119
120 Hardware objects can be created and destroyed dynamically, providing
121 the ability to hot plug/unplug objects in and out of the DPRC.
122
123 A DPRC has a mappable MMIO region (an MC portal) that can be used
124 to send MC commands. It has an interrupt for status events (like
125 hotplug).
126
127 All objects in a container share the same hardware "isolation context".
128 This means that with respect to an IOMMU the isolation granularity
129 is at the DPRC (container) level, not at the individual object
130 level.
131
132 DPRCs can be defined statically and populated with objects
133 via a config file passed to the MC when firmware starts
134 it. There is also a Linux user space tool called "restool"
135 that can be used to create/destroy containers and objects
136 dynamically.
137
138-DPAA2 Objects for an Ethernet Network Interface
139
140 A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX
141 queuing mechanisms, configuration mechanisms, buffer management,
142 physical ports, and interrupts. DPAA2 uses a more granular approach
143 utilizing multiple hardware objects. Each object provides specialized
144 functions. Groups of these objects are used by software to provide
145 Ethernet network interface functionality. This approach provides
146 efficient use of finite hardware resources, flexibility, and
147 performance advantages.
148
149 The diagram below shows the objects needed for a simple
150 network interface configuration on a system with 2 CPUs.
151
152 +---+---+ +---+---+
153 CPU0 CPU1
154 +---+---+ +---+---+
155 | |
156 +---+---+ +---+---+
157 DPIO DPIO
158 +---+---+ +---+---+
159 \ /
160 \ /
161 \ /
162 +---+---+
163 DPNI --- DPBP,DPMCP
164 +---+---+
165 |
166 |
167 +---+---+
168 DPMAC
169 +---+---+
170 |
171 port/PHY
172
173 Below the objects are described. For each object a brief description
174 is provided along with a summary of the kinds of operations the object
175 supports and a summary of key resources of the object (MMIO regions
176 and IRQs).
177
178 -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a
179 hardware device that connects to an Ethernet PHY and allows
180 physical transmission and reception of Ethernet frames.
181 -MMIO regions: none
182 -IRQs: DPNI link change
183 -commands: set link up/down, link config, get stats,
184 IRQ config, enable, reset
185
186 -DPNI (Datapath Network Interface): contains TX/RX queues,
187 network interface configuration, and RX buffer pool configuration
188 mechanisms. The TX/RX queues are in memory and are identified by
189 queue number.
190 -MMIO regions: none
191 -IRQs: link state
192 -commands: port config, offload config, queue config,
193 parse/classify config, IRQ config, enable, reset
194
195 -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue
196 packets and do hardware buffer pool management operations. The DPAA2
197 architecture separates the mechanism to access queues (the DPIO object)
198 from the queues themselves. The DPIO provides an MMIO interface to
199 enqueue/dequeue packets. To enqueue something a descriptor is written
200 to the DPIO MMIO region, which includes the target queue number.
201 There will typically be one DPIO assigned to each CPU. This allows all
202 CPUs to simultaneously perform enqueue/dequeued operations. DPIOs are
203 expected to be shared by different DPAA2 drivers.
204 -MMIO regions: queue operations, buffer management
205 -IRQs: data availability, congestion notification, buffer
206 pool depletion
207 -commands: IRQ config, enable, reset
208
209 -DPBP (Datapath Buffer Pool): represents a hardware buffer
210 pool.
211 -MMIO regions: none
212 -IRQs: none
213 -commands: enable, reset
214
215 -DPMCP (Datapath MC Portal): provides an MC command portal.
216 Used by drivers to send commands to the MC to manage
217 objects.
218 -MMIO regions: MC command portal
219 -IRQs: command completion
220 -commands: IRQ config, enable, reset
221
222 Object Connections
223 ------------------
224 Some objects have explicit relationships that must
225 be configured:
226
227 -DPNI <--> DPMAC
228 -DPNI <--> DPNI
229 -DPNI <--> L2-switch-port
230 A DPNI must be connected to something such as a DPMAC,
231 another DPNI, or L2 switch port. The DPNI connection
232 is made via a DPRC command.
233
234 +-------+ +-------+
235 | DPNI | | DPMAC |
236 +---+---+ +---+---+
237 | |
238 +==========+
239
240 -DPNI <--> DPBP
241 A network interface requires a 'buffer pool' (DPBP
242 object) which provides a list of pointers to memory
243 where received Ethernet data is to be copied. The
244 Ethernet driver configures the DPBPs associated with
245 the network interface.
246
247 Interrupts
248 ----------
249 All interrupts generated by DPAA2 objects are message
250 interrupts. At the hardware level message interrupts
251 generated by devices will normally have 3 components--
252 1) a non-spoofable 'device-id' expressed on the hardware
253 bus, 2) an address, 3) a data value.
254
255 In the case of DPAA2 devices/objects, all objects in the
256 same container/DPRC share the same 'device-id'.
257 For ARM-based SoC this is the same as the stream ID.
258
259
260DPAA2 Linux Driver Overview
261---------------------------
262
263This section provides an overview of the Linux kernel drivers for
264DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure"
265drivers and 2) functional object drivers (such as Ethernet).
266
267As described previously, a DPRC is a container that holds the other
268types of DPAA2 objects. It is functionally similar to a plug-and-play
269bus controller.
270
271Each object in the DPRC is a Linux "device" and is bound to a driver.
272The diagram below shows the Linux drivers involved in a networking
273scenario and the objects bound to each driver. A brief description
274of each driver follows.
275
276 +------------+
277 | OS Network |
278 | Stack |
279 +------------+ +------------+
280 | Allocator |. . . . . . . | Ethernet |
281 |(DPMCP,DPBP)| | (DPNI) |
282 +-.----------+ +---+---+----+
283 . . ^ |
284 . . <data avail, | |<enqueue,
285 . . tx confirm> | | dequeue>
286 +-------------+ . | |
287 | DPRC driver | . +---+---V----+ +---------+
288 | (DPRC) | . . . . . .| DPIO driver| | MAC |
289 +----------+--+ | (DPIO) | | (DPMAC) |
290 | +------+-----+ +-----+---+
291 |<dev add/remove> | |
292 | | |
293 +----+--------------+ | +--+---+
294 | MC-bus driver | | | PHY |
295 | | | |driver|
296 | /soc/fsl-mc | | +--+---+
297 +-------------------+ | |
298 | |
299 ================================ HARDWARE =========|=================|======
300 DPIO |
301 | |
302 DPNI---DPBP |
303 | |
304 DPMAC |
305 | |
306 PHY ---------------+
307 ===================================================|========================
308
309A brief description of each driver is provided below.
310
311 MC-bus driver
312 -------------
313 The MC-bus driver is a platform driver and is probed from a
314 node in the device tree (compatible "fsl,qoriq-mc") passed in by boot
315 firmware. It is responsible for bootstrapping the DPAA2 kernel
316 infrastructure.
317 Key functions include:
318 -registering a new bus type named "fsl-mc" with the kernel,
319 and implementing bus call-backs (e.g. match/uevent/dev_groups)
320 -implementing APIs for DPAA2 driver registration and for device
321 add/remove
322 -creates an MSI IRQ domain
323 -doing a 'device add' to expose the 'root' DPRC, in turn triggering
324 a bind of the root DPRC to the DPRC driver
325
326 DPRC driver
327 -----------
328 The DPRC driver is bound to DPRC objects and does runtime management
329 of a bus instance. It performs the initial bus scan of the DPRC
330 and handles interrupts for container events such as hot plug by
331 re-scanning the DPRC.
332
333 Allocator
334 ----------
335 Certain objects such as DPMCP and DPBP are generic and fungible,
336 and are intended to be used by other drivers. For example,
337 the DPAA2 Ethernet driver needs:
338 -DPMCPs to send MC commands, to configure network interfaces
339 -DPBPs for network buffer pools
340
341 The allocator driver registers for these allocatable object types
342 and those objects are bound to the allocator when the bus is probed.
343 The allocator maintains a pool of objects that are available for
344 allocation by other DPAA2 drivers.
345
346 DPIO driver
347 -----------
348 The DPIO driver is bound to DPIO objects and provides services that allow
349 other drivers such as the Ethernet driver to enqueue and dequeue data for
350 their respective objects.
351 Key services include:
352 -data availability notifications
353 -hardware queuing operations (enqueue and dequeue of data)
354 -hardware buffer pool management
355
356 To transmit a packet the Ethernet driver puts data on a queue and
357 invokes a DPIO API. For receive, the Ethernet driver registers
358 a data availability notification callback. To dequeue a packet
359 a DPIO API is used.
360
361 There is typically one DPIO object per physical CPU for optimum
362 performance, allowing different CPUs to simultaneously enqueue
363 and dequeue data.
364
365 The DPIO driver operates on behalf of all DPAA2 drivers
366 active in the kernel-- Ethernet, crypto, compression,
367 etc.
368
369 Ethernet
370 --------
371 The Ethernet driver is bound to a DPNI and implements the kernel
372 interfaces needed to connect the DPAA2 network interface to
373 the network stack.
374
375 Each DPNI corresponds to a Linux network interface.
376
377 MAC driver
378 ----------
379 An Ethernet PHY is an off-chip, board specific component and is managed
380 by the appropriate PHY driver via an mdio bus. The MAC driver
381 plays a role of being a proxy between the PHY driver and the
382 MC. It does this proxy via the MC commands to a DPMAC object.
383 If the PHY driver signals a link change, the MAC driver notifies
384 the MC via a DPMAC command. If a network interface is brought
385 up or down, the MC notifies the DPMAC driver via an interrupt and
386 the driver can take appropriate action.
387