1Copyright (C) 2015 Freescale Semiconductor Inc.
2
3DPAA2 (Data Path Acceleration Architecture Gen2)
4------------------------------------------------
5
6This document provides an overview of the Freescale DPAA2 architecture
7and how it is integrated into the Linux kernel.
8
9Contents summary
10 -DPAA2 overview
11 -Overview of DPAA2 objects
12 -DPAA2 Linux driver architecture overview
13 -bus driver
14 -dprc driver
15 -allocator
16 -dpio driver
17 -Ethernet
18 -mac
19
20DPAA2 Overview
21--------------
22
23DPAA2 is a hardware architecture designed for high-speeed network
24packet processing. DPAA2 consists of sophisticated mechanisms for
25processing Ethernet packets, queue management, buffer management,
26autonomous L2 switching, virtual Ethernet bridging, and accelerator
27(e.g. crypto) sharing.
28
29A DPAA2 hardware component called the Management Complex (or MC) manages the
30DPAA2 hardware resources. The MC provides an object-based abstraction for
31software drivers to use the DPAA2 hardware.
32
33The MC uses DPAA2 hardware resources such as queues, buffer pools, and
34network ports to create functional objects/devices such as network
35interfaces, an L2 switch, or accelerator instances.
36
37The MC provides memory-mapped I/O command interfaces (MC portals)
38which DPAA2 software drivers use to operate on DPAA2 objects:
39
40 +--------------------------------------+
41 | OS |
42 | DPAA2 drivers |
43 | | |
44 +-----------------------------|--------+
45 |
46 | (create,discover,connect
47 | config,use,destroy)
48 |
49 DPAA2 |
50 +------------------------| mc portal |-+
51 | | |
52 | +- - - - - - - - - - - - -V- - -+ |
53 | | | |
54 | | Management Complex (MC) | |
55 | | | |
56 | +- - - - - - - - - - - - - - - -+ |
57 | |
58 | Hardware Hardware |
59 | Resources Objects |
60 | --------- ------- |
61 | -queues -DPRC |
62 | -buffer pools -DPMCP |
63 | -Eth MACs/ports -DPIO |
64 | -network interface -DPNI |
65 | profiles -DPMAC |
66 | -queue portals -DPBP |
67 | -MC portals ... |
68 | ... |
69 | |
70 +--------------------------------------+
71
72The MC mediates operations such as create, discover,
73connect, configuration, and destroy. Fast-path operations
74on data, such as packet transmit/receive, are not mediated by
75the MC and are done directly using memory mapped regions in
76DPIO objects.
77
78Overview of DPAA2 Objects
79-------------------------
80The section provides a brief overview of some key objects
81in the DPAA2 hardware. A simple scenario is described illustrating
82the objects involved in creating a network interfaces.
83
84-DPRC (Datapath Resource Container)
85
86 A DPRC is an container object that holds all the other
87 types of DPAA2 objects. In the example diagram below there
88 are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC)
89 in the container.
90
91 +---------------------------------------------------------+
92 | DPRC |
93 | |
94 | +-------+ +-------+ +-------+ +-------+ +-------+ |
95 | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | |
96 | +-------+ +-------+ +-------+ +---+---+ +---+---+ |
97 | | DPMCP | | DPIO | |
98 | +-------+ +-------+ |
99 | | DPMCP | |
100 | +-------+ |
101 | |
102 +---------------------------------------------------------+
103
104 From the point of view of an OS, a DPRC is bus-like. Like
105 a plug-and-play bus, such as PCI, DPRC commands can be used to
106 enumerate the contents of the DPRC, discover the hardware
107 objects present (including mappable regions and interrupts).
108
109 dprc.1 (bus)
110 |
111 +--+--------+-------+-------+-------+
112 | | | | |
113 dpmcp.1 dpio.1 dpbp.1 dpni.1 dpmac.1
114 dpmcp.2 dpio.2
115 dpmcp.3
116
117 Hardware objects can be created and destroyed dynamically, providing
118 the ability to hot plug/unplug objects in and out of the DPRC.
119
120 A DPRC has a mappable mmio region (an MC portal) that can be used
121 to send MC commands. It has an interrupt for status events (like
122 hotplug).
123
124 All objects in a container share the same hardware "isolation context".
125 This means that with respect to an IOMMU the isolation granularity
126 is at the DPRC (container) level, not at the individual object
127 level.
128
129 DPRCs can be defined statically and populated with objects
130 via a config file passed to the MC when firmware starts
131 it. There is also a Linux user space tool called "restool"
132 that can be used to create/destroy containers and objects
133 dynamically.
134
135-DPAA2 Objects for an Ethernet Network Interface
136
137 A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX
138 queuing mechanisms, configuration mechanisms, buffer management,
139 physical ports, and interrupts. DPAA2 uses a more granular approach
140 utilizing multiple hardware objects. Each object has specialized
141 functions, and are used together by software to provide Ethernet network
142 interface functionality. This approach provides efficient use of finite
143 hardware resources, flexibility, and performance advantages.
144
145 The diagram below shows the objects needed for a simple
146 network interface configuration on a system with 2 CPUs.
147
148 +---+---+ +---+---+
149 CPU0 CPU1
150 +---+---+ +---+---+
151 | |
152 +---+---+ +---+---+
153 DPIO DPIO
154 +---+---+ +---+---+
155 \ /
156 \ /
157 \ /
158 +---+---+
159 DPNI --- DPBP,DPMCP
160 +---+---+
161 |
162 |
163 +---+---+
164 DPMAC
165 +---+---+
166 |
167 port/PHY
168
169 Below the objects are described. For each object a brief description
170 is provided along with a summary of the kinds of operations the object
171 supports and a summary of key resources of the object (mmio regions
172 and irqs).
173
174 -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a
175 hardware device that connects to an Ethernet PHY and allows
176 physical transmission and reception of Ethernet frames.
177 -mmio regions: none
178 -irqs: dpni link change
179 -commands: set link up/down, link config, get stats,
180 irq config, enable, reset
181
182 -DPNI (Datapath Network Interface): contains TX/RX queues,
183 network interface configuration, and rx buffer pool configuration
184 mechanisms.
185 -mmio regions: none
186 -irqs: link state
187 -commands: port config, offload config, queue config,
188 parse/classify config, irq config, enable, reset
189
190 -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue
191 packets and do hardware buffer pool management operations. For
192 optimum performance there is typically DPIO per CPU. This allows
193 each CPU to perform simultaneous enqueue/dequeue operations.
194 -mmio regions: queue operations, buffer mgmt
195 -irqs: data availability, congestion notification, buffer
196 pool depletion
197 -commands: irq config, enable, reset
198
199 -DPBP (Datapath Buffer Pool): represents a hardware buffer
200 pool.
201 -mmio regions: none
202 -irqs: none
203 -commands: enable, reset
204
205 -DPMCP (Datapath MC Portal): provides an MC command portal.
206 Used by drivers to send commands to the MC to manage
207 objects.
208 -mmio regions: MC command portal
209 -irqs: command completion
210 -commands: irq config, enable, reset
211
212 Object Connections
213 ------------------
214 Some objects have explicit relationships that must
215 be configured:
216
217 -DPNI <--> DPMAC
218 -DPNI <--> DPNI
219 -DPNI <--> L2-switch-port
220 A DPNI must be connected to something such as a DPMAC,
221 another DPNI, or L2 switch port. The DPNI connection
222 is made via a DPRC command.
223
224 +-------+ +-------+
225 | DPNI | | DPMAC |
226 +---+---+ +---+---+
227 | |
228 +==========+
229
230 -DPNI <--> DPBP
231 A network interface requires a 'buffer pool' (DPBP
232 object) which provides a list of pointers to memory
233 where received Ethernet data is to be copied. The
234 Ethernet driver configures the DPBPs associated with
235 the network interface.
236
237 Interrupts
238 ----------
239 All interrupts generated by DPAA2 objects are message
240 interrupts. At the hardware level message interrupts
241 generated by devices will normally have 3 components--
242 1) a non-spoofable 'device-id' expressed on the hardware
243 bus, 2) an address, 3) a data value.
244
245 In the case of DPAA2 devices/objects, all objects in the
246 same container/DPRC share the same 'device-id'.
247 For ARM-based SoC this is the same as the stream ID.
248
249
250DPAA2 Linux Driver Overview
251---------------------------
252
253This section provides an overview of the Linux kernel drivers for
254DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure"
255drivers and 2) functional object drivers (such as Ethernet).
256
257As described previously, a DPRC is a container that holds the other
258types of DPAA2 objects. It is functionally similar to a plug-and-play
259bus controller.
260
261Each object in the DPRC is a Linux "device" and is bound to a driver.
262The diagram below shows the Linux drivers involved in a networking
263scenario and the objects bound to each driver. A brief description
264of each driver follows.
265
266 +------------+
267 | OS Network |
268 | Stack |
269 +------------+ +------------+
270 | Allocator |. . . . . . . | Ethernet |
271 |(dpmcp,dpbp)| | (dpni) |
272 +-.----------+ +---+---+----+
273 . . ^ |
274 . . <data avail, | |<enqueue,
275 . . tx confirm> | | dequeue>
276 +-------------+ . | |
277 | DPRC driver | . +---+---V----+ +---------+
278 | (dprc) | . . . . . .| DPIO driver| | MAC |
279 +----------+--+ | (dpio) | | (dpmac) |
280 | +------+-----+ +-----+---+
281 |<dev add/remove> | |
282 | | |
283 +----+--------------+ | +--+---+
284 | mc-bus driver | | | PHY |
285 | | | |driver|
286 | /fsl-mc@80c000000 | | +--+---+
287 +-------------------+ | |
288 | |
289 ================================ HARDWARE =========|=================|======
290 DPIO |
291 | |
292 DPNI---DPBP |
293 | |
294 DPMAC |
295 | |
296 PHY ---------------+
297 ===================================================|========================
298
299A brief description of each driver is provided below.
300
301 mc-bus driver
302 -------------
303 The mc-bus driver is a platform driver and is probed from an
304 "/fsl-mc@xxxx" node in the device tree passed in by boot firmware.
305 It is responsible for bootstrapping the DPAA2 kernel infrastructure.
306 Key functions include:
307 -registering a new bus type named "fsl-mc" with the kernel,
308 and implementing bus call-backs (e.g. match/uevent/dev_groups)
309 -implemeting APIs for DPAA2 driver registration and for device
310 add/remove
311 -creates an MSI irq domain
312 -do a device add of the 'root' DPRC device, which is needed
313 to bootstrap things
314
315 DPRC driver
316 -----------
317 The dprc-driver is bound DPRC objects and does runtime management
318 of a bus instance. It performs the initial bus scan of the DPRC
319 and handles interrupts for container events such as hot plug.
320
321 Allocator
322 ----------
323 Certain objects such as DPMCP and DPBP are generic and fungible,
324 and are intended to be used by other drivers. For example,
325 the DPAA2 Ethernet driver needs:
326 -DPMCPs to send MC commands, to configure network interfaces
327 -DPBPs for network buffer pools
328
329 The allocator driver registers for these allocatable object types
330 and those objects are bound to the allocator when the bus is probed.
331 The allocator maintains a pool of objects that are available for
332 allocation by other DPAA2 drivers.
333
334 DPIO driver
335 -----------
336 The DPIO driver is bound to DPIO objects and provides services that allow
337 other drivers such as the Ethernet driver to receive and transmit data.
338 Key services include:
339 -data availability notifications
340 -hardware queuing operations (enqueue and dequeue of data)
341 -hardware buffer pool management
342
343 There is typically one DPIO object per physical CPU for optimum
344 performance, allowing each CPU to simultaneously enqueue
345 and dequeue data.
346
347 The DPIO driver operates on behalf of all DPAA2 drivers
348 active in the kernel-- Ethernet, crypto, compression,
349 etc.
350
351 Ethernet
352 --------
353 The Ethernet driver is bound to a DPNI and implements the kernel
354 interfaces needed to connect the DPAA2 network interface to
355 the network stack.
356
357 Each DPNI corresponds to a Linux network interface.
358
359 MAC driver
360 ----------
361 An Ethernet PHY is an off-chip, board specific component and is managed
362 by the appropriate PHY driver via an mdio bus. The MAC driver
363 plays a role of being a proxy between the PHY driver and the
364 MC. It does this proxy via the MC commands to a DPMAC object.
365