1.. SPDX-License-Identifier: GPL-2.0 2 3====== 4AF_XDP 5====== 6 7Overview 8======== 9 10AF_XDP is an address family that is optimized for high performance 11packet processing. 12 13This document assumes that the reader is familiar with BPF and XDP. If 14not, the Cilium project has an excellent reference guide at 15http://cilium.readthedocs.io/en/latest/bpf/. 16 17Using the XDP_REDIRECT action from an XDP program, the program can 18redirect ingress frames to other XDP enabled netdevs, using the 19bpf_redirect_map() function. AF_XDP sockets enable the possibility for 20XDP programs to redirect frames to a memory buffer in a user-space 21application. 22 23An AF_XDP socket (XSK) is created with the normal socket() 24syscall. Associated with each XSK are two rings: the RX ring and the 25TX ring. A socket can receive packets on the RX ring and it can send 26packets on the TX ring. These rings are registered and sized with the 27setsockopts XDP_RX_RING and XDP_TX_RING, respectively. It is mandatory 28to have at least one of these rings for each socket. An RX or TX 29descriptor ring points to a data buffer in a memory area called a 30UMEM. RX and TX can share the same UMEM so that a packet does not have 31to be copied between RX and TX. Moreover, if a packet needs to be kept 32for a while due to a possible retransmit, the descriptor that points 33to that packet can be changed to point to another and reused right 34away. This again avoids copying data. 35 36The UMEM consists of a number of equally sized chunks. A descriptor in 37one of the rings references a frame by referencing its addr. The addr 38is simply an offset within the entire UMEM region. The user space 39allocates memory for this UMEM using whatever means it feels is most 40appropriate (malloc, mmap, huge pages, etc). This memory area is then 41registered with the kernel using the new setsockopt XDP_UMEM_REG. The 42UMEM also has two rings: the FILL ring and the COMPLETION ring. The 43fill ring is used by the application to send down addr for the kernel 44to fill in with RX packet data. References to these frames will then 45appear in the RX ring once each packet has been received. The 46completion ring, on the other hand, contains frame addr that the 47kernel has transmitted completely and can now be used again by user 48space, for either TX or RX. Thus, the frame addrs appearing in the 49completion ring are addrs that were previously transmitted using the 50TX ring. In summary, the RX and FILL rings are used for the RX path 51and the TX and COMPLETION rings are used for the TX path. 52 53The socket is then finally bound with a bind() call to a device and a 54specific queue id on that device, and it is not until bind is 55completed that traffic starts to flow. 56 57The UMEM can be shared between processes, if desired. If a process 58wants to do this, it simply skips the registration of the UMEM and its 59corresponding two rings, sets the XDP_SHARED_UMEM flag in the bind 60call and submits the XSK of the process it would like to share UMEM 61with as well as its own newly created XSK socket. The new process will 62then receive frame addr references in its own RX ring that point to 63this shared UMEM. Note that since the ring structures are 64single-consumer / single-producer (for performance reasons), the new 65process has to create its own socket with associated RX and TX rings, 66since it cannot share this with the other process. This is also the 67reason that there is only one set of FILL and COMPLETION rings per 68UMEM. It is the responsibility of a single process to handle the UMEM. 69 70How is then packets distributed from an XDP program to the XSKs? There 71is a BPF map called XSKMAP (or BPF_MAP_TYPE_XSKMAP in full). The 72user-space application can place an XSK at an arbitrary place in this 73map. The XDP program can then redirect a packet to a specific index in 74this map and at this point XDP validates that the XSK in that map was 75indeed bound to that device and ring number. If not, the packet is 76dropped. If the map is empty at that index, the packet is also 77dropped. This also means that it is currently mandatory to have an XDP 78program loaded (and one XSK in the XSKMAP) to be able to get any 79traffic to user space through the XSK. 80 81AF_XDP can operate in two different modes: XDP_SKB and XDP_DRV. If the 82driver does not have support for XDP, or XDP_SKB is explicitly chosen 83when loading the XDP program, XDP_SKB mode is employed that uses SKBs 84together with the generic XDP support and copies out the data to user 85space. A fallback mode that works for any network device. On the other 86hand, if the driver has support for XDP, it will be used by the AF_XDP 87code to provide better performance, but there is still a copy of the 88data into user space. 89 90Concepts 91======== 92 93In order to use an AF_XDP socket, a number of associated objects need 94to be setup. 95 96Jonathan Corbet has also written an excellent article on LWN, 97"Accelerating networking with AF_XDP". It can be found at 98https://lwn.net/Articles/750845/. 99 100UMEM 101---- 102 103UMEM is a region of virtual contiguous memory, divided into 104equal-sized frames. An UMEM is associated to a netdev and a specific 105queue id of that netdev. It is created and configured (chunk size, 106headroom, start address and size) by using the XDP_UMEM_REG setsockopt 107system call. A UMEM is bound to a netdev and queue id, via the bind() 108system call. 109 110An AF_XDP is socket linked to a single UMEM, but one UMEM can have 111multiple AF_XDP sockets. To share an UMEM created via one socket A, 112the next socket B can do this by setting the XDP_SHARED_UMEM flag in 113struct sockaddr_xdp member sxdp_flags, and passing the file descriptor 114of A to struct sockaddr_xdp member sxdp_shared_umem_fd. 115 116The UMEM has two single-producer/single-consumer rings, that are used 117to transfer ownership of UMEM frames between the kernel and the 118user-space application. 119 120Rings 121----- 122 123There are a four different kind of rings: Fill, Completion, RX and 124TX. All rings are single-producer/single-consumer, so the user-space 125application need explicit synchronization of multiple 126processes/threads are reading/writing to them. 127 128The UMEM uses two rings: Fill and Completion. Each socket associated 129with the UMEM must have an RX queue, TX queue or both. Say, that there 130is a setup with four sockets (all doing TX and RX). Then there will be 131one Fill ring, one Completion ring, four TX rings and four RX rings. 132 133The rings are head(producer)/tail(consumer) based rings. A producer 134writes the data ring at the index pointed out by struct xdp_ring 135producer member, and increasing the producer index. A consumer reads 136the data ring at the index pointed out by struct xdp_ring consumer 137member, and increasing the consumer index. 138 139The rings are configured and created via the _RING setsockopt system 140calls and mmapped to user-space using the appropriate offset to mmap() 141(XDP_PGOFF_RX_RING, XDP_PGOFF_TX_RING, XDP_UMEM_PGOFF_FILL_RING and 142XDP_UMEM_PGOFF_COMPLETION_RING). 143 144The size of the rings need to be of size power of two. 145 146UMEM Fill Ring 147~~~~~~~~~~~~~~ 148 149The Fill ring is used to transfer ownership of UMEM frames from 150user-space to kernel-space. The UMEM addrs are passed in the ring. As 151an example, if the UMEM is 64k and each chunk is 4k, then the UMEM has 15216 chunks and can pass addrs between 0 and 64k. 153 154Frames passed to the kernel are used for the ingress path (RX rings). 155 156The user application produces UMEM addrs to this ring. Note that, if 157running the application with aligned chunk mode, the kernel will mask 158the incoming addr. E.g. for a chunk size of 2k, the log2(2048) LSB of 159the addr will be masked off, meaning that 2048, 2050 and 3000 refers 160to the same chunk. If the user application is run in the unaligned 161chunks mode, then the incoming addr will be left untouched. 162 163 164UMEM Completion Ring 165~~~~~~~~~~~~~~~~~~~~ 166 167The Completion Ring is used transfer ownership of UMEM frames from 168kernel-space to user-space. Just like the Fill ring, UMEM indicies are 169used. 170 171Frames passed from the kernel to user-space are frames that has been 172sent (TX ring) and can be used by user-space again. 173 174The user application consumes UMEM addrs from this ring. 175 176 177RX Ring 178~~~~~~~ 179 180The RX ring is the receiving side of a socket. Each entry in the ring 181is a struct xdp_desc descriptor. The descriptor contains UMEM offset 182(addr) and the length of the data (len). 183 184If no frames have been passed to kernel via the Fill ring, no 185descriptors will (or can) appear on the RX ring. 186 187The user application consumes struct xdp_desc descriptors from this 188ring. 189 190TX Ring 191~~~~~~~ 192 193The TX ring is used to send frames. The struct xdp_desc descriptor is 194filled (index, length and offset) and passed into the ring. 195 196To start the transfer a sendmsg() system call is required. This might 197be relaxed in the future. 198 199The user application produces struct xdp_desc descriptors to this 200ring. 201 202XSKMAP / BPF_MAP_TYPE_XSKMAP 203---------------------------- 204 205On XDP side there is a BPF map type BPF_MAP_TYPE_XSKMAP (XSKMAP) that 206is used in conjunction with bpf_redirect_map() to pass the ingress 207frame to a socket. 208 209The user application inserts the socket into the map, via the bpf() 210system call. 211 212Note that if an XDP program tries to redirect to a socket that does 213not match the queue configuration and netdev, the frame will be 214dropped. E.g. an AF_XDP socket is bound to netdev eth0 and 215queue 17. Only the XDP program executing for eth0 and queue 17 will 216successfully pass data to the socket. Please refer to the sample 217application (samples/bpf/) in for an example. 218 219Usage 220===== 221 222In order to use AF_XDP sockets there are two parts needed. The 223user-space application and the XDP program. For a complete setup and 224usage example, please refer to the sample application. The user-space 225side is xdpsock_user.c and the XDP side is part of libbpf. 226 227The XDP code sample included in tools/lib/bpf/xsk.c is the following:: 228 229 SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx) 230 { 231 int index = ctx->rx_queue_index; 232 233 // A set entry here means that the correspnding queue_id 234 // has an active AF_XDP socket bound to it. 235 if (bpf_map_lookup_elem(&xsks_map, &index)) 236 return bpf_redirect_map(&xsks_map, index, 0); 237 238 return XDP_PASS; 239 } 240 241Naive ring dequeue and enqueue could look like this:: 242 243 // struct xdp_rxtx_ring { 244 // __u32 *producer; 245 // __u32 *consumer; 246 // struct xdp_desc *desc; 247 // }; 248 249 // struct xdp_umem_ring { 250 // __u32 *producer; 251 // __u32 *consumer; 252 // __u64 *desc; 253 // }; 254 255 // typedef struct xdp_rxtx_ring RING; 256 // typedef struct xdp_umem_ring RING; 257 258 // typedef struct xdp_desc RING_TYPE; 259 // typedef __u64 RING_TYPE; 260 261 int dequeue_one(RING *ring, RING_TYPE *item) 262 { 263 __u32 entries = *ring->producer - *ring->consumer; 264 265 if (entries == 0) 266 return -1; 267 268 // read-barrier! 269 270 *item = ring->desc[*ring->consumer & (RING_SIZE - 1)]; 271 (*ring->consumer)++; 272 return 0; 273 } 274 275 int enqueue_one(RING *ring, const RING_TYPE *item) 276 { 277 u32 free_entries = RING_SIZE - (*ring->producer - *ring->consumer); 278 279 if (free_entries == 0) 280 return -1; 281 282 ring->desc[*ring->producer & (RING_SIZE - 1)] = *item; 283 284 // write-barrier! 285 286 (*ring->producer)++; 287 return 0; 288 } 289 290 291For a more optimized version, please refer to the sample application. 292 293Sample application 294================== 295 296There is a xdpsock benchmarking/test application included that 297demonstrates how to use AF_XDP sockets with both private and shared 298UMEMs. Say that you would like your UDP traffic from port 4242 to end 299up in queue 16, that we will enable AF_XDP on. Here, we use ethtool 300for this:: 301 302 ethtool -N p3p2 rx-flow-hash udp4 fn 303 ethtool -N p3p2 flow-type udp4 src-port 4242 dst-port 4242 \ 304 action 16 305 306Running the rxdrop benchmark in XDP_DRV mode can then be done 307using:: 308 309 samples/bpf/xdpsock -i p3p2 -q 16 -r -N 310 311For XDP_SKB mode, use the switch "-S" instead of "-N" and all options 312can be displayed with "-h", as usual. 313 314FAQ 315======= 316 317Q: I am not seeing any traffic on the socket. What am I doing wrong? 318 319A: When a netdev of a physical NIC is initialized, Linux usually 320 allocates one Rx and Tx queue pair per core. So on a 8 core system, 321 queue ids 0 to 7 will be allocated, one per core. In the AF_XDP 322 bind call or the xsk_socket__create libbpf function call, you 323 specify a specific queue id to bind to and it is only the traffic 324 towards that queue you are going to get on you socket. So in the 325 example above, if you bind to queue 0, you are NOT going to get any 326 traffic that is distributed to queues 1 through 7. If you are 327 lucky, you will see the traffic, but usually it will end up on one 328 of the queues you have not bound to. 329 330 There are a number of ways to solve the problem of getting the 331 traffic you want to the queue id you bound to. If you want to see 332 all the traffic, you can force the netdev to only have 1 queue, queue 333 id 0, and then bind to queue 0. You can use ethtool to do this:: 334 335 sudo ethtool -L <interface> combined 1 336 337 If you want to only see part of the traffic, you can program the 338 NIC through ethtool to filter out your traffic to a single queue id 339 that you can bind your XDP socket to. Here is one example in which 340 UDP traffic to and from port 4242 are sent to queue 2:: 341 342 sudo ethtool -N <interface> rx-flow-hash udp4 fn 343 sudo ethtool -N <interface> flow-type udp4 src-port 4242 dst-port \ 344 4242 action 2 345 346 A number of other ways are possible all up to the capabilitites of 347 the NIC you have. 348 349Credits 350======= 351 352- Björn Töpel (AF_XDP core) 353- Magnus Karlsson (AF_XDP core) 354- Alexander Duyck 355- Alexei Starovoitov 356- Daniel Borkmann 357- Jesper Dangaard Brouer 358- John Fastabend 359- Jonathan Corbet (LWN coverage) 360- Michael S. Tsirkin 361- Qi Z Zhang 362- Willem de Bruijn 363