1Virtual Routing and Forwarding (VRF) 2==================================== 3The VRF device combined with ip rules provides the ability to create virtual 4routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the 5Linux network stack. One use case is the multi-tenancy problem where each 6tenant has their own unique routing tables and in the very least need 7different default gateways. 8 9Processes can be "VRF aware" by binding a socket to the VRF device. Packets 10through the socket then use the routing table associated with the VRF 11device. An important feature of the VRF device implementation is that it 12impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected 13(ie., they do not need to be run in each VRF). The design also allows 14the use of higher priority ip rules (Policy Based Routing, PBR) to take 15precedence over the VRF device rules directing specific traffic as desired. 16 17In addition, VRF devices allow VRFs to be nested within namespaces. For 18example network namespaces provide separation of network interfaces at the 19device layer, VLANs on the interfaces within a namespace provide L2 separation 20and then VRF devices provide L3 separation. 21 22Design 23------ 24A VRF device is created with an associated route table. Network interfaces 25are then enslaved to a VRF device: 26 27 +-----------------------------+ 28 | vrf-blue | ===> route table 10 29 +-----------------------------+ 30 | | | 31 +------+ +------+ +-------------+ 32 | eth1 | | eth2 | ... | bond1 | 33 +------+ +------+ +-------------+ 34 | | 35 +------+ +------+ 36 | eth8 | | eth9 | 37 +------+ +------+ 38 39Packets received on an enslaved device and are switched to the VRF device 40in the IPv4 and IPv6 processing stacks giving the impression that packets 41flow through the VRF device. Similarly on egress routing rules are used to 42send packets to the VRF device driver before getting sent out the actual 43interface. This allows tcpdump on a VRF device to capture all packets into 44and out of the VRF as a whole.[1] Similarly, netfilter[2] and tc rules can be 45applied using the VRF device to specify rules that apply to the VRF domain 46as a whole. 47 48[1] Packets in the forwarded state do not flow through the device, so those 49 packets are not seen by tcpdump. Will revisit this limitation in a 50 future release. 51 52[2] Iptables on ingress supports PREROUTING with skb->dev set to the real 53 ingress device and both INPUT and PREROUTING rules with skb->dev set to 54 the VRF device. For egress POSTROUTING and OUTPUT rules can be written 55 using either the VRF device or real egress device. 56 57Setup 58----- 591. VRF device is created with an association to a FIB table. 60 e.g, ip link add vrf-blue type vrf table 10 61 ip link set dev vrf-blue up 62 632. An l3mdev FIB rule directs lookups to the table associated with the device. 64 A single l3mdev rule is sufficient for all VRFs. The VRF device adds the 65 l3mdev rule for IPv4 and IPv6 when the first device is created with a 66 default preference of 1000. Users may delete the rule if desired and add 67 with a different priority or install per-VRF rules. 68 69 Prior to the v4.8 kernel iif and oif rules are needed for each VRF device: 70 ip ru add oif vrf-blue table 10 71 ip ru add iif vrf-blue table 10 72 733. Set the default route for the table (and hence default route for the VRF). 74 ip route add table 10 unreachable default 75 764. Enslave L3 interfaces to a VRF device. 77 ip link set dev eth1 master vrf-blue 78 79 Local and connected routes for enslaved devices are automatically moved to 80 the table associated with VRF device. Any additional routes depending on 81 the enslaved device are dropped and will need to be reinserted to the VRF 82 FIB table following the enslavement. 83 84 The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 global 85 addresses as VRF enslavement changes. 86 sysctl -w net.ipv6.conf.all.keep_addr_on_down=1 87 885. Additional VRF routes are added to associated table. 89 ip route add table 10 ... 90 91 92Applications 93------------ 94Applications that are to work within a VRF need to bind their socket to the 95VRF device: 96 97 setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1); 98 99or to specify the output device using cmsg and IP_PKTINFO. 100 101TCP & UDP services running in the default VRF context (ie., not bound 102to any VRF device) can work across all VRF domains by enabling the 103tcp_l3mdev_accept and udp_l3mdev_accept sysctl options: 104 sysctl -w net.ipv4.tcp_l3mdev_accept=1 105 sysctl -w net.ipv4.udp_l3mdev_accept=1 106 107netfilter rules on the VRF device can be used to limit access to services 108running in the default VRF context as well. 109 110The default VRF does not have limited scope with respect to port bindings. 111That is, if a process does a wildcard bind to a port in the default VRF it 112owns the port across all VRF domains within the network namespace. 113 114################################################################################ 115 116Using iproute2 for VRFs 117======================= 118iproute2 supports the vrf keyword as of v4.7. For backwards compatibility this 119section lists both commands where appropriate -- with the vrf keyword and the 120older form without it. 121 1221. Create a VRF 123 124 To instantiate a VRF device and associate it with a table: 125 $ ip link add dev NAME type vrf table ID 126 127 As of v4.8 the kernel supports the l3mdev FIB rule where a single rule 128 covers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on first 129 device create. 130 1312. List VRFs 132 133 To list VRFs that have been created: 134 $ ip [-d] link show type vrf 135 NOTE: The -d option is needed to show the table id 136 137 For example: 138 $ ip -d link show type vrf 139 11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 140 link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0 141 vrf table 1 addrgenmode eui64 142 12: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 143 link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0 144 vrf table 10 addrgenmode eui64 145 13: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 146 link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0 147 vrf table 66 addrgenmode eui64 148 14: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 149 link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0 150 vrf table 81 addrgenmode eui64 151 152 153 Or in brief output: 154 155 $ ip -br link show type vrf 156 mgmt UP 72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP> 157 red UP b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP> 158 blue UP 36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP> 159 green UP e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP> 160 161 1623. Assign a Network Interface to a VRF 163 164 Network interfaces are assigned to a VRF by enslaving the netdevice to a 165 VRF device: 166 $ ip link set dev NAME master NAME 167 168 On enslavement connected and local routes are automatically moved to the 169 table associated with the VRF device. 170 171 For example: 172 $ ip link set dev eth0 master mgmt 173 174 1754. Show Devices Assigned to a VRF 176 177 To show devices that have been assigned to a specific VRF add the master 178 option to the ip command: 179 $ ip link show vrf NAME 180 $ ip link show master NAME 181 182 For example: 183 $ ip link show vrf red 184 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000 185 link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff 186 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000 187 link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff 188 7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000 189 link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff 190 191 192 Or using the brief output: 193 $ ip -br link show vrf red 194 eth1 UP 02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP> 195 eth2 UP 02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP> 196 eth5 DOWN 02:00:00:00:02:06 <BROADCAST,MULTICAST> 197 198 1995. Show Neighbor Entries for a VRF 200 201 To list neighbor entries associated with devices enslaved to a VRF device 202 add the master option to the ip command: 203 $ ip [-6] neigh show vrf NAME 204 $ ip [-6] neigh show master NAME 205 206 For example: 207 $ ip neigh show vrf red 208 10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE 209 10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE 210 211 $ ip -6 neigh show vrf red 212 2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE 213 214 2156. Show Addresses for a VRF 216 217 To show addresses for interfaces associated with a VRF add the master 218 option to the ip command: 219 $ ip addr show vrf NAME 220 $ ip addr show master NAME 221 222 For example: 223 $ ip addr show vrf red 224 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000 225 link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff 226 inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1 227 valid_lft forever preferred_lft forever 228 inet6 2002:1::2/120 scope global 229 valid_lft forever preferred_lft forever 230 inet6 fe80::ff:fe00:202/64 scope link 231 valid_lft forever preferred_lft forever 232 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000 233 link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff 234 inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2 235 valid_lft forever preferred_lft forever 236 inet6 2002:2::2/120 scope global 237 valid_lft forever preferred_lft forever 238 inet6 fe80::ff:fe00:203/64 scope link 239 valid_lft forever preferred_lft forever 240 7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000 241 link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff 242 243 Or in brief format: 244 $ ip -br addr show vrf red 245 eth1 UP 10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64 246 eth2 UP 10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64 247 eth5 DOWN 248 249 2507. Show Routes for a VRF 251 252 To show routes for a VRF use the ip command to display the table associated 253 with the VRF device: 254 $ ip [-6] route show vrf NAME 255 $ ip [-6] route show table ID 256 257 For example: 258 $ ip route show vrf red 259 prohibit default 260 broadcast 10.2.1.0 dev eth1 proto kernel scope link src 10.2.1.2 261 10.2.1.0/24 dev eth1 proto kernel scope link src 10.2.1.2 262 local 10.2.1.2 dev eth1 proto kernel scope host src 10.2.1.2 263 broadcast 10.2.1.255 dev eth1 proto kernel scope link src 10.2.1.2 264 broadcast 10.2.2.0 dev eth2 proto kernel scope link src 10.2.2.2 265 10.2.2.0/24 dev eth2 proto kernel scope link src 10.2.2.2 266 local 10.2.2.2 dev eth2 proto kernel scope host src 10.2.2.2 267 broadcast 10.2.2.255 dev eth2 proto kernel scope link src 10.2.2.2 268 269 $ ip -6 route show vrf red 270 local 2002:1:: dev lo proto none metric 0 pref medium 271 local 2002:1::2 dev lo proto none metric 0 pref medium 272 2002:1::/120 dev eth1 proto kernel metric 256 pref medium 273 local 2002:2:: dev lo proto none metric 0 pref medium 274 local 2002:2::2 dev lo proto none metric 0 pref medium 275 2002:2::/120 dev eth2 proto kernel metric 256 pref medium 276 local fe80:: dev lo proto none metric 0 pref medium 277 local fe80:: dev lo proto none metric 0 pref medium 278 local fe80::ff:fe00:202 dev lo proto none metric 0 pref medium 279 local fe80::ff:fe00:203 dev lo proto none metric 0 pref medium 280 fe80::/64 dev eth1 proto kernel metric 256 pref medium 281 fe80::/64 dev eth2 proto kernel metric 256 pref medium 282 ff00::/8 dev red metric 256 pref medium 283 ff00::/8 dev eth1 metric 256 pref medium 284 ff00::/8 dev eth2 metric 256 pref medium 285 286 2878. Route Lookup for a VRF 288 289 A test route lookup can be done for a VRF: 290 $ ip [-6] route get vrf NAME ADDRESS 291 $ ip [-6] route get oif NAME ADDRESS 292 293 For example: 294 $ ip route get 10.2.1.40 vrf red 295 10.2.1.40 dev eth1 table red src 10.2.1.2 296 cache 297 298 $ ip -6 route get 2002:1::32 vrf red 299 2002:1::32 from :: dev eth1 table red proto kernel src 2002:1::2 metric 256 pref medium 300 301 3029. Removing Network Interface from a VRF 303 304 Network interfaces are removed from a VRF by breaking the enslavement to 305 the VRF device: 306 $ ip link set dev NAME nomaster 307 308 Connected routes are moved back to the default table and local entries are 309 moved to the local table. 310 311 For example: 312 $ ip link set dev eth0 nomaster 313 314-------------------------------------------------------------------------------- 315 316Commands used in this example: 317 318cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF 3191 mgmt 32010 red 32166 blue 32281 green 323EOF 324 325function vrf_create 326{ 327 VRF=$1 328 TBID=$2 329 330 # create VRF device 331 ip link add ${VRF} type vrf table ${TBID} 332 333 if [ "${VRF}" != "mgmt" ]; then 334 ip route add table ${TBID} unreachable default 335 fi 336 ip link set dev ${VRF} up 337} 338 339vrf_create mgmt 1 340ip link set dev eth0 master mgmt 341 342vrf_create red 10 343ip link set dev eth1 master red 344ip link set dev eth2 master red 345ip link set dev eth5 master red 346 347vrf_create blue 66 348ip link set dev eth3 master blue 349 350vrf_create green 81 351ip link set dev eth4 master green 352 353 354Interface addresses from /etc/network/interfaces: 355auto eth0 356iface eth0 inet static 357 address 10.0.0.2 358 netmask 255.255.255.0 359 gateway 10.0.0.254 360 361iface eth0 inet6 static 362 address 2000:1::2 363 netmask 120 364 365auto eth1 366iface eth1 inet static 367 address 10.2.1.2 368 netmask 255.255.255.0 369 370iface eth1 inet6 static 371 address 2002:1::2 372 netmask 120 373 374auto eth2 375iface eth2 inet static 376 address 10.2.2.2 377 netmask 255.255.255.0 378 379iface eth2 inet6 static 380 address 2002:2::2 381 netmask 120 382 383auto eth3 384iface eth3 inet static 385 address 10.2.3.2 386 netmask 255.255.255.0 387 388iface eth3 inet6 static 389 address 2002:3::2 390 netmask 120 391 392auto eth4 393iface eth4 inet static 394 address 10.2.4.2 395 netmask 255.255.255.0 396 397iface eth4 inet6 static 398 address 2002:4::2 399 netmask 120 400