1Virtual Routing and Forwarding (VRF) 2==================================== 3The VRF device combined with ip rules provides the ability to create virtual 4routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the 5Linux network stack. One use case is the multi-tenancy problem where each 6tenant has their own unique routing tables and in the very least need 7different default gateways. 8 9Processes can be "VRF aware" by binding a socket to the VRF device. Packets 10through the socket then use the routing table associated with the VRF 11device. An important feature of the VRF device implementation is that it 12impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected 13(ie., they do not need to be run in each VRF). The design also allows 14the use of higher priority ip rules (Policy Based Routing, PBR) to take 15precedence over the VRF device rules directing specific traffic as desired. 16 17In addition, VRF devices allow VRFs to be nested within namespaces. For 18example network namespaces provide separation of network interfaces at the 19device layer, VLANs on the interfaces within a namespace provide L2 separation 20and then VRF devices provide L3 separation. 21 22Design 23------ 24A VRF device is created with an associated route table. Network interfaces 25are then enslaved to a VRF device: 26 27 +-----------------------------+ 28 | vrf-blue | ===> route table 10 29 +-----------------------------+ 30 | | | 31 +------+ +------+ +-------------+ 32 | eth1 | | eth2 | ... | bond1 | 33 +------+ +------+ +-------------+ 34 | | 35 +------+ +------+ 36 | eth8 | | eth9 | 37 +------+ +------+ 38 39Packets received on an enslaved device and are switched to the VRF device 40in the IPv4 and IPv6 processing stacks giving the impression that packets 41flow through the VRF device. Similarly on egress routing rules are used to 42send packets to the VRF device driver before getting sent out the actual 43interface. This allows tcpdump on a VRF device to capture all packets into 44and out of the VRF as a whole.[1] Similarly, netfilter[2] and tc rules can be 45applied using the VRF device to specify rules that apply to the VRF domain 46as a whole. 47 48[1] Packets in the forwarded state do not flow through the device, so those 49 packets are not seen by tcpdump. Will revisit this limitation in a 50 future release. 51 52[2] Iptables on ingress supports PREROUTING with skb->dev set to the real 53 ingress device and both INPUT and PREROUTING rules with skb->dev set to 54 the VRF device. For egress POSTROUTING and OUTPUT rules can be written 55 using either the VRF device or real egress device. 56 57Setup 58----- 591. VRF device is created with an association to a FIB table. 60 e.g, ip link add vrf-blue type vrf table 10 61 ip link set dev vrf-blue up 62 632. An l3mdev FIB rule directs lookups to the table associated with the device. 64 A single l3mdev rule is sufficient for all VRFs. The VRF device adds the 65 l3mdev rule for IPv4 and IPv6 when the first device is created with a 66 default preference of 1000. Users may delete the rule if desired and add 67 with a different priority or install per-VRF rules. 68 69 Prior to the v4.8 kernel iif and oif rules are needed for each VRF device: 70 ip ru add oif vrf-blue table 10 71 ip ru add iif vrf-blue table 10 72 733. Set the default route for the table (and hence default route for the VRF). 74 ip route add table 10 unreachable default 75 764. Enslave L3 interfaces to a VRF device. 77 ip link set dev eth1 master vrf-blue 78 79 Local and connected routes for enslaved devices are automatically moved to 80 the table associated with VRF device. Any additional routes depending on 81 the enslaved device are dropped and will need to be reinserted to the VRF 82 FIB table following the enslavement. 83 84 The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 global 85 addresses as VRF enslavement changes. 86 sysctl -w net.ipv6.conf.all.keep_addr_on_down=1 87 885. Additional VRF routes are added to associated table. 89 ip route add table 10 ... 90 91 92Applications 93------------ 94Applications that are to work within a VRF need to bind their socket to the 95VRF device: 96 97 setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1); 98 99or to specify the output device using cmsg and IP_PKTINFO. 100 101TCP services running in the default VRF context (ie., not bound to any VRF 102device) can work across all VRF domains by enabling the tcp_l3mdev_accept 103sysctl option: 104 sysctl -w net.ipv4.tcp_l3mdev_accept=1 105 106netfilter rules on the VRF device can be used to limit access to services 107running in the default VRF context as well. 108 109The default VRF does not have limited scope with respect to port bindings. 110That is, if a process does a wildcard bind to a port in the default VRF it 111owns the port across all VRF domains within the network namespace. 112 113################################################################################ 114 115Using iproute2 for VRFs 116======================= 117iproute2 supports the vrf keyword as of v4.7. For backwards compatibility this 118section lists both commands where appropriate -- with the vrf keyword and the 119older form without it. 120 1211. Create a VRF 122 123 To instantiate a VRF device and associate it with a table: 124 $ ip link add dev NAME type vrf table ID 125 126 As of v4.8 the kernel supports the l3mdev FIB rule where a single rule 127 covers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on first 128 device create. 129 1302. List VRFs 131 132 To list VRFs that have been created: 133 $ ip [-d] link show type vrf 134 NOTE: The -d option is needed to show the table id 135 136 For example: 137 $ ip -d link show type vrf 138 11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 139 link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0 140 vrf table 1 addrgenmode eui64 141 12: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 142 link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0 143 vrf table 10 addrgenmode eui64 144 13: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 145 link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0 146 vrf table 66 addrgenmode eui64 147 14: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 148 link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0 149 vrf table 81 addrgenmode eui64 150 151 152 Or in brief output: 153 154 $ ip -br link show type vrf 155 mgmt UP 72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP> 156 red UP b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP> 157 blue UP 36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP> 158 green UP e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP> 159 160 1613. Assign a Network Interface to a VRF 162 163 Network interfaces are assigned to a VRF by enslaving the netdevice to a 164 VRF device: 165 $ ip link set dev NAME master NAME 166 167 On enslavement connected and local routes are automatically moved to the 168 table associated with the VRF device. 169 170 For example: 171 $ ip link set dev eth0 master mgmt 172 173 1744. Show Devices Assigned to a VRF 175 176 To show devices that have been assigned to a specific VRF add the master 177 option to the ip command: 178 $ ip link show vrf NAME 179 $ ip link show master NAME 180 181 For example: 182 $ ip link show vrf red 183 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000 184 link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff 185 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000 186 link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff 187 7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000 188 link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff 189 190 191 Or using the brief output: 192 $ ip -br link show vrf red 193 eth1 UP 02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP> 194 eth2 UP 02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP> 195 eth5 DOWN 02:00:00:00:02:06 <BROADCAST,MULTICAST> 196 197 1985. Show Neighbor Entries for a VRF 199 200 To list neighbor entries associated with devices enslaved to a VRF device 201 add the master option to the ip command: 202 $ ip [-6] neigh show vrf NAME 203 $ ip [-6] neigh show master NAME 204 205 For example: 206 $ ip neigh show vrf red 207 10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE 208 10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE 209 210 $ ip -6 neigh show vrf red 211 2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE 212 213 2146. Show Addresses for a VRF 215 216 To show addresses for interfaces associated with a VRF add the master 217 option to the ip command: 218 $ ip addr show vrf NAME 219 $ ip addr show master NAME 220 221 For example: 222 $ ip addr show vrf red 223 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000 224 link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff 225 inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1 226 valid_lft forever preferred_lft forever 227 inet6 2002:1::2/120 scope global 228 valid_lft forever preferred_lft forever 229 inet6 fe80::ff:fe00:202/64 scope link 230 valid_lft forever preferred_lft forever 231 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000 232 link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff 233 inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2 234 valid_lft forever preferred_lft forever 235 inet6 2002:2::2/120 scope global 236 valid_lft forever preferred_lft forever 237 inet6 fe80::ff:fe00:203/64 scope link 238 valid_lft forever preferred_lft forever 239 7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000 240 link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff 241 242 Or in brief format: 243 $ ip -br addr show vrf red 244 eth1 UP 10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64 245 eth2 UP 10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64 246 eth5 DOWN 247 248 2497. Show Routes for a VRF 250 251 To show routes for a VRF use the ip command to display the table associated 252 with the VRF device: 253 $ ip [-6] route show vrf NAME 254 $ ip [-6] route show table ID 255 256 For example: 257 $ ip route show vrf red 258 prohibit default 259 broadcast 10.2.1.0 dev eth1 proto kernel scope link src 10.2.1.2 260 10.2.1.0/24 dev eth1 proto kernel scope link src 10.2.1.2 261 local 10.2.1.2 dev eth1 proto kernel scope host src 10.2.1.2 262 broadcast 10.2.1.255 dev eth1 proto kernel scope link src 10.2.1.2 263 broadcast 10.2.2.0 dev eth2 proto kernel scope link src 10.2.2.2 264 10.2.2.0/24 dev eth2 proto kernel scope link src 10.2.2.2 265 local 10.2.2.2 dev eth2 proto kernel scope host src 10.2.2.2 266 broadcast 10.2.2.255 dev eth2 proto kernel scope link src 10.2.2.2 267 268 $ ip -6 route show vrf red 269 local 2002:1:: dev lo proto none metric 0 pref medium 270 local 2002:1::2 dev lo proto none metric 0 pref medium 271 2002:1::/120 dev eth1 proto kernel metric 256 pref medium 272 local 2002:2:: dev lo proto none metric 0 pref medium 273 local 2002:2::2 dev lo proto none metric 0 pref medium 274 2002:2::/120 dev eth2 proto kernel metric 256 pref medium 275 local fe80:: dev lo proto none metric 0 pref medium 276 local fe80:: dev lo proto none metric 0 pref medium 277 local fe80::ff:fe00:202 dev lo proto none metric 0 pref medium 278 local fe80::ff:fe00:203 dev lo proto none metric 0 pref medium 279 fe80::/64 dev eth1 proto kernel metric 256 pref medium 280 fe80::/64 dev eth2 proto kernel metric 256 pref medium 281 ff00::/8 dev red metric 256 pref medium 282 ff00::/8 dev eth1 metric 256 pref medium 283 ff00::/8 dev eth2 metric 256 pref medium 284 285 2868. Route Lookup for a VRF 287 288 A test route lookup can be done for a VRF: 289 $ ip [-6] route get vrf NAME ADDRESS 290 $ ip [-6] route get oif NAME ADDRESS 291 292 For example: 293 $ ip route get 10.2.1.40 vrf red 294 10.2.1.40 dev eth1 table red src 10.2.1.2 295 cache 296 297 $ ip -6 route get 2002:1::32 vrf red 298 2002:1::32 from :: dev eth1 table red proto kernel src 2002:1::2 metric 256 pref medium 299 300 3019. Removing Network Interface from a VRF 302 303 Network interfaces are removed from a VRF by breaking the enslavement to 304 the VRF device: 305 $ ip link set dev NAME nomaster 306 307 Connected routes are moved back to the default table and local entries are 308 moved to the local table. 309 310 For example: 311 $ ip link set dev eth0 nomaster 312 313-------------------------------------------------------------------------------- 314 315Commands used in this example: 316 317cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF 3181 mgmt 31910 red 32066 blue 32181 green 322EOF 323 324function vrf_create 325{ 326 VRF=$1 327 TBID=$2 328 329 # create VRF device 330 ip link add ${VRF} type vrf table ${TBID} 331 332 if [ "${VRF}" != "mgmt" ]; then 333 ip route add table ${TBID} unreachable default 334 fi 335 ip link set dev ${VRF} up 336} 337 338vrf_create mgmt 1 339ip link set dev eth0 master mgmt 340 341vrf_create red 10 342ip link set dev eth1 master red 343ip link set dev eth2 master red 344ip link set dev eth5 master red 345 346vrf_create blue 66 347ip link set dev eth3 master blue 348 349vrf_create green 81 350ip link set dev eth4 master green 351 352 353Interface addresses from /etc/network/interfaces: 354auto eth0 355iface eth0 inet static 356 address 10.0.0.2 357 netmask 255.255.255.0 358 gateway 10.0.0.254 359 360iface eth0 inet6 static 361 address 2000:1::2 362 netmask 120 363 364auto eth1 365iface eth1 inet static 366 address 10.2.1.2 367 netmask 255.255.255.0 368 369iface eth1 inet6 static 370 address 2002:1::2 371 netmask 120 372 373auto eth2 374iface eth2 inet static 375 address 10.2.2.2 376 netmask 255.255.255.0 377 378iface eth2 inet6 static 379 address 2002:2::2 380 netmask 120 381 382auto eth3 383iface eth3 inet static 384 address 10.2.3.2 385 netmask 255.255.255.0 386 387iface eth3 inet6 static 388 address 2002:3::2 389 netmask 120 390 391auto eth4 392iface eth4 inet static 393 address 10.2.4.2 394 netmask 255.255.255.0 395 396iface eth4 inet6 static 397 address 2002:4::2 398 netmask 120 399