Network virtualization with VXLAN

Vincent Bernat

Vir­tual eXtensible Lo­cal Area Net­work (VXLAN) is a pro­to­col to over­lay a vir­tu­al­ized L2 net­work over an ex­ist­ing IP net­work with lit­tle setup. It is cur­rently de­scribed in an In­ter­net-Draft. It adds the fol­low­ing perks to VLANs while still pro­vid­ing isolation:

  1. It uses a 24-bit VXLAN Net­work Iden­ti­fier (VNI) which should be enough to ad­dress any scale-based con­cerns of mul­ti­tenancy.
  2. It wraps L2 frames into UDP data­grams. This al­lows one to rely on some in­ter­est­ing prop­er­ties of IP net­works like avail­abil­ity and scal­a­bil­ity. A VXLAN seg­ment can be ex­tended far be­yond the typ­i­cal reach of to­day’s VLANs.

The VXLAN Tun­nel End Point (VTEP) orig­i­nates and ter­mi­nates VXLAN tun­nels. Thanks to a se­ries of patches from Stephen Hem­minger, Linux can now act as a VTEP (Linux 3.7). Let’s see how this works.

Up­date (2017-05)

The im­ple­men­ta­tion ex­posed in this post heav­ily re­lies on mul­ti­cast. A fol­lowup ex­plor­ing the use of uni­cast is avail­able, as well as an­other one about BGP EVPN.

Up­date (2018-10)

In Au­gust 2014, the In­ter­net-Draft has been pub­lished as RFC 7348.

About IPv6#

When pos­si­ble, I try to use IPv6 for my labs. This is not the case here for sev­eral reasons:

  1. IP mul­ti­cast is re­quired and PIM-SM im­ple­men­ta­tions for IPv6 are not wide­spread yet. How­ever, they ex­ist. This ex­plains why I use XORP for this lab: it sup­ports PIM-SM for both IPv4 and IPv6.
  2. VXLAN In­ter­net-Draft specif­i­cally ad­dresses only IPv4. This seems a bit odd for a pro­to­col run­ning on top of UDP and I hope this will be fixed soon. This is not a ma­jor stop­per since some VXLAN im­ple­men­ta­tions sup­port IPv6.
  3. How­ever, the cur­rent im­ple­men­ta­tion for Linux does not sup­port IPv6. IPv6 sup­port will be added later.

Once IPv6 sup­port is avail­able, the lab should be easy to adapt.

Up­date (2017-01)

The lat­est draft ad­dresses IPv6 sup­port. It is avail­able in Linux 3.12. VXLAN is much im­proved with Linux 3.12: DOVE ex­ten­sions sup­port (3.8), im­proved off­load sup­port (3.8+), uni­cast sup­port (3.10), and IPv6 sup­port (3.12).

Lab#

So, here is the lab used. R1, R2 and R3 will act as VTEPs. They do not make use of PIM-SM. In­stead, they have a generic mul­ti­cast route on eth0. E1, E2 and E3 are edge routers while C1, C2 and C3 are core routers. The pro­posed lab is not re­silient but con­ve­nient to ex­plain how things work. It is built on top of QEMU hosts. Have a look at my pre­vi­ous ar­ti­cle for more de­tails on this.

VXLAN lab
Topology of VXLAN lab

The lab is hosted on GitHub. I have made the lab eas­ier to try by in­clud­ing the ker­nel I have used for my tests. XORP comes pre­con­fig­ured, you just have to con­fig­ure the VXLAN part. For this, you need a re­cent ver­sion of ip.

$ sudo apt-get install screen vde2 qemu-system-x86 iproute xorp git
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git
$ cd iproute2
$ ./configure && make
You get `ip' as `ip/ip' and `bridge' as `bridge/bridge'.
$ cd ..
$ git clone git://github.com/vincentbernat/network-lab.git
$ cd network-lab/lab-multicast-vxlan
$ ./setup

Unicast routing#

The first step is to setup uni­cast rout­ing. OSPF is used for this pur­pose. The cho­sen rout­ing dae­mon is XORP. With xorpsh, we can check if OSPF is work­ing as expected:

root@c1# xorpsh
root@c1$ show ospf4 neighbor
  Address         Interface             State      ID              Pri  Dead
192.168.11.11    eth0/eth0              Full      3.0.0.1          128    36
192.168.12.22    eth1/eth1              Full      3.0.0.2          128    33
192.168.101.133  eth2/eth2              Full      2.0.0.3          128    36
192.168.102.122  eth3/eth3              Full      2.0.0.2          128    38
root@c1$ show route table ipv4 unicast ospf
192.168.1.0/24  [ospf(110)/2]
                > to 192.168.11.11 via eth0/eth0
192.168.2.0/24  [ospf(110)/2]
                > to 192.168.12.22 via eth1/eth1
192.168.3.0/24  [ospf(110)/3]
                > to 192.168.102.122 via eth3/eth3
192.168.13.0/24 [ospf(110)/2]
                > to 192.168.102.122 via eth3/eth3
192.168.21.0/24 [ospf(110)/2]
                > to 192.168.101.133 via eth2/eth2
192.168.22.0/24 [ospf(110)/2]
                > to 192.168.12.22 via eth1/eth1
192.168.23.0/24 [ospf(110)/2]
                > to 192.168.101.133 via eth2/eth2
192.168.103.0/24        [ospf(110)/2]
                > to 192.168.102.122 via eth3/eth3

Multicast routing#

Once uni­cast rout­ing is up and run­ning, we need to setup mul­ti­cast rout­ing. There are two pro­to­cols for this: IGMP and PIM-SM. The for­mer one al­lows hosts to sub­scribe to a mul­ti­cast group while the lat­ter one en­ables routers to dis­trib­ute mul­ti­cast routes.

IGMP#

IGMP is used by hosts and ad­ja­cent routers to es­tab­lish mul­ti­cast group mem­ber­ship. In our case, it will be used by R2 to let E2 know it sub­scribed to 239.0.0.11 (a mul­ti­cast group).

Con­fig­ur­ing XORP to sup­port IGMP is sim­ple. Let’s test with iperf to have a mul­ti­cast lis­tener on R2:

root@r2# iperf -u -s -l 1000 -i 1 -B 239.0.0.11
Server listening on UDP port 5001
Binding to local address 239.0.0.11
Joining multicast group  239.0.0.11
Receiving 1000 byte datagrams
UDP buffer size:  208 KByte (default)

On E2, we can now check that R2 is prop­erly reg­is­tered for 239.0.0.11:

root@e2$ show igmp group
Interface    Group           Source          LastReported Timeout V State
eth0         239.0.0.11      0.0.0.0         192.168.2.2      248 2     E

XORP doc­u­men­ta­tion con­tains a good over­view of IGMP.

PIM-SM#

PIM-SM is far more com­plex. It does not have its own topol­ogy dis­cov­ery pro­to­col and re­lies on rout­ing in­for­ma­tion from other pro­to­cols, OSPF in our case.

I will de­scribe here a sim­pli­fied view on how PIM-SM works. XORP doc­u­men­ta­tion con­tains more de­tails about PIM-SM.

The first step for all PIM-SM routers is to elect a ren­dez-vous point (RP). In our lab, only C1, C2 and C3 have been con­fig­ured to be elected as a RP. More­over, we give bet­ter pri­or­ity to C3 to en­sure it wins.

RP election
C3 has been elected as RP
root@e1$ show pim rps
RP              Type      Pri Holdtime Timeout ActiveGroups GroupPrefix
192.168.101.133 bootstrap 100      150     135            0 239.0.0.0/8

Sup­pose we start iperf on both R2 and R3. Us­ing IGMP, they sub­scribe to mul­ti­cast group 239.0.0.11 with E2 and E3 re­spec­tively. Then, E2 and E3 send a join mes­sage (also known as a (*,G) join) to the RP (C3) for that mul­ti­cast group. Us­ing the uni­cast path from E2 and E3 to the RP, the routers along the paths build the RP tree (RPT), rooted at C3. Each router in the tree knows how to send mul­ti­cast pack­ets to 239.0.0.11: it will send them to the leaves.

RP tree
RP tree for 239.0.0.11 has been built
root@e3$ show pim join
Group           Source          RP              Flags
239.0.0.11      0.0.0.0         192.168.101.133 WC
    Upstream interface (RP):   eth2
    Upstream MRIB next hop (RP): 192.168.23.133
    Upstream RPF'(*,G):        192.168.23.133
    Upstream state:            Joined
    Join timer:                5
    Local receiver include WC: O...
    Joins RP:                  ....
    Joins WC:                  ....
    Join state:                ....
    Prune state:               ....
    Prune pending state:       ....
    I am assert winner state:  ....
    I am assert loser state:   ....
    Assert winner WC:          ....
    Assert lost WC:            ....
    Assert tracking WC:        O.O.
    Could assert WC:           O...
    I am DR:                   O..O
    Immediate olist RP:        ....
    Immediate olist WC:        O...
    Inherited olist SG:        O...
    Inherited olist SG_RPT:    O...
    PIM include WC:            O...

Sup­pose that R1 wants to send mul­ti­cast pack­ets to 239.0.0.11. It sends them to E1 which does not have any in­for­ma­tion on how to con­tact all the mem­bers of the mul­ti­cast group be­cause it is not the RP. There­fore, it en­cap­su­lates the mul­ti­cast pack­ets into PIM Reg­is­ter pack­ets and sends them to the RP. The RP de­cap­su­lates them and sends them na­tively. The mul­ti­cast pack­ets are routed from the RP to R2 and R3 us­ing the re­verse path formed by the join messages.

Multicast routing via the RP
R1 sends multicast packets to 239.0.0.11 via the RP
root@r1# iperf -c 239.0.0.11 -u -b 10k -t 30 -T 10
Client connecting to 239.0.0.11, UDP port 5001
Sending 1470 byte datagrams
Setting multicast TTL to 10
UDP buffer size:  208 KByte (default)
root@e1# tcpdump -pni eth0
10:58:23.424860 IP 192.168.1.1.35277 > 239.0.0.11.5001: UDP, length 1470
root@c3# tcpdump -pni eth0
10:58:23.552903 IP 192.168.11.11 > 192.168.101.133: PIMv2, Register, length 1480
root@e2# tcpdump -pni eth0
10:58:23.896171 IP 192.168.1.1.35277 > 239.0.0.11.5001: UDP, length 1470
root@e3# tcpdump -pni eth0
10:58:23.824647 IP 192.168.1.1.35277 > 239.0.0.11.5001: UDP, length 1470

As pre­sented here, the rout­ing is not op­ti­mal: pack­ets from R1 to R2 could avoid the RP. More­over, en­cap­su­lat­ing mul­ti­cast pack­ets into uni­cast pack­ets is not ef­fi­cient ei­ther. At some point, the RP will de­cide to switch to na­tive mul­ti­cast.1 Rooted at R1, the short­est-path tree (SPT) for the mul­ti­cast group will be built us­ing source-spe­cific join mes­sages (also known as a (S,G) join).

Multicast routing without RP
R1 sends multicast packets to 239.0.0.11 using native multicast following the shortest-path tree

From here, each router in the tree knows how to han­dle mul­ti­cast pack­ets from R1 to the group with­out in­volv­ing the RP. For ex­am­ple, E1 knows it must du­pli­cate the packet and send one through the in­ter­face to C3 and the other one through the in­ter­face to C1:

root@e1$ show pim join
Group           Source          RP              Flags
239.0.0.11      192.168.1.1     192.168.101.133 SG SPT DirectlyConnectedS
    Upstream interface (S):    eth0
    Upstream interface (RP):   eth1
    Upstream MRIB next hop (RP): 192.168.11.111
    Upstream MRIB next hop (S):  UNKNOWN
    Upstream RPF'(S,G):        UNKNOWN
    Upstream state:            Joined
    Register state:            RegisterPrune RegisterCouldRegister
    Join timer:                7
    KAT(S,G) running:          true
    Local receiver include WC: ....
    Local receiver include SG: ....
    Local receiver exclude SG: ....
    Joins RP:                  ....
    Joins WC:                  ....
    Joins SG:                  .OO.
    Join state:                .OO.
    Prune state:               ....
    Prune pending state:       ....
    I am assert winner state:  ....
    I am assert loser state:   ....
    Assert winner WC:          ....
    Assert winner SG:          ....
    Assert lost WC:            ....
    Assert lost SG:            ....
    Assert lost SG_RPT:        ....
    Assert tracking SG:        OOO.
    Could assert WC:           ....
    Could assert SG:           .OO.
    I am DR:                   O..O
    Immediate olist RP:        ....
    Immediate olist WC:        ....
    Immediate olist SG:        .OO.
    Inherited olist SG:        .OO.
    Inherited olist SG_RPT:    ....
    PIM include WC:            ....
    PIM include SG:            ....
    PIM exclude SG:            ....
root@e1$ show pim mfc
Group           Source          RP
239.0.0.11      192.168.1.1     192.168.101.133
    Incoming interface :      eth0
    Outgoing interfaces:      .OO.
root@e1$ exit
[Connection to XORP closed]
root@e1# ip mroute
(192.168.1.1, 239.0.0.11)        Iif: eth0       Oifs: eth1 eth2

Setting up VXLAN#

Once IP mul­ti­cast is run­ning, set­ting up VXLAN is quite easy. Here are the soft­ware re­quirements:

  • A re­cent ker­nel. Pick at least 3.7-rc3. You need to en­able CONFIG_​VXLAN op­tion. You also cur­rently need a patch on top of it to be able to spec­ify a TTL greater than 1 for mul­ti­cast packets.
  • A re­cent ver­sion of ip. Cur­rently, you need the ver­sion from git.

On R1, R2 and R3, we cre­ate a vxlan42 in­ter­face with the fol­low­ing commands:

root@rX# ./ip link add vxlan42 type vxlan id 42 \
>           group 239.0.0.42 \
>           ttl 10 dev eth0
root@rX# ip link set up dev vxlan42
root@rX# ./ip -d link show vxlan42
10: vxlan42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether 3e:09:1c:e1:09:2e brd ff:ff:ff:ff:ff:ff
vxlan id 42 group 239.0.0.42 dev eth0 port 32768 61000 ttl 10 ageing 300

Let’s as­sign an IP in 192.168.99.0/24 for each router and check they can ping each other:

root@r1# ip addr add 192.168.99.1/24 dev vxlan42
root@r2# ip addr add 192.168.99.2/24 dev vxlan42
root@r3# ip addr add 192.168.99.3/24 dev vxlan42
root@r1# ping 192.168.99.2
PING 192.168.99.2 (192.168.99.2) 56(84) bytes of data.
64 bytes from 192.168.99.2: icmp_req=1 ttl=64 time=3.90 ms
64 bytes from 192.168.99.2: icmp_req=2 ttl=64 time=1.38 ms
64 bytes from 192.168.99.2: icmp_req=3 ttl=64 time=1.82 ms

--- 192.168.99.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.389/2.375/3.907/1.098 ms

We can check the pack­ets are en­cap­sulated:

root@r1# tcpdump -pni eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:30:36.561185 IP 192.168.1.1.43349 > 192.168.2.2.8472: UDP, length 106
11:30:36.563179 IP 192.168.2.2.33894 > 192.168.1.1.8472: UDP, length 106
11:30:37.562677 IP 192.168.1.1.43349 > 192.168.2.2.8472: UDP, length 106
11:30:37.564316 IP 192.168.2.2.33894 > 192.168.1.1.8472: UDP, length 106

More­over, if we send broad­cast pack­ets (with ping -b or ARP re­quests), they are en­cap­su­lated into mul­ti­cast packets:

root@r1# tcpdump -pni eth0
11:31:27.464198 IP 192.168.1.1.41958 > 239.0.0.42.8472: UDP, length 106
11:31:28.463584 IP 192.168.1.1.41958 > 239.0.0.42.8472: UDP, length 106

Re­cent ver­sions of iproute also come with bridge, a util­ity al­low­ing one to in­spect the FDB of bridge-like in­terfaces:

root@r1# ../bridge/bridge fdb show vxlan42
3e:09:1c:e1:09:2e dev vxlan42 dst 192.168.2.2 self
0e:98:40:c6:58:10 dev vxlan42 dst 192.168.3.3 self

Demo#

For a demo, have a look at the fol­low­ing video: