VXLAN & Linux

Vincent Bernat

VXLAN is an over­lay net­work to carry Eth­er­net traf­fic over an ex­ist­ing (highly avail­able and scal­able) IP net­work while ac­com­mo­dat­ing a very large num­ber of ten­ants. It is de­fined in RFC 7348.

Start­ing from Linux 3.12, the VXLAN im­ple­men­ta­tion is quite com­plete as both mul­ti­cast and uni­cast are sup­ported as well as IPv6 and IPv4. Let’s ex­plore the var­i­ous meth­ods to con­figure it.

VXLAN setup
VXLAN deployment example

To il­lus­trate our ex­am­ples, we use the fol­low­ing setup:

  • an un­der­lay IP net­work (highly avail­able and scal­able, pos­si­bly the In­ternet);
  • three Linux bridges act­ing as VXLAN tun­nel end­points (VTEP); and
  • four servers be­liev­ing they share a com­mon Eth­er­net segment.

A VXLAN tun­nel ex­tends the in­di­vid­ual Eth­er­net seg­ments across the three bridges, pro­vid­ing a unique (vir­tual) Eth­er­net seg­ment. From one host (e.g. H1), we can reach di­rectly all the other hosts in the vir­tual segment:

$ ping -c10 -w1 -t1 ff02::1%eth0
PING ff02::1%eth0(ff02::1%eth0) 56 data bytes
64 bytes from fe80::5254:33ff:fe00:8%eth0: icmp_seq=1 ttl=64 time=0.016 ms
64 bytes from fe80::5254:33ff:fe00:b%eth0: icmp_seq=1 ttl=64 time=4.98 ms (DUP!)
64 bytes from fe80::5254:33ff:fe00:9%eth0: icmp_seq=1 ttl=64 time=4.99 ms (DUP!)
64 bytes from fe80::5254:33ff:fe00:a%eth0: icmp_seq=1 ttl=64 time=4.99 ms (DUP!)

--- ff02::1%eth0 ping statistics ---
1 packets transmitted, 1 received, +3 duplicates, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.016/3.745/4.991/2.152 ms

Basic usage#

The ref­er­ence de­ploy­ment for VXLAN is to use an IP mul­ti­cast group to join the other VTEPs:

# ip -6 link add vxlan100 type vxlan \
>   id 100 \
>   dstport 4789 \
>   local 2001:db8:1::1 \
>   group ff05::100 \
>   dev eth0 \
>   ttl 5
# brctl addbr br100
# brctl addif br100 vxlan100
# brctl addif br100 vnet22
# brctl addif br100 vnet25
# brctl stp br100 off
# ip link set up dev br100
# ip link set up dev vxlan100

The above com­mands cre­ate a new in­ter­face act­ing as a VXLAN tun­nel end­point, named vxlan100, and put it in a bridge with some reg­u­lar in­ter­faces.1 Each VXLAN seg­ment is as­so­ci­ated with a 24-bit seg­ment ID, the VXLAN Net­work Iden­ti­fier (VNI). In our ex­am­ple, the de­fault VNI is spec­i­fied with id 100.

When VXLAN was first im­ple­mented in Linux 3.7, the UDP port to use was not de­fined. Sev­eral ven­dors were us­ing 8472 and Linux took the same value. To avoid break­ing ex­ist­ing de­ploy­ments, this is still the de­fault value. There­fore, if you want to use the IANA-as­signed port, you need to ex­plic­itly set it with dstport 4789.

As we want to use mul­ti­cast, we have to spec­ify a mul­ti­cast group to join (group ff05::​100), as well as a phys­i­cal de­vice (dev eth0). With mul­ti­cast, the de­fault TTL is 1. If your mul­ti­cast net­work lever­ages some rout­ing, you’ll have to in­crease the value a bit, like here with ttl 5.

The vxlan100 de­vice acts as a bridge de­vice with re­mote VTEPs as vir­tual ports:

  • it sends broad­cast, un­known uni­cast, and mul­ti­cast (BUM) frames to all VTEPs us­ing the mul­ti­cast group; and
  • it dis­cov­ers the as­so­ci­a­tion from Eth­er­net MAC ad­dresses to VTEP IP ad­dresses us­ing source-ad­dress learning.

The fol­low­ing fig­ure sum­ma­rizes the con­fig­u­ra­tion, with the FDB of the Linux bridge (learn­ing lo­cal MAC ad­dresses) and the FDB of the VXLAN de­vice (learn­ing dis­tant MAC ad­dresses):

Bridged VXLAN device
VXLAN device attached to a Linux bridge and communicating with two remote VTEPs

The FDB of the VXLAN de­vice can be ob­served with the bridge com­mand. If the des­ti­na­tion MAC is present, the frame is sent to the as­so­ci­ated VTEP (uni­cast). The all-zero ad­dress is only used when a lookup for the des­ti­na­tion MAC fails.

# bridge fdb show dev vxlan100 | grep dst
00:00:00:00:00:00 dst ff05::100 via eth0 self permanent
50:54:33:00:00:0b dst 2001:db8:3::1 self
50:54:33:00:00:08 dst 2001:db8:1::1 self

If you are in­ter­ested to get more de­tails on how to set up a mul­ti­cast net­work and build VXLAN seg­ments on top of it, see my “Net­work vir­tu­al­iza­tion with VXLAN” article.

Without multicast#

Us­ing VXLAN over a mul­ti­cast IP net­work has sev­eral benefits:

  • au­to­matic dis­cov­ery of other VTEPs shar­ing the same mul­ti­cast group
  • good band­width us­age (pack­ets are repli­cated as late as possible)
  • de­cen­tral­ized and con­troller-less design2

How­ever, mul­ti­cast is not avail­able ev­ery­where and man­ag­ing it at scale can be dif­fi­cult. In Linux 3.8, the DOVE ex­ten­sions have been added to the VXLAN im­ple­men­ta­tion, re­mov­ing the de­pen­dency on multicast.

Unicast with static flooding#

We can re­place mul­ti­cast by head-end repli­ca­tion of BUM frames to a stat­i­cally con­fig­ured list of re­mote VTEPs:3

# ip -6 link add vxlan100 type vxlan \
>   id 100 \
>   dstport 4789 \
>   local 2001:db8:1::1
# bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:2::1
# bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:3::1

The VXLAN is de­fined with­out a re­mote mul­ti­cast group. In­stead, all the re­mote VTEPs are as­so­ci­ated with the all-zero ad­dress: a BUM frame will be du­pli­cated to all these des­ti­na­tions. The VXLAN de­vice will still learn re­mote ad­dresses au­to­mat­i­cally us­ing source-ad­dress learning.

It is a very sim­ple so­lu­tion. With a bit of au­to­ma­tion, you can keep the de­fault FDB en­tries up-to-date eas­ily. How­ever, the host will have to du­pli­cate each BUM frame (head-end repli­ca­tion) as many times as there are re­mote VTEPs. This is quite rea­son­able if you have a dozen of them. This may be­come out-of-hand if you have thou­sands of them.

Cu­mu­lus vxfld dae­mon is an ex­am­ple of this strat­egy (in the head-end repli­ca­tion mode). “Keepalived and uni­cast over mul­ti­ple in­ter­faces” shows an­other usage.

Unicast with static L2 entries#

When the as­so­ci­a­tions of MAC ad­dresses and VTEPs are known, it is pos­si­ble to pre-pop­u­late the FDB and dis­able learning:

# ip -6 link add vxlan100 type vxlan \
>   id 100 \
>   dstport 4789 \
>   local 2001:db8:1::1 \
>   nolearning
# bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:2::1
# bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:3::1
# bridge fdb append 50:54:33:00:00:09 dev vxlan100 dst 2001:db8:2::1
# bridge fdb append 50:54:33:00:00:0a dev vxlan100 dst 2001:db8:2::1
# bridge fdb append 50:54:33:00:00:0b dev vxlan100 dst 2001:db8:3::1

Thanks to the nolearning flag, source-ad­dress learn­ing is dis­abled. There­fore, if a MAC is miss­ing, the frame will al­ways be sent us­ing the all-zero entries.

The all-zero en­tries are still needed for broad­cast and mul­ti­cast traf­fic (e.g. ARP and IPv6 neigh­bor dis­cov­ery). This kind of setup works well to pro­vide vir­tual L2 net­works to vir­tual ma­chines (no L3 in­for­ma­tion avail­able). You need some glue to up­date the FDB entries.

BGP EVPN with FRR is an ex­am­ple of this strat­egy (see “VXLAN: BGP EVPN with FRR” for ad­di­tional in­for­mation).

Unicast with static L3 entries#

In the pre­vi­ous ex­am­ple, we had to keep the all-zero en­tries for ARP and IPv6 neigh­bor dis­cov­ery to work cor­rectly. How­ever, Linux can an­swer to neigh­bor re­quests on be­half of the re­mote nodes.4 When this fea­ture is en­abled, the de­fault en­tries are not needed any­more (but you could keep them):

# ip -6 link add vxlan100 type vxlan \
>   id 100 \
>   dstport 4789 \
>   local 2001:db8:1::1 \
>   nolearning \
>   proxy
# ip -6 neigh add 2001:db8:ff::11 lladdr 50:54:33:00:00:09 dev vxlan100
# ip -6 neigh add 2001:db8:ff::12 lladdr 50:54:33:00:00:0a dev vxlan100
# ip -6 neigh add 2001:db8:ff::13 lladdr 50:54:33:00:00:0b dev vxlan100
# bridge fdb append 50:54:33:00:00:09 dev vxlan100 dst 2001:db8:2::1
# bridge fdb append 50:54:33:00:00:0a dev vxlan100 dst 2001:db8:2::1
# bridge fdb append 50:54:33:00:00:0b dev vxlan100 dst 2001:db8:3::1

This setup elim­i­nates head-end repli­ca­tion. How­ever, pro­to­cols re­ly­ing on mul­ti­cast won’t work ei­ther. With some au­to­ma­tion, this is a setup that should work well with con­tain­ers: if there is a reg­istry keep­ing a list of all IP and MAC ad­dresses in use, a pro­gram could lis­ten to it and ad­just the FDB and the neigh­bor tables.

The VXLAN back­end of Docker’s lib­net­work is an ex­am­ple of this strat­egy (but it also uses the next method).

Unicast with dynamic L3 entries#

Linux can also no­tify a pro­gram that an (L2 or L3) en­try is miss­ing. The pro­gram queries some cen­tral reg­istry and dy­nam­i­cally adds the re­quested en­try. How­ever, for L2 en­tries, no­ti­fi­ca­tions are is­sued only if:

  • the des­ti­na­tion MAC ad­dress is not known;
  • there is no all-zero en­try in the FDB; and
  • the des­ti­na­tion MAC ad­dress is not a mul­ti­cast or broad­cast one.

These lim­i­ta­tions pre­vent us from do­ing a “uni­cast with dy­namic L2 en­tries” scenario.

First, let’s cre­ate the VXLAN de­vice with the l2miss and l3miss options:5

ip -6 link add vxlan100 type vxlan \
   id 100 \
   dstport 4789 \
   local 2001:db8:1::1 \
   nolearning \
   l2miss \
   l3miss \
   proxy

No­ti­fi­ca­tions are sent to pro­grams lis­ten­ing to an AF_​NETLINK socket us­ing the NETLINK_​ROUTE pro­to­col. This socket needs to be bound to the RTNLGRP_​NEIGH group. The fol­low­ing is do­ing ex­actly that and de­codes the re­ceived no­ti­fi­cations:

# ip monitor neigh dev vxlan100
miss 2001:db8:ff::12 STALE
miss lladdr 50:54:33:00:00:0a STALE

The first no­ti­fi­ca­tion is about a miss­ing neigh­bor en­try for the re­quested IP ad­dress. We can add it with the fol­low­ing command:

ip -6 neigh replace 2001:db8:ff::12 \
    lladdr 50:54:33:00:00:0a \
    dev vxlan100 \
    nud reachable

The en­try is not per­ma­nent so that we don’t need to delete it when it ex­pires. If the ad­dress be­comes stale, we will get an­other no­ti­fi­ca­tion to re­fresh it.

Once the host re­ceives our proxy an­swer for the neigh­bor dis­cov­ery re­quest, it can send a frame with the MAC we gave as a des­ti­na­tion. The sec­ond no­ti­fi­ca­tion is about the miss­ing FDB en­try for this MAC ad­dress. We add the ap­pro­pri­ate en­try with the fol­low­ing command:6

bridge fdb replace 50:54:33:00:00:0a \
    dst 2001:db8:2::1 \
    dev vxlan100 dynamic

The en­try is not per­ma­nent ei­ther as it would pre­vent the MAC to mi­grate to the lo­cal VTEP (a dy­namic en­try can­not over­ride a per­ma­nent entry).

This setup works well with con­tain­ers and a global reg­istry. How­ever, there is a small la­tency penalty for the first con­nec­tions. More­over, mul­ti­cast and broad­cast won’t be avail­able in the un­der­lay net­work. The VXLAN back­end for flan­nel, a net­work fab­ric for Ku­ber­netes, is an ex­am­ple of this strategy.

Decision#

There is no one-size-fits-all solution.

You should con­sider the mul­ti­cast so­lution if:

  • you are in an en­vi­ron­ment where mul­ti­cast is available;
  • you are ready to op­er­ate (and scale) a mul­ti­cast network;
  • you need mul­ti­cast and broad­cast in­side the vir­tual seg­ments; and
  • you don’t have L2/L3 ad­dresses avail­able be­forehand.

The scal­a­bil­ity of such a so­lu­tion is pretty good if you take care of not putting all VXLAN in­ter­faces into the same mul­ti­cast group (e.g. use the last byte of the VNI as the last byte of the mul­ti­cast group).

When mul­ti­cast is not avail­able, an­other generic so­lu­tion is BGP EVPN: BGP is used as a con­troller to en­sure the dis­tri­bu­tion of the list of VTEPs and their re­spec­tive FDBs. As men­tioned ear­lier, an im­ple­men­ta­tion of this so­lu­tion is FRR. I ex­plore this op­tion in a sep­a­rate post: VXLAN: BGP EVPN with FRR.

If you op­er­ate in a con­tainer-like en­vi­ron­ment where L2/L3 ad­dresses are known be­fore­hand, a so­lu­tion us­ing static and/or dy­namic L2 and L3 en­tries based on a cen­tral reg­istry and no source-ad­dress learn­ing would also fit the bill. This pro­vides a more se­cu­rity-tight so­lu­tion (bound re­sources, MiTM at­tacks damp­ened down, in­abil­ity to am­plify band­width us­age through ex­ces­sive broad­cast). Var­i­ous en­vi­ron­ment-spe­cific so­lu­tions are avail­able7 or you can build your own.

Other considerations#

In­de­pen­dently of the cho­sen strat­egy, here are a few im­por­tant points to keep in mind when im­ple­ment­ing a VXLAN overlay.

Isolation#

While you may ex­pect VXLAN in­ter­faces to only carry L2 traf­fic, Linux doesn’t dis­able IP pro­cess­ing. If the des­ti­na­tion MAC is a lo­cal one, Linux will route or de­liver the en­cap­su­lated IP packet. Check my post about the proper iso­la­tion of a Linux bridge.

Encryption#

VXLAN en­forces iso­la­tion be­tween ten­ants, but the traf­fic is un­en­crypted. The most di­rect so­lu­tion to pro­vide en­cryp­tion is to use IPsec. Some con­tainer-based so­lu­tions may come with IPsec sup­port out-of-the-box (no­tably Docker’s lib­net­work, but flan­nel has a plan for it too). This is quite im­por­tant for de­ploy­ment over a pub­lic cloud.

Overhead#

The for­mat of a VXLAN-en­cap­su­lated frame is the following:

VXLAN encapsulation
VXLAN encapsulation inside a UDP datagram (additional overhead: 50 bytes)

VXLAN adds a fixed over­head of 50 bytes. If you also use IPsec, the over­head de­pends on many fac­tors. In trans­port mode, with AES and SHA256, the over­head is 56 bytes. With NAT tra­ver­sal, this is 64 bytes (ad­di­tional UDP header). In tun­nel mode, this is 72 bytes.

Some users will ex­pect to be able to use an Eth­er­net MTU of 1500 for the over­lay net­work. There­fore, the un­der­lay MTU should be in­creased. If it is not pos­si­ble, en­sure the in­ner MTU (in­side the con­tain­ers or the vir­tual ma­chines) is cor­rectly de­creased.8

IPv6#

While all the ex­am­ples above are us­ing IPv6, the ecosys­tem is not quite ready yet. The mul­ti­cast L2-only strat­egy works fine with IPv6 but ev­ery other sce­nario cur­rently needs some patches (1, 2, 3, 4).

Up­date (2020-11)

From Linux 4.13, all the men­tioned is­sues are solved. IPv6 is also more well-tested as Cu­mu­lus pro­motes IPv6 as the trans­port and sig­nal­ing pro­to­col for their own so­lutions.

On top of that, IPv6 may not have been im­ple­mented in VXLAN-re­lated tools:

Multicast#

Linux VXLAN im­ple­men­ta­tion doesn’t sup­port IGMP snoop­ing. Mul­ti­cast traf­fic will be broad­casted to all VTEPs un­less mul­ti­cast MAC ad­dresses are in­serted into the FDB.