Anycast DNS

Vincent Bernat

In a re­cently pub­lished ar­ti­cle, Paul Vixie, past au­thor and ar­chi­tect of BIND, one of the most pop­u­lar in­ter­net do­main servers, ex­plains why DNS servers should use any­cast for in­creased re­li­a­bil­ity and faster an­swers. The main ar­gu­ment against uni­cast is that some re­solvers may not be smart enough to se­lect the best server to query and there­fore, they may se­lect a server on the other side of the planet. Since you can­not go faster than light, you get a penalty of about 100 mil­lisec­onds in this case.

Unicast & anycast#

Most ad­dresses that you han­dle are uni­cast ad­dresses, i.e. the ad­dress of a sin­gle re­ceiver. Any time you send a packet to a uni­cast ad­dress, it will be for­warded to this sin­gle re­ceiver. An­other com­mon net­work ad­dress­ing sys­tem is mul­ti­cast (which hap­pens to in­clude broad­cast). When you send a packet to a mul­ti­cast ad­dress, sev­eral re­ceivers may get it. The packet is copied as needed by some nodes.

Any­cast ad­dresses al­low you to send a packet to a set of re­ceivers but only one of them will re­ceive it. The packet will not be copied. Usu­ally, the idea be­hind any­cast is that the packet is sent to the near­est receiver.

There is no uni­ver­sal way to de­ter­mine if an IP ad­dress is a uni­cast ad­dress or an any­cast ad­dress just by look­ing at it. For ex­am­ple 192.5.5.241 is an any­cast ad­dress while 78.47.78.132 is a uni­cast address.

ISC, which main­tains BIND, op­er­ates the “F” root do­main server, one of the 13 In­ter­net root name servers. This server is queried by us­ing 192.5.5.241 IPv4 any­cast ad­dress or 2001:500:2f::f IPv6 any­cast ad­dress. ISC has pub­lished a doc­u­ment ex­plain­ing how any­cast works. Sev­eral nodes around the globe an­nounce the same sub­net us­ing BGP. When one router run­ning BGP has to for­ward a query to the any­cast ad­dress, it will have sev­eral choices in its rout­ing ta­ble. It will usu­ally se­lect the short­est path and route the query to the cor­re­spond­ing router.

Some nodes may choose to re­strict the an­nounce­ment about the sub­net con­tain­ing the any­cast ad­dress. They are called lo­cal nodes (by op­po­si­tion to other global nodes). This al­lows them to serve queries only from their clients and use cheaper con­nec­tiv­ity and hardware.

Lab#

Let’s play with any­cast by set­ting up a lab to ex­per­i­ment with it. We want to set up an IPv6 any­cast DNS ser­vice us­ing two global nodes, G1 and G2, and one lo­cal node, L1. We build some kind of mini-In­ter­net. See:

Anycast lab
Topology for anycast DNS lab

Setting up the lab#

This lab will use UML guests for all nodes. For more de­tails on how this works, look at my pre­vi­ous ar­ti­cle about net­work labs with User Mode Linux. You first need to grab the lab:

$ git clone https://github.com/vincentbernat/network-lab.git
$ cd network-lab/lab-anycast-dns
$ ./setup

You will need to in­stall the de­pen­den­cies needed by the script on your host sys­tem. The script was tested on De­bian un­sta­ble. If some of the de­pen­den­cies do not ex­ist in your dis­tri­bu­tion, if they do not be­have as in De­bian or if you do not want to in­stall them on your host sys­tem, you can use debootstrap to cre­ate a work­ing system:

$ sudo debootstrap sid ./sid-chroot http://ftp.debian.org/debian/
[…]
$ sudo chroot ./sid-chroot /bin/bash
# apt-get install iproute zsh aufs-tools
[…]
# apt-get install bird6
[…]
# exit
$ sudo mkdir -p ./sid-chroot/$HOME
$ sudo chown $(whoami) ./sid-chroot/$HOME
$ ROOT=./sid-chroot ./setup

We are run­ning a lot of UML guests in this lab. We as­sign 64 MiB to each host. UML guests do not use host mem­ory un­til they need it but be­cause of cache sub­sys­tem, you can ex­pect that each guest will use 64 MiB of mem­ory. This means that you need about 2 GiB of RAM to run this lab. More­over, each guest will cre­ate a file rep­re­sent­ing its mem­ory. There­fore, you need about 2 GiB of free space in /tmp too.

Routers are named af­ter their AS num­ber. The router of AS 64652 is just 64652. Paris, NewYork and Tokyo are the ex­ception.

The lab will use only one switch but we will use VLAN to cre­ate sev­eral L2 net­works. Each link in our schema is an L2 net­work. Each link will get its own L3 net­work. These net­works are pre­fixed with 2001:db8:ffff:. Leaf AS have their own L3 net­work 2001:db8:X::/48 where X is the AS num­ber mi­nus 60000. Look at /etc/hosts on each host to get the IP of all hosts.

Mini-Internet routing#

AS 64600 is some kind of Tier-1 tran­sit net­work (like Level 3 Com­mu­ni­ca­tions). It has three points of pres­ence: Paris, New York and Tokyo. AS 64650 is a Eu­ro­pean re­gional tran­sit net­work, AS 64640 is an East Asian re­gional tran­sit net­work and AS 64610, 64620 and 64630 are North Amer­i­can re­gional tran­sit net­works which are peer­ing each other. Other leaf AS are local ISP.

This is not re­ally what In­ter­net looks like but shap­ing our lab like a tree al­lows us to not deal with a lot of BGP stuff: we en­sure that the short­est path (in num­ber of BGP hops) from one client to one server is the best path. Each BGP router will get a ba­sic BGP con­fig­u­ra­tion con­tain­ing only the list of peers. Since Paris, Tokyo and NewYork are in the same AS, they will use iBGP. They are fully meshed for this pur­pose but this does not pro­vide re­dun­dancy be­cause iBGP routes are not re­dis­tributed in­side iBGP. A more com­plex setup would be needed to pro­vide re­dun­dancy in­side AS 64600. We do not use an IGP in­side leaf AS as we should: for the sake of sim­plic­ity, we re­dis­tribute di­rectly con­nected routes into BGP.

While in my pre­vi­ous lab, I used Quagga, I will use BIRD in this one. It is a more mod­ern dae­mon with a clean de­sign on how rout­ing ta­bles may in­ter­act. It also per­forms bet­ter. How­ever, it lacks sev­eral fea­tures from Quagga. The con­fig­u­ra­tion for BIRD is au­to­gen­er­ated from /etc/hosts. We use only one rout­ing ta­ble in­side BIRD. This ta­ble will be ex­ported to the ker­nel and to any BGP in­stance. Some di­rectly con­nected routes will be im­ported (ones rep­re­sent­ing client net­works) as well as all routes from BGP in­stances. Here is an ex­cerpt of AS 64610’s con­fig­uration:

protocol direct {
   description "Client networks";
   import filter {
     if net ~ [ 2001:db8:4600::/40{40,48} ] then accept;
     reject;
   };
   export none;
}

protocol kernel {
   persist;
   import none;
   export all;
}

protocol bgp {
   description "BGP with peer NewYork";
   local as 64610;
   neighbor 2001:db8:ffff:4600:4610::1 as 64600;
   gateway direct;
   hold time 30;
   export all;
   import all;
}

We do not ex­port in­ter-router sub­nets to keep our rout­ing ta­bles small. This also means that these IP ad­dresses are not routable in spite of the use of pub­lic IP ad­dresses. Here is the rout­ing ta­ble as seen by 64610 (use birdc6 to con­nect to BIRD):

# birdc show route
2001:db8:4622::/48 via 2001:db8:ffff:4610:4620::2 on eth3 [bgp4 18:52] * (100) [AS64622i]
                   via 2001:db8:ffff:4600:4610::1 on eth0 [bgp1 18:52] (100) [AS64622i]
2001:db8:4621::/48 via 2001:db8:ffff:4610:4620::2 on eth3 [bgp4 18:52] * (100) [AS64621i]
                   via 2001:db8:ffff:4600:4610::1 on eth0 [bgp1 18:52] (100) [AS64621i]
2001:db8:4612::/48 via 2001:db8:ffff:4610:4612::2 on eth2 [bgp3 18:52] * (100) [AS64612i]
2001:db8:4611::/48 via 2001:db8:ffff:4610:4611::2 on eth1 [bgp2 18:52] * (100) [AS64611i]

As you can no­tice, it knows sev­eral paths to the same net­work, how­ever, it only se­lects the best one (based on the length of the AS path):

# birdc show route 2001:db8:4622::/48 all
2001:db8:4622::/48 via 2001:db8:ffff:4610:4620::2 on eth3 [bgp4 18:52] * (100) [AS64622i]
        Type: BGP unicast univ
        BGP.origin: IGP
        BGP.as_path: 64620 64622
        BGP.next_hop: 2001:db8:ffff:4610:4620::2 fe80::e05f:a3ff:fef2:f4e0
        BGP.local_pref: 100
                   via 2001:db8:ffff:4600:4610::1 on eth0 [bgp1 18:52] (100) [AS64622i]
        Type: BGP unicast univ
        BGP.origin: IGP
        BGP.as_path: 64600 64620 64622
        BGP.next_hop: 2001:db8:ffff:4600:4610::1 fe80::3426:63ff:fe5e:afbd
        BGP.local_pref: 100

Let’s check that ev­ery­thing works with traceroute6 on C1:

# traceroute6 G2.lab
traceroute to G2.lab (2001:db8:4612::2:53), 30 hops max, 80 byte packets
 1  64652-eth1.lab (2001:db8:4652::1)  -338.277 ms  -338.449 ms  -338.502 ms
 2  64650-eth2.lab (2001:db8:ffff:4650:4652::1)  -338.324 ms  -338.363 ms  -338.384 ms
 3  Paris-eth2.lab (2001:db8:ffff:4600:4650::1)  -338.191 ms  -338.329 ms  -338.282 ms
 4  NewYork-eth0.lab (2001:db8:ffff:4600:4600:1:0:2)  -338.144 ms  -338.182 ms  -338.235 ms
 5  64610-eth0.lab (2001:db8:ffff:4600:4610::2)  -338.078 ms  -337.958 ms  -337.944 ms
 6  64612-eth0.lab (2001:db8:ffff:4610:4612::2)  -337.778 ms  -338.103 ms  -338.041 ms
 7  G2.lab (2001:db8:4612::2:53)  -338.001 ms  -338.045 ms  -338.080 ms

Let’s see what hap­pens if we shut the link be­tween NewYork and 64610. This link is VLAN 10. Since vde_switch does not al­low us to shut a port, we will change the VLAN of port 9 to VLAN 4093. Af­ter at most 30 sec­onds, the BGP ses­sion be­tween NewYork and 64610 will be torn down and pack­ets from C1 to G2 will take a new path:

# traceroute6 G2.lab
traceroute to G2.lab (2001:db8:4612::2:53), 30 hops max, 80 byte packets
 1  64652-eth1.lab (2001:db8:4652::1)  -338.382 ms  -338.499 ms  -338.503 ms
 2  64650-eth2.lab (2001:db8:ffff:4650:4652::1)  -338.284 ms  -338.317 ms  -338.151 ms
 3  Paris-eth2.lab (2001:db8:ffff:4600:4650::1)  -337.954 ms  -338.031 ms  -337.924 ms
 4  NewYork-eth0.lab (2001:db8:ffff:4600:4600:1:0:2)  -337.655 ms  -337.751 ms  -337.844 ms
 5  64620-eth0.lab (2001:db8:ffff:4600:4620::2)  -337.762 ms  -337.507 ms  -337.611 ms
 6  64610-eth3.lab (2001:db8:ffff:4610:4620::1)  -337.531 ms  -338.024 ms  -337.998 ms
 7  64612-eth0.lab (2001:db8:ffff:4610:4612::2)  -337.860 ms  -337.931 ms  -337.992 ms
 8  G2.lab (2001:db8:4612::2:53)  -337.863 ms  -337.768 ms  -337.808 ms

Setting up anycast DNS#

Our any­cast DNS ser­vice has three nodes. Two global ones, lo­cated in Eu­rope and North Amer­ica and one lo­cal one tar­geted at cus­tomers of AS 64620 only. The IP ad­dress of this any­cast DNS server is 2001:db8:aaaa::53/48. We as­sign this ad­dress to G1, G2 and L1. We also as­sign 2001:db8:aaaa::1/48 to 64651, 64612 and 64621. BIRD con­fig­u­ra­tion is adapted to ad­ver­tise this network:

protocol direct {
   description "Client networks";
   import filter {
     if net ~ [ 2001:db8:4600::/40{40,48} ] then accept;
     if net ~ [ 2001:db8:aaaa::/48 ] then accept;
     reject;
   };
   export none;
}

We use NSD as a name server. This is an au­thor­i­ta­tive-only name server. The con­fig­u­ra­tion is pretty sim­ple. We just en­sure that it is bound to the IPv6 any­cast ad­dress (oth­er­wise, it may an­swer with the wrong ad­dress). The zone file is cus­tom­ized to an­swer the name of the server:

joe@C1$ host -t TXT example.com 2001:db8:aaaa::53
Using domain server:
Name: 2001:db8:aaaa::53
Address: 2001:db8:aaaa::53#53

example.com descriptive text "G1"

joe@C5$ host -t TXT example.com 2001:db8:aaaa::53
Using domain server:
Name: 2001:db8:aaaa::53
Address: 2001:db8:aaaa::53#53

example.com descriptive text "G2"

If we try from C3, we may be redi­rected to L1 while we are not a cus­tomer of AS 64620. A com­mon way to han­dle this is to let 64621 tag the route with a spe­cial com­mu­nity that will let 64620 know that the route should not be exported.

protocol direct {
   description "Client networks";
   import filter {
     if net ~ [ 2001:db8:4600::/40{40,48} ] then accept;
     if net ~ [ 2001:db8:aaaa::/48 ] then {
          bgp_community.add((65535,65281));
          accept;
     }
     reject;
   };
   export none;
}

The com­mu­nity used is known as the no-export com­mu­nity. BIRD knows how to han­dle it. How­ever, we hit here some lim­i­ta­tions of BIRD:

  1. The cor­rect way to let 64622 know about this route is to build a BGP con­fed­er­a­tion for AS 64620 and let AS 64622 and AS 64621 be part of it. This means these three AS will be seen as AS 64200 from out­side of the con­fed­er­a­tion. More­over, the no-export com­mu­nity would al­low us to ex­port the route only within the con­fed­er­a­tion. This is ex­actly what we need but BIRD does not sup­port con­fed­erations.

  2. BIRD first se­lects the best route to ex­port then ap­ply fil­ter­ing. This means that even if 64620 knows non-fil­tered route to 2001:db8:aaaa::/48, it will se­lect the route to L1, fil­ter it out and ex­port noth­ing to 64622.

There­fore, we need to fall­back to ask BIRD to not ex­port the route to AS 64600, 64610 and 64630 on 64620. We could use a spe­cial com­mu­nity to fil­ter out the route but we keep it sim­ple and just fil­ter the route directly.

filter keep_local_dns {
   if net ~ [ 2001:db8:aaaa::/48 ] then reject;
   accept;
}
protocol bgp {
   description "BGP with peer NewYork";
   local as 64620;
   neighbor 2001:db8:ffff:4600:4620::1 as 64600;
   gateway direct;
   hold time 30;
   export filter keep_local_dns;
   import all;
}

Voilà. Only C4 will be able to query L1. If AS 64621 is miss­ing, it will query G2 in­stead. One ma­jor point about any­cast is to know that re­dun­dancy is done at the AS level only. If the name server on L1 does not work cor­rectly, re­quests will still be routed to it. There­fore, with any­cast, you still need to pro­vide re­dun­dancy in­side each AS (or stop ad­ver­tis­ing the any­cast net­work as soon as there is a failure).

Beyond DNS#

Any­cast can be used to serve HTTP re­quests too. This is less com­monly used be­cause your con­nec­tion may be rerouted to an­other node while it is still run­ning. This is not a prob­lem with DNS be­cause there is no con­nec­tion (you send a query, you re­ceive an an­swer). There are at least two CDN (MaxCDN and CacheFly) which use any­cast to en­sure low roundtrip time and very high re­li­ability.

Set­ting up an HTTP server with an any­cast ad­dress is no dif­fer­ent than set­ting up a DNS server. How­ever, you need to en­sure max­i­mum sta­bil­ity of the rout­ing path to avoid the con­nec­tion be­ing dropped while run­ning. I sup­pose that this is done by hir­ing some high-level BGP ex­pert. You can also mix uni­cast and anycast:

  • any­cast DNS an­swers the IP ad­dress of a nearby uni­cast HTTP server; or
  • any­cast DNS an­swers the IP ad­dress of an any­cast HTTP server that will is­sue a re­di­rect to a uni­cast HTTP server when down­load­ing or stream­ing large files.