Securing BGP on the host with origin validation

Vincent Bernat

An in­creas­ingly pop­u­lar de­sign for a dat­a­cen­ter net­work is BGP on the host: each host ships with a BGP dae­mon to ad­ver­tise the IPs it han­dles and re­ceives the routes to its fel­low servers. Com­pared to a L2-based de­sign, it is very scal­able, re­silient, cross-ven­dor and safe to op­er­ate.1 Take a look at “L3 rout­ing to the hy­per­vi­sor with BGP” for a us­age example.

Spine-leaf fabric two spine routers, six leaf routers and nine physical hosts. All links have a BGP session established over them. Some of the servers have a speech balloon specifying the IP prefix they want to handle.
BGP on the host with a spine-leaf IP fabric. A BGP session is established over each link and each host advertises its own IP prefixes.

While rout­ing on the host elim­i­nates the se­cu­rity prob­lems re­lated to Eth­er­net net­works, a server may an­nounce any IP pre­fix. In the above pic­ture, two of them are an­nounc­ing 2001:db8:cc::/64. This could be a le­git use of any­cast or a pre­fix hi­jack. BGP of­fers sev­eral so­lu­tions to im­prove this as­pect and one of them is to re­use the fea­tures around the RPKI.

Short introduction to the RPKI#

On the In­ter­net, BGP is mostly re­ly­ing on trust. This con­trib­utes to var­i­ous in­ci­dents due to op­er­a­tor er­rors, like the one that af­fected Cloud­flare a few months ago, or to ma­li­cious at­tack­ers, like the hi­jack of Ama­zon DNS to steal cryp­tocur­rency wal­lets. RFC 7454 ex­plains the best prac­tices to avoid such issues.

IP ad­dresses are al­lo­cated by five Re­gional In­ter­net Reg­istries (RIR). Each of them main­tains a data­base of the as­signed In­ter­net re­sources, no­tably the IP ad­dresses and the as­so­ci­ated AS num­bers. These data­bases may not be en­tirely re­li­able but are widely used to build ACLs to en­sure peers only an­nounce the pre­fixes they are ex­pected to. Here is an ex­am­ple of ACLs gen­er­ated by bgpq3 when peer­ing di­rectly with Apple:2

$ bgpq3 -l v6-IMPORT-APPLE -6 -R 48 -m 48 -A -J -E AS-APPLE
policy-options {
 policy-statement v6-IMPORT-APPLE {
replace:
  from {
    route-filter 2403:300::/32 upto /48;
    route-filter 2620:0:1b00::/47 prefix-length-range /48-/48;
    route-filter 2620:0:1b02::/48 exact;
    route-filter 2620:0:1b04::/47 prefix-length-range /48-/48;
    route-filter 2620:149::/32 upto /48;
    route-filter 2a01:b740::/32 upto /48;
    route-filter 2a01:b747::/32 upto /48;
  }
 }
}

The RPKI (RFC 6480) adds pub­lic-key cryp­tog­ra­phy on top of it to sign the au­tho­riza­tion for an AS to be the ori­gin of an IP pre­fix. Such record is a Route Orig­i­na­tion Au­tho­riza­tion (ROA). You can browse the data­bases of these ROAs through the RIPE’s RPKI Val­ida­tor instance:

Screenshot from an instance of RPKI validator showing the validity of 85.190.88.0/21 for AS 64476
RPKI validator shows one ROA for 85.190.88.0/21

BGP dae­mons do not have to down­load the data­bases or to check dig­i­tal sig­na­tures to val­i­date the re­ceived pre­fixes. In­stead, they off­load these tasks to a lo­cal RPKI val­ida­tor im­ple­ment­ing the “RPKI-to-Router Pro­to­col” (RTR, RFC 6810).

For more de­tails, have a look at “RPKI and BGP: our path to se­cur­ing In­ter­net Routing.”

Using origin validation in the datacenter#

While it is pos­si­ble to cre­ate our own RPKI for use in­side the dat­a­cen­ter, we can take a short­cut and use a val­ida­tor im­ple­ment­ing RTR, like GoRTR, and ac­cept­ing an­other source of truth. Let’s work on the fol­low­ing topology:

Spine-leaf fabric two spine routers, six leaf routers and nine physical hosts. All links have a BGP session established over them. Three of the physical hosts are validators and RTR sessions are established between them and the top-of-the-rack routers—except their own top-of-the-racks.
BGP on the host with prefix validation using RTR. Each server has its own AS number. The leaf routers establish RTR sessions to the validators.

As­sume we have a place to main­tain a map­ping be­tween the pri­vate AS num­bers used by each host and the al­lowed prefixes:3

ASN Allowed prefixes
AS 65005 2001:db8:aa::/64
AS 65006 2001:db8:bb::/64,
2001:db8:11::/64
AS 65007 2001:db8:cc::/64
AS 65008 2001:db8:dd::/64
AS 65009 2001:db8:ee::/64,
2001:db8:11::/64
AS 65010 2001:db8:ff::/64

From this ta­ble, we build a JSON file for GoRTR, as­sum­ing each host can an­nounce the pro­vided pre­fixes or longer ones (like 2001:db8:aa::­42:d9ff:­fefc:287a/128 for AS 65005):

{
  "roas": [
    {
      "prefix": "2001:db8:aa::/64",
      "maxLength": 128,
      "asn": "AS65005"
    }, {
      "…": "…"
    }, {
      "prefix": "2001:db8:ff::/64",
      "maxLength": 128,
      "asn": "AS65010"
    }, {
      "prefix": "2001:db8:11::/64",
      "maxLength": 128,
      "asn": "AS65006"
    }, {
      "prefix": "2001:db8:11::/64",
      "maxLength": 128,
      "asn": "AS65009"
    }
  ]
}

This file is de­ployed to all val­ida­tors and served by a web server. GoRTR is con­fig­ured to fetch it and up­date it ev­ery 10 minutes:

$ gortr -refresh=600 \
>       -verify=false -checktime=false \
>       -cache=http://127.0.0.1/rpki.json
INFO[0000] New update (7 uniques, 8 total prefixes). 0 bytes. Updating sha256 hash  -> 68a1d3b52db8d654bd8263788319f08e3f5384ae54064a7034e9dbaee236ce96
INFO[0000] Updated added, new serial 1

The re­fresh time could be low­ered but GoRTR can be no­ti­fied of an up­date us­ing the SIGHUP sig­nal. Clients are im­me­di­ately no­ti­fied of the change.

The next step is to con­fig­ure the leaf routers to val­i­date the re­ceived pre­fixes us­ing the farm of val­ida­tors. Most ven­dors sup­port RTR:

Platform Over TCP? Over SSH?
Juniper Junos ✔️ ❌
Cisco IOS XR ✔️ ✔️
Cisco IOS XE ✔️ ❌
Cisco IOS ✔️ ❌
Arista EOS ✔️ ❌
BIRD ✔️ ✔️
FRR ✔️ ✔️
GoBGP ✔️ ❌

Configuring Junos#

Junos only sup­ports plain-text TCP. First, let’s con­fig­ure the con­nec­tions to the val­i­da­tion servers:

routing-options {
    validation {
        group RPKI {
            session validator1 {
                hold-time 60;         # session is considered down after 1 minute
                record-lifetime 3600; # cache is kept for 1 hour
                refresh-time 30;      # cache is refreshed every 30 seconds
                port 8282;
            }
            session validator2 { /* OMITTED */ }
            session validator3 { /* OMITTED */ }
        }
    }
}

By de­fault, at most two ses­sions are ran­domly es­tab­lished at the same time. This pro­vides a good way to load-bal­ance them among the val­ida­tors while main­tain­ing good avail­abil­ity. The sec­ond step is to de­fine the pol­icy for route val­idation:

policy-options {
    policy-statement ACCEPT-VALID {
        term valid {
            from {
                protocol bgp;
                validation-database valid;
            }
            then {
                validation-state valid;
                accept;
            }
        }
        term invalid {
            from {
                protocol bgp;
                validation-database invalid;
            }
            then {
                validation-state invalid;
                reject;
            }
        }
    }
    policy-statement REJECT-ALL {
        then reject;
    }
}

The pol­icy state­ment ACCEPT-VALID turns the val­i­da­tion state of a pre­fix from unknown to valid if the ROA data­base says it is valid. It also ac­cepts the route. If the pre­fix is in­valid, the pre­fix is marked as such and re­jected. We have also pre­pared a REJECT-ALL state­ment to re­ject ev­ery­thing else, no­tably un­known prefixes.

A ROA only cer­ti­fies the ori­gin of a pre­fix. A ma­li­cious ac­tor can there­fore prepend the ex­pected AS num­ber to the AS path to cir­cum­vent the val­i­da­tion. For ex­am­ple, AS 65007 could an­nounce 2001:db8:dd::/64, a pre­fix al­lo­cated to AS 65006, by ad­ver­tis­ing it with the AS path 65007 65006. To avoid that, we de­fine an ad­di­tional pol­icy state­ment to re­ject AS paths with more than one ASN:4

policy-options {
    as-path EXACTLY-ONE-ASN "^.$";
    policy-statement ONLY-DIRECTLY-CONNECTED {
        term exactly-one-asn {
            from {
                protocol bgp;
                as-path EXACTLY-ONE-ASN;
            }
            then next policy;
        }
        then reject;
    }
}

The last step is to con­fig­ure the BGP sessions:

protocols {
    bgp {
        group HOSTS {
            local-as 65100;
            type external;
            # export [ … ];
            import [ ONLY-DIRECTLY-CONNECTED ACCEPT-VALID REJECT-ALL ];
            enforce-first-as;
            neighbor 2001:db8:42::a10 {
                peer-as 65005;
            }
            neighbor 2001:db8:42::a12 {
                peer-as 65006;
            }
            neighbor 2001:db8:42::a14 {
                peer-as 65007;
            }
        }
    }
}

The im­port pol­icy re­jects any AS path longer than one AS, ac­cepts any val­i­dated pre­fix and re­jects ev­ery­thing else. The enforce-first-as di­rec­tive is also pretty im­por­tant: it en­sures the first (and, here, only) AS in the AS path matches the peer AS. With­out it, a ma­li­cious neigh­bor could in­ject a pre­fix us­ing an AS dif­fer­ent than its own, de­feat­ing our purpose.5

Let’s check the state of the RTR ses­sions and the database:

> show validation session
Session                                  State   Flaps     Uptime #IPv4/IPv6 records
2001:db8:4242::10                        Up          0   00:16:09 0/9
2001:db8:4242::11                        Up          0   00:16:07 0/9
2001:db8:4242::12                        Connect     0            0/0

> show validation database
RV database for instance master

Prefix                 Origin-AS Session                                 State   Mismatch
2001:db8:11::/64-128       65006 2001:db8:4242::10                       valid
2001:db8:11::/64-128       65006 2001:db8:4242::11                       valid
2001:db8:11::/64-128       65009 2001:db8:4242::10                       valid
2001:db8:11::/64-128       65009 2001:db8:4242::11                       valid
2001:db8:aa::/64-128       65005 2001:db8:4242::10                       valid
2001:db8:aa::/64-128       65005 2001:db8:4242::11                       valid
2001:db8:bb::/64-128       65006 2001:db8:4242::10                       valid
2001:db8:bb::/64-128       65006 2001:db8:4242::11                       valid
2001:db8:cc::/64-128       65007 2001:db8:4242::10                       valid
2001:db8:cc::/64-128       65007 2001:db8:4242::11                       valid
2001:db8:dd::/64-128       65008 2001:db8:4242::10                       valid
2001:db8:dd::/64-128       65008 2001:db8:4242::11                       valid
2001:db8:ee::/64-128       65009 2001:db8:4242::10                       valid
2001:db8:ee::/64-128       65009 2001:db8:4242::11                       valid
2001:db8:ff::/64-128       65010 2001:db8:4242::10                       valid
2001:db8:ff::/64-128       65010 2001:db8:4242::11                       valid

  IPv4 records: 0
  IPv6 records: 18

Here is an ex­am­ple of ac­cepted route:

> show route protocol bgp table inet6 extensive all
inet6.0: 11 destinations, 11 routes (8 active, 0 holddown, 3 hidden)
2001:db8:bb::42/128 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Next hop type: Router, Next hop index: 0
                Address: 0xd050470
                Next-hop reference count: 4
                Source: 2001:db8:42::a12
                Next hop: 2001:db8:42::a12 via em1.0, selected
                Session Id: 0x0
                State: <Active NotInstall Ext>
                Local AS: 65006 Peer AS: 65000
                Age: 12:11
                Validation State: valid
                Task: BGP_65000.2001:db8:42::a12+179
                AS path: 65006 I
                Accepted
                Localpref: 100
                Router ID: 1.1.1.1

A re­jected route would be sim­i­lar with the rea­son “re­jected by im­port pol­icy” shown in the de­tails and the val­i­da­tion state would be invalid.

Configuring BIRD#

BIRD sup­ports both plain-text TCP and SSH. Let’s con­fig­ure it to use SSH. We need to gen­er­ate key­pairs for both the leaf router and the val­ida­tors (they can all share the same key­pair). We also have to cre­ate a known_hosts file for BIRD:

(validatorX)$ ssh-keygen -qN "" -t rsa -f /etc/gortr/ssh_key
(validatorX)$ echo -n "validatorX:8283 " ; \
>             cat /etc/bird/ssh_key_rtr.pub
validatorX:8283 ssh-rsa AAAAB3[…]Rk5TW0=
(leaf1)$ ssh-keygen -qN "" -t rsa -f /etc/bird/ssh_key
(leaf1)$ echo 'validator1:8283 ssh-rsa AAAAB3[…]Rk5TW0=' >> /etc/bird/known_hosts
(leaf1)$ echo 'validator2:8283 ssh-rsa AAAAB3[…]Rk5TW0=' >> /etc/bird/known_hosts
(leaf1)$ cat /etc/bird/ssh_key.pub
ssh-rsa AAAAB3[…]byQ7s=
(validatorX)$ echo 'ssh-rsa AAAAB3[…]byQ7s=' >> /etc/gortr/authorized_keys

GoRTR needs ad­di­tional flags to al­low con­nec­tions over SSH:

$ gortr -refresh=600 -verify=false -checktime=false \
>     -cache=http://127.0.0.1/rpki.json \
>     -ssh.bind=:8283 \
>     -ssh.key=/etc/gortr/ssh_key \
>     -ssh.method.key=true \
>     -ssh.auth.user=rpki \
>     -ssh.auth.key.file=/etc/gortr/authorized_keys
INFO[0000] Enabling ssh with the following authentications: password=false, key=true
INFO[0000] New update (7 uniques, 8 total prefixes). 0 bytes. Updating sha256 hash  -> 68a1d3b52db8d654bd8263788319f08e3f5384ae54064a7034e9dbaee236ce96
INFO[0000] Updated added, new serial 1

Then, we can con­fig­ure BIRD to use these RTR servers:

roa6 table ROA6;
template rpki VALIDATOR {
   roa6 { table ROA6; };
   transport ssh {
     user "rpki";
     remote public key "/etc/bird/known_hosts";
     bird private key "/etc/bird/ssh_key";
   };
   refresh keep 30;
   retry keep 30;
   expire keep 3600;
}
protocol rpki VALIDATOR1 from VALIDATOR {
   remote validator1 port 8283;
}
protocol rpki VALIDATOR2 from VALIDATOR {
   remote validator2 port 8283;
}

Un­like Junos, BIRD does not have a fea­ture to only use a sub­set of val­ida­tors. There­fore, we only con­fig­ure two of them. As a safety mea­sure, if both con­nec­tions be­come un­avail­able, BIRD will keep the ROAs for one hour.

We can query the state of the RTR ses­sions and the database:

> show protocols all VALIDATOR1
Name       Proto      Table      State  Since         Info
VALIDATOR1 RPKI       ---        up     17:28:56.321  Established
  Cache server:     rpki@validator1:8283
  Status:           Established
  Transport:        SSHv2
  Protocol version: 1
  Session ID:       0
  Serial number:    1
  Last update:      before 25.212 s
  Refresh timer   : 4.787/30
  Retry timer     : ---
  Expire timer    : 3574.787/3600
  No roa4 channel
  Channel roa6
    State:          UP
    Table:          ROA6
    Preference:     100
    Input filter:   ACCEPT
    Output filter:  REJECT
    Routes:         9 imported, 0 exported, 9 preferred
    Route change stats:     received   rejected   filtered    ignored   accepted
      Import updates:              9          0          0          0          9
      Import withdraws:            0          0        ---          0          0
      Export updates:              0          0          0        ---          0
      Export withdraws:            0        ---        ---        ---          0

> show route table ROA6
Table ROA6:
    2001:db8:11::/64-128 AS65006  [VALIDATOR1 17:28:56.333] * (100)
                                  [VALIDATOR2 17:28:56.414] (100)
    2001:db8:11::/64-128 AS65009  [VALIDATOR1 17:28:56.333] * (100)
                                  [VALIDATOR2 17:28:56.414] (100)
    2001:db8:aa::/64-128 AS65005  [VALIDATOR1 17:28:56.333] * (100)
                                  [VALIDATOR2 17:28:56.414] (100)
    2001:db8:bb::/64-128 AS65006  [VALIDATOR1 17:28:56.333] * (100)
                                  [VALIDATOR2 17:28:56.414] (100)
    2001:db8:cc::/64-128 AS65007  [VALIDATOR1 17:28:56.333] * (100)
                                  [VALIDATOR2 17:28:56.414] (100)
    2001:db8:dd::/64-128 AS65008  [VALIDATOR1 17:28:56.333] * (100)
                                  [VALIDATOR2 17:28:56.414] (100)
    2001:db8:ee::/64-128 AS65009  [VALIDATOR1 17:28:56.333] * (100)
                                  [VALIDATOR2 17:28:56.414] (100)
    2001:db8:ff::/64-128 AS65010  [VALIDATOR1 17:28:56.333] * (100)
                                  [VALIDATOR2 17:28:56.414] (100)

Like for the Junos case, a ma­li­cious ac­tor could try to work­around the val­i­da­tion by build­ing an AS path where the last AS num­ber is the le­git­i­mate one. BIRD is flex­i­ble enough to al­low us to use any AS to check the IP pre­fix. In­stead of check­ing the ori­gin AS, we ask it to check the peer AS with this func­tion, with­out look­ing at the AS path:

function validated(int peeras) {
   if (roa_check(ROA6, net, peeras) != ROA_VALID) then {
      print "Ignore invalid ROA ", net, " for ASN ", peeras;
      reject;
   }
   accept;
}

The BGP in­stance is then con­fig­ured to use the above func­tion as the im­port policy:

protocol bgp PEER1 {
   local as 65100;
   neighbor 2001:db8:42::a10 as 65005;
   connect delay time 30;
   ipv6 {
      import keep filtered;
      import where validated(65005);
      # export …;
   };
}

You can view the re­jected routes with show route filtered, but BIRD does not store in­for­ma­tion about the val­i­da­tion state in the routes. You can also watch the logs:

2019-07-31 17:29:08.491 <INFO> Ignore invalid ROA 2001:db8:bb::40:/126 for ASN 65005

Cur­rently, BIRD does not reeval­u­ate the fil­ters when the ROAs are up­dated. There is work in progress to fix this. If this fea­ture is im­por­tant to you, have a look at FRR in­stead: it also sup­ports the RTR pro­to­col and trig­gers a soft re­con­fig­u­ra­tion of the BGP ses­sions when ROAs are updated.

Up­date (2021-03)

From ver­sion 2.0.8, BIRD reeval­u­ates the fil­ters when the ROAs are up­dated. You need to re­place import keep filtered with import table yes in the BGP in­stance con­fig­u­ra­tion. You can also drop the connect delay time di­rec­tive in the pro­posed con­fig­u­ra­tion. Its pur­pose was to en­sure the ROAs are loaded be­fore the BGP con­nec­tion is es­tablished.