BGP LLGR: robust and reactive BGP sessions

Vincent Bernat

On a BGP-routed net­work with mul­ti­ple re­dun­dant paths, we seek to achieve two goals con­cern­ing re­li­ability:

  1. A fail­ure on a path should quickly bring down the re­lated BGP ses­sions. A com­mon ex­pec­ta­tion is to re­cover in less than a sec­ond by di­vert­ing the traf­fic to the re­main­ing paths.

  2. As long as a path is op­er­a­tional, the re­lated BGP ses­sions should stay up, even un­der duress.

Detecting failures fast: BFD#

To quickly de­tect a fail­ure, BGP can be as­so­ci­ated with BFD, a pro­to­col to de­tect faults in bidi­rec­tional paths,1 de­fined in RFC 5880 and RFC 5882. BFD can use very low timers, like 100 ms.

How­ever, when BFD runs in a process on top of a generic ker­nel,2 no­tably when run­ning BGP on the host, it is not un­ex­pected to lose a few BFD pack­ets un­der ad­verse con­di­tions: the dae­mon han­dling the BFD ses­sions may not get enough CPU to an­swer in a timely man­ner. In this sce­nario, it is not un­likely for all the BGP ses­sions to go down at the same time, cre­at­ing an out­age, as de­picted in the last case in the di­a­gram below.

BGP and failed sessions
Examples of failures on a network using BGP as the underlying routing protocol. A link failure is detected by BFD and the failed path is removed from the ECMP route. However, when high CPU usage on the bottom router prevents BFD packets from being processed timely, all paths are removed.

So far, we have two con­tra­dict­ing roads:

  • lower the BFD timers to quickly de­tect a fail­ure along the path; or
  • raise the BFD timers to en­sure BGP ses­sions re­main op­er­ational.

Fixing false positives: BGP LLGR#

Long-lived BGP Grace­ful Restart is a new BGP ca­pa­bil­ity to re­tain stale routes for a longer pe­riod af­ter a ses­sion fail­ure but treat­ing them as least-pre­ferred. It also de­fines a well-known com­mu­nity to share this in­for­ma­tion with other routers. It is de­fined in RFC 9494 and sev­eral im­ple­men­ta­tions al­ready exist:

  • Ju­niper Junos (since 15.1, see the doc­u­men­tation);
  • Cisco IOS XR (tested on 7.3.2);
  • BIRD (since 1.6.5 and 2.0.3);
  • GoBGP (since 1.33); and
  • FRR (since 8.2).

The fol­low­ing il­lus­tra­tion shows what hap­pens dur­ing two fail­ure sce­nar­ios. Like with­out LLGR, in ❷, a link fail­ure is de­tected by BFD and the failed path is re­moved from the route as two other paths re­main with a higher pref­er­ence. A cou­ple of min­utes later, the faulty path has its stale timer ex­pired and will not be used any­more. Shortly af­ter, in ❸, the bot­tom router ex­pe­ri­ences high CPU us­age, pre­vent­ing BFD pack­ets from be­ing pro­cessed timely. The BGP ses­sions are closed and the re­main­ing paths be­come stale but as there is no bet­ter path left, they are still used un­til the LLGR timer ex­pires. In the mean­time, we ex­pect the BGP ses­sions to resume.

BGP with LLGR
Examples of failures on a network using BGP as the underlying routing protocol with LLGR enabled.

From the point of view of the top router, the first failed path was con­sid­ered as stale be­cause the BGP ses­sion with R1 was down. How­ever, dur­ing the sec­ond fail­ure, the two re­main­ing paths were con­sid­ered as stale be­cause they were tagged with the well-known com­mu­nity LLGR_​STALE (65535:6) by R2 and R3.

An­other in­ter­est­ing point of BGP LLGR is the abil­ity to restart the BGP dae­mon with­out any im­pact—as long as all paths keep a steady state shortly be­fore and dur­ing restart. This is quite in­ter­est­ing when run­ning BGP on the host.3

BIRD#

Let’s see how to con­fig­ure BIRD 1.6. As BGP LLGR is built on top of the reg­u­lar BGP grace­ful restart (BGP GR) ca­pa­bil­ity, we need to en­able both. The timer for BGP LLGR starts af­ter the timer for BGP GR. Dur­ing a reg­u­lar grace­ful restart, routes are kept with the same pref­er­ence. There­fore it is im­por­tant to set this timer to 0.

template bgp BGP_LLGR {
  bfd graceful;
  graceful restart yes;
  graceful restart time 0;
  long lived graceful restart yes;
  long lived stale time 120;
}

When a prob­lem ap­pears on the path, the BGP ses­sion goes down and the LLGR timer starts:

$ birdc show protocol R1_1 all
name     proto    table    state  since       info
R1_1     BGP      master   start  11:20:17    Connect
  Preference:     100
  Input filter:   ACCEPT
  Output filter:  ACCEPT
  Routes:         1 imported, 0 exported, 0 preferred
  Route change stats:     received   rejected   filtered    ignored   accepted
    Import updates:              2          0          0          0          4
    Import withdraws:            0          0        ---          0          0
    Export updates:             12         10          0        ---          2
    Export withdraws:            1        ---        ---        ---          0
  BGP state:          Connect
    Neighbor address: 2001:db8:104::1
    Neighbor AS:      65000
    Neighbor graceful restart active
    LL stale timer:   112/-

The re­lated paths are marked as stale (as re­ported by the s in 100s) and tagged with the well-known com­mu­nity LLGR_STALE:

$ birdc show route 2001:db8:10::1/128 all
2001:db8:10::1/128 via 2001:db8:204::1 on eth0.204 [R1_2 10:35:01] * (100) [i]
        Type: BGP unicast univ
        BGP.origin: IGP
        BGP.as_path:
        BGP.next_hop: 2001:db8:204::1 fe80::5254:3300:cc00:5
        BGP.local_pref: 100
                   via 2001:db8:104::1 on eth0.104 [R1_1 11:22:51] (100s) [i]
        Type: BGP unicast univ
        BGP.origin: IGP
        BGP.as_path:
        BGP.next_hop: 2001:db8:104::1 fe80::5254:3300:6800:5
        BGP.local_pref: 100
        BGP.community: (65535,6)

We are left with only one path for the route in the kernel:

$ ip route show 2001:db8:10::1
2001:db8:10::1 via 2001:db8:204::1 dev eth0.204 proto bird metric 1024 pref medium

To up­grade BIRD with­out im­pact, it needs to run with the -R flag and the graceful restart yes di­rec­tive should be present in the kernel pro­to­cols. Then, be­fore up­grade, stop it us­ing SIGKILL in­stead of SIGTERM to avoid a clean close of the BGP sessions.

Juniper Junos#

With Junos, we only have to en­able BGP LLGR for each fam­ily—as­sum­ing BFD is al­ready con­figured:

# Enable BGP LLGR
edit protocols bgp group peers family inet6 unicast
set graceful-restart long-lived restarter stale-time 2m

Once a path is fail­ing, the as­so­ci­ated BGP ses­sion goes down and the BGP LLGR timer starts:

> show bgp neighbor 2001:db8:104::4
Peer: 2001:db8:104::4+179 AS 65000 Local: 2001:db8:104::1+57667 AS 65000
  Group: peers                 Routing-Instance: master
  Forwarding routing-instance: master
  Type: Internal    State: Connect        Flags: <>
  Last State: Active        Last Event: ConnectRetry
  Last Error: None
  Export: [ LOOPBACK NOTHING ]
  Options: <Preference HoldTime Ttl AddressFamily Multipath Refresh>
  Options: <BfdEnabled LLGR>
  Address families configured: inet6-unicast
  Holdtime: 6 Preference: 170
  NLRI inet6-unicast:
  Number of flaps: 2
  Last flap event: Restart
  Time until long-lived stale routes deleted: inet6-unicast 00:01:05
  Table inet6.0 Bit: 20000
    RIB State: BGP restart is complete
    Send state: not advertising
    Active prefixes:              0
    Received prefixes:            1
    Accepted prefixes:            1
    Suppressed due to damping:    0
    LLGR-stale prefixes:          1

The as­so­ci­ated path is marked as stale and is there­fore in­ac­tive as there are bet­ter paths available:

> show route 2001:db8:10::4 extensive
[…]
BGP    Preference: 170/-101
       Source: 2001:db8:104::4
       Next hop type: Router, Next hop index: 778
       Next hop: 2001:db8:104::4 via em1.104, selected
       Protocol next hop: 2001:db8:104::4
       Indirect next hop: 0xb1d27c0 1048578 INH Session ID: 0x15c
       State: <Int Ext>
       Inactive reason: LLGR stale
       Local AS: 65000 Peer AS: 65000
       Age: 4  Metric2: 0
       Communities: llgr-stale
       Accepted LongLivedStale
       Localpref: 100
       Router ID: 1.0.0.4
[…]

Have a look at the Git repos­i­tory for the com­plete con­fig­u­ra­tions as well as the ex­pected out­puts dur­ing nor­mal op­er­a­tions. There is also a vari­ant with the con­fig­u­ra­tions of BIRD and Junos when act­ing as a BGP route re­flec­tor. Now that FRR got BFD sup­port, I hope it will get LLGR sup­port as well.

Up­date (2022-01)

Do­natas Abraitis re­cently added sup­port for LLGR helper mode in FRR! It will be present in ver­sion 8.2.

Up­date (2024-01)

RFC 9494 has been pub­lished in No­vem­ber 2023!