SNMP support for Keepalived

Vincent Bernat

Keepalived is a high-avail­abil­ity and load-bal­anc­ing so­lu­tion. With VRRP, it al­lows you to share IP ad­dresses be­tween sev­eral servers or routers. At a given mo­ment, only one of them (the mas­ter) is handed the IP ad­dress. VRRP is usu­ally set up for re­dun­dant routers, but you can also set up re­dun­dant ser­vices. If the mas­ter be­comes un­avail­able or fails, one of the back­ups takes over the IP ad­dress. VRRP is not a clus­ter re­source man­ager. It is tar­geted at shar­ing one (or sev­eral) IPs be­tween a set of hosts. If you want to share other re­sources with com­plex con­di­tions like “at least two web servers should run and the data­base should not run on the same node as a web server,” you should look at heart­beat instead.

For load-bal­anc­ing, Keepalived re­lies on IPVS, a Linux sub­sys­tem fea­tur­ing layer-4 switch­ing. Keepalived checks if your servers are alive and tells IPVS. IPVS han­dles in­com­ing con­nec­tions and sends them to the ap­pro­pri­ate alive server de­pend­ing on the pol­icy you can con­fig­ure. For ex­am­ple, one pol­icy is round-robin: servers are handed new con­nec­tions in cir­cu­lar order.

Keepalived & SNMP#

Keepalived did not fea­ture na­tive SNMP sup­port. I have added com­plete SNMP sup­port and I hope these patches will be ac­cepted up­stream. Mean­while, you can grab Keepalived with SNMP sup­port source code from GitHub. Cur­rently, it is based on the lat­est ver­sion of Keepalived.

Up­date (2014-03)

SNMP sup­port has been merged in Keepalived 1.2.5.

Here is what you can do:

  • query the con­fig­u­ra­tion of the run­ning Keepalived with­out pars­ing con­fig­u­ra­tion files;
  • query run­time sta­tus (like VRRP sta­tus, pri­or­ity, or cur­rent state of a vir­tual server) with­out look­ing in the logs;
  • query run­time sta­tis­tics about vir­tual servers (how many con­nec­tions were han­dled by this real server, for ex­am­ple) with­out pars­ing the out­put of ipvsadm;
  • get no­ti­fied with the help of SNMP traps when some­thing changed (VRRP tran­si­tion or a real server that be­came un­available);
  • change the pri­or­ity of a VRRP in­stance (to force a tran­si­tion as mas­ter); and
  • change the weight of a real server (to re­move it from a pool of servers).

Compilation#

Up­date (2022-01)

The Git repos­i­tory be­low is not up-to-date. As SNMP sup­port is present since 1.2.5, use a reg­u­lar Keepalived re­lease instead.

To use it:

$ git clone https://github.com/vincentbernat/keepalived.git
$ cd keepalived
$ git checkout snmp
$ ./configure --enable-snmp
$ make

You get bin/​keepalived which is the SNMP-en­abled Keepalived dae­mon. You can use it di­rectly or in­stall it with make install as root.

On RHEL, SNMP is not prop­erly pack­aged. Try the fol­low­ing com­mand in­stead of make:

$ make LDFLAGS="$(net-snmp-config --agent-libs) -lpopt -lssl -lcrypto"

Use#

To use it, you need to en­sure that your main snmpd dae­mon can be­come a mas­ter agent. Check for the pres­ence of the master agentx line in snmpd.​conf. The next step is to start Keepalived with the -x switch. Keepalived con­nects to your main snmpd dae­mon. You can check the logs for a line like this:

May 28 17:21:19 L1 Keepalived_vrrp: NET-SNMP version 5.4.3 AgentX subagent connected

Up­date (2012-02)

The in­ter­face be­tween Keepalived and the mas­ter agent snmpd through AgentX pro­to­col should be asyn­chro­nous. How­ever, the reg­u­lar check of the con­nec­tion is done syn­chronously. If the mas­ter agent be­comes un­re­spon­sive, Keepalived will also be un­re­spon­sive which is crit­i­cal for the VRRP part. You may mit­i­gate the prob­lem by ap­ply­ing some work­arounds.

Lab#

To test it, we can set up a lit­tle lab with UML. You can find ad­di­tional de­tails about this kind of lab in my post about net­work lab with UML. Grab the com­plete lab from GitHub. En­sure that you cor­rect the path to Keepalived sources at the top of the setup script.

Keepalived & SNMP lab
Topology of Keepalived lab

The lab fea­tures four web servers that are merged into a sin­gle web ser­vice with the help of Keepalived. Since Keepalived now fea­tures full IPv6 sup­port, we also use IPv6 in our lab.

Here is the VRRP con­fig­u­ra­tion of the first Keepalived :

vrrp_instance VI_1 {
   state MASTER
   interface eth2
   dont_track_primary
   track_interface {
      eth0
      eth1
   }
   virtual_router_id 1
   priority 150
   advert_int 2
   authentication {
      auth_type PASS
      auth_pass blibli
   }
   virtual_ipaddress {
      1.1.1.15/24 dev eth0
      2001:db8::15/64 dev eth0
      192.168.1.1/24 dev eth1
      fd00::1/64 dev eth1
  }
}

We de­fine two vir­tual servers, one for IPv4, one for IPv6. Here is an ex­cerpt of the con­fig­uration:

virtual_server_group VS_GROUP_IPv4 {
   1.1.1.15 80
}
virtual_server group VS_GROUP_IPv4 {
   delay_loop 10
   lb_algo rr
   lb_kind NAT
   protocol TCP
   real_server 192.168.1.10 80 {
      weight 1
      HTTP_GET {
        url {
          path /
          status_code 200
        }
        connect_timeout 10
      }
   }
   […]
}

When run­ning the lab (with ./setup), we can check that ev­ery­thing works as ex­pected. From the client node, we can make sev­eral re­quests, both with IPv4 and IPv6:

$ curl  http://1.1.1.15
W3
$ curl  http://1.1.1.15
W2
$ wget -o /dev/null -O - 'http://[2001:db8::15]'
W3
$ wget -o /dev/null -O - 'http://[2001:db8::15]'
W2

If we put down eth1 on L1, L2 be­comes mas­ter and ev­ery­thing works as ex­pected. Our client node can still do the requests.

SNMP#

We can query the con­fig­u­ra­tion of our run­ning Keepalived from L1. The ap­pro­pri­ate MIB is KEEPALIVED-MIB. The base OID for this MIB is .1.3.6.1.4.1.9586.100.5. It is hosted in the OID space al­lo­cated by IANA to the De­bian project.

For ex­am­ple, to get the con­fig­u­ra­tion of our VRRP instance:

# snmpwalk -v2c -cpublic localhost KEEPALIVED-MIB::vrrpInstanceTable
KEEPALIVED-MIB::vrrpInstanceName.1 = STRING: VI_1
KEEPALIVED-MIB::vrrpInstanceVirtualRouterId.1 = Gauge32: 1
KEEPALIVED-MIB::vrrpInstanceState.1 = INTEGER: master(2)
KEEPALIVED-MIB::vrrpInstanceInitialState.1 = INTEGER: master(2)
KEEPALIVED-MIB::vrrpInstanceWantedState.1 = INTEGER: master(2)
KEEPALIVED-MIB::vrrpInstanceBasePriority.1 = INTEGER: 150
KEEPALIVED-MIB::vrrpInstanceEffectivePriority.1 = INTEGER: 150
KEEPALIVED-MIB::vrrpInstanceVipsStatus.1 = INTEGER: allSet(1)
KEEPALIVED-MIB::vrrpInstancePrimaryInterface.1 = STRING: eth2
KEEPALIVED-MIB::vrrpInstanceTrackPrimaryIf.1 = INTEGER: tracked(1)
KEEPALIVED-MIB::vrrpInstanceAdvertisementsInt.1 = Gauge32: 2 seconds
KEEPALIVED-MIB::vrrpInstancePreempt.1 = INTEGER: preempt(1)
KEEPALIVED-MIB::vrrpInstancePreemptDelay.1 = Gauge32: 0 seconds
KEEPALIVED-MIB::vrrpInstanceAuthType.1 = INTEGER: password(1)
KEEPALIVED-MIB::vrrpInstanceLvsSyncDaemon.1 = INTEGER: disabled(2)
KEEPALIVED-MIB::vrrpInstanceGarpDelay.1 = Gauge32: 0 seconds
KEEPALIVED-MIB::vrrpInstanceSmtpAlert.1 = INTEGER: disabled(2)
KEEPALIVED-MIB::vrrpInstanceNotifyExec.1 = INTEGER: disabled(2)

We can get sta­tis­tics from IPVS subsystem:

# snmpwalk -v 2c -c public localhost KEEPALIVED-MIB::virtualServerTable | grep Stats
KEEPALIVED-MIB::virtualServerStatsConns.1 = Gauge32: 6 connections
KEEPALIVED-MIB::virtualServerStatsConns.2 = Gauge32: 6 connections
KEEPALIVED-MIB::virtualServerStatsInPkts.1 = Counter32: 36 packets
KEEPALIVED-MIB::virtualServerStatsInPkts.2 = Counter32: 36 packets
KEEPALIVED-MIB::virtualServerStatsOutPkts.1 = Counter32: 24 packets
KEEPALIVED-MIB::virtualServerStatsOutPkts.2 = Counter32: 24 packets
KEEPALIVED-MIB::virtualServerStatsInBytes.1 = Counter64: 2970 bytes
KEEPALIVED-MIB::virtualServerStatsInBytes.2 = Counter64: 3312 bytes
KEEPALIVED-MIB::virtualServerStatsOutBytes.1 = Counter64: 2592 bytes
KEEPALIVED-MIB::virtualServerStatsOutBytes.2 = Counter64: 3072 bytes

We can also dis­able a real server, for ex­ample W1:

# snmpget -v2c -cpublic localhost KEEPALIVED-MIB::realServerAddrType.1.1 \
>                               KEEPALIVED-MIB::realServerAddress.1.1
KEEPALIVED-MIB::realServerAddrType.1.1 = INTEGER: ipv4(1)
KEEPALIVED-MIB::realServerAddress.1.1 = Hex-STRING: C0 A8 01 0A
# snmpget -v2c -cpublic localhost KEEPALIVED-MIB::realServerAddress.2.1 \
>                               KEEPALIVED-MIB::realServerAddress.2.1
KEEPALIVED-MIB::realServerAddress.2.1 = Hex-STRING: FD 00 00 00 00 00 00 00 00 00 00 00 00 00 00 10
KEEPALIVED-MIB::realServerAddress.2.1 = Hex-STRING: FD 00 00 00 00 00 00 00 00 00 00 00 00 00 00 10
# snmpset -v2c -cprivate localhost KEEPALIVED-MIB::realServerWeight.1.1 = 0
KEEPALIVED-MIB::realServerWeight.1.1 = INTEGER: 0
# snmpset -v2c -cprivate localhost KEEPALIVED-MIB::realServerWeight.2.1 = 0
KEEPALIVED-MIB::realServerWeight.2.1 = INTEGER: 0

From C1, you can now check that W1 is not queried anymore.

Let’s graph the num­ber of con­nec­tions per sec­ond each real server gets. With a sim­ple script us­ing rrdtool, we can get the fol­low­ing graph:

rrdtool graph built from stats returned by Keepalived SNMP agent
RRD plot from values retrieved with SNMP