Integration of a Go service with systemd: readiness & liveness

Vincent Bernat

Un­like other pro­gram­ming lan­guages, Go’s run­time doesn’t pro­vide a way to re­li­ably dae­mo­nize a ser­vice. A sys­tem dae­mon has to sup­ply this func­tion­al­ity. Most dis­tri­bu­tions ship sys­temd which would fit the bill. A cor­rect in­te­gra­tion with sys­temd is quite straight­for­ward. There are two in­ter­est­ing as­pects: readi­ness & liveness.

As an ex­am­ple, we will dae­mo­nize this ser­vice whose goal is to an­swer re­quests with nifty 404 errors:

package main

import (
    "log"
    "net"
    "net/http"
)

func main() {
    l, err := net.Listen("tcp", ":8081")
    if err != nil {
        log.Panicf("cannot listen: %s", err)
    }
    http.Serve(l, nil)
}

You can build it with go build 404.go.

Here is the ser­vice file, 404.service:1

[Unit]
Description=404 micro-service

[Service]
Type=notify
ExecStart=/usr/bin/404
WatchdogSec=30s
Restart=on-failure

[Install]
WantedBy=multi-user.target

Readiness#

The clas­sic way for a Unix dae­mon to sig­nal its readi­ness is to dae­mo­nize. Tech­ni­cally, this is done by call­ing fork(2) twice (which also serves other in­tents). This is a very com­mon task and the BSD sys­tems, as well as some other C li­braries, sup­ply a dae­mon(3) func­tion for this pur­pose. Ser­vices are ex­pected to dae­mo­nize only when they are ready (af­ter read­ing con­fig­u­ra­tion files and set­ting up a lis­ten­ing socket, for ex­am­ple). Then, a sys­tem can re­li­ably ini­tial­ize its ser­vices with a sim­ple lin­ear script:

syslogd
unbound
ntpd -s

Each dae­mon can rely on the pre­vi­ous one be­ing ready to do its work. The se­quence of ac­tions is the following:

  1. syslogd reads its con­fig­u­ra­tion, ac­ti­vates /dev/log, dae­monizes.
  2. unbound reads its con­fig­u­ra­tion, lis­tens on 127.0.0.1:53, dae­monizes.
  3. ntpd reads its con­fig­u­ra­tion, con­nects to NTP peers, waits for clock to be syn­chro­nized,2 dae­monizes.

With sys­temd, we would use Type=fork in the ser­vice file. How­ever, Go’s run­time does not sup­port that. In­stead, we use Type=notify. In this case, sys­temd ex­pects the dae­mon to sig­nal its readi­ness with a mes­sage to a Unix socket. go-sys­temd pack­age han­dles the de­tails for us:

package main

import (
    "log"
    "net"
    "net/http"

    "github.com/coreos/go-systemd/daemon"
)

func main() {
    l, err := net.Listen("tcp", ":8081")
    if err != nil {
        log.Panicf("cannot listen: %s", err)
    }
    daemon.SdNotify(false, daemon.SdNotifyReady) // ❶
    http.Serve(l, nil)                           // ❷
}

It’s im­por­tant to place the no­ti­fi­ca­tion af­ter net.Listen() (in ❶): if the no­ti­fi­ca­tion was sent ear­lier, a client would get “con­nec­tion re­fused” when try­ing to use the ser­vice. When a dae­mon lis­tens to a socket, con­nec­tions are queued by the ker­nel un­til the dae­mon is able to ac­cept them (in ❷).

If the ser­vice is not run through sys­temd, the added line is a no-op.

Liveness#

An­other in­ter­est­ing fea­ture of sys­temd is to watch the ser­vice and restart it if it hap­pens to crash (thanks to the Restart=on-failure di­rec­tive). It’s also pos­si­ble to use a watch­dog: the ser­vice sends watch­dog keep-alives at reg­u­lar in­ter­vals. If it fails to do so, sys­temd will restart it.

We could in­sert the fol­low­ing code just be­fore http.Serve() call:

go func() {
    interval, err := daemon.SdWatchdogEnabled(false)
    if err != nil || interval == 0 {
        return
    }
    for {
        daemon.SdNotify(false, daemon.SdNotifyWatchdog)
        time.Sleep(interval / 3)
    }
}()

How­ever, this doesn’t add much value: the gor­ou­tine is un­re­lated to the core busi­ness of the ser­vice. If for some rea­son, the HTTP part gets stuck, the gor­ou­tine will hap­pily con­tinue to send keep-alives to systemd.

In our ex­am­ple, we can just do an HTTP query be­fore send­ing the keep-alive. The in­ter­nal loop can be re­placed with this code:

for {
    _, err := http.Get("http://127.0.0.1:8081") // ❸
    if err == nil {
        daemon.SdNotify(false, daemon.SdNotifyWatchdog)
    }
    time.Sleep(interval / 3)
}

In ❸, we con­nect to the ser­vice to check if it’s still work­ing. If we get some kind of an­swer, we send a watch­dog keep-alive. If the ser­vice is un­avail­able or if http.Get() gets stuck, sys­temd will trig­ger a restart.

There is no uni­ver­sal recipe. How­ever, checks can be split into two groups:

  • Be­fore send­ing a keep-alive, you ex­e­cute an ac­tive check on the com­po­nents of your ser­vice. The keep-alive is sent only if all checks are suc­cess­ful. The checks can be in­ter­nal (like in the above ex­am­ple) or ex­ter­nal (for ex­am­ple, check with a query to the database).

  • Each com­po­nent re­ports its sta­tus, telling if it’s alive or not. Be­fore send­ing a keep-alive, you check the re­ported sta­tus of all com­po­nents (pas­sive check). If some com­po­nents are late or re­ported fa­tal er­rors, don’t send the keep-alive.

If pos­si­ble, re­cov­ery from er­rors (for ex­am­ple, with a back­off retry) and self-heal­ing (for ex­am­ple, by reestab­lish­ing a net­work con­nec­tion) is al­ways bet­ter, but the watch­dog is a good tool to han­dle the worst cases and avoid too com­plex re­cov­ery logic.

For ex­am­ple, if a com­po­nent doesn’t know how to re­cover from an ex­cep­tional con­di­tion,3 in­stead of us­ing panic(), it could sig­nal its sit­u­a­tion be­fore dy­ing. An­other ded­i­cated com­po­nent could try to re­solve the sit­u­a­tion by restart­ing the faulty com­po­nent. If it fails to reach a healthy state in time, the watch­dog timer will trig­ger and the whole ser­vice will be restarted.

Up­date (2018-03)

Have a look at “In­te­gra­tion of a Go ser­vice with sys­temd: socket ac­ti­va­tion” for a fol­lowup of this article.