Pragmatic Debian packaging (2016)

Vincent Bernat

Up­date (2019-05)

Have a look at the 2019 edi­tion of this guide if you tar­get only re­cent dis­tri­bu­tions (De­bian Stretch and Ubuntu Bionic).

While the cre­ation of De­bian pack­ages is abun­dantly doc­u­mented, most tu­to­ri­als are tar­geted to pack­ages im­ple­ment­ing the De­bian pol­icy. More­over, De­bian pack­ag­ing has a rep­u­ta­tion of be­ing un­nec­es­sar­ily dif­fi­cult1 and many peo­ple pre­fer to use less con­strained tools2 like fpm or CheckInstall.

How­ever, build­ing De­bian pack­ages with the of­fi­cial tools can be­come straight­for­ward if you bend some rules:

  1. No source pack­age will be gen­er­ated. Pack­ages will be built di­rectly from a check­out of a VCS repository.

  2. Ad­di­tional de­pen­den­cies can be down­loaded dur­ing build. Pack­ag­ing in­di­vid­u­ally each de­pen­dency is a painstak­ing work, no­tably when you have to deal with some fast-paced ecosys­tems like Java, JavaScript and Go.

  3. The pro­duced pack­ages may bun­dle de­pen­den­cies. This is likely to raise some con­cerns about se­cu­rity and long-term main­te­nance, but this is a com­mon trade-off in many ecosys­tems, no­tably Java, JavaScript and Go.

Pragmatic packages 101#

In the De­bian ar­chive, you have two kinds of pack­ages: the source pack­ages and the bi­nary pack­ages. Each bi­nary pack­age is built from a source pack­age. You need a name for each package.

As stated in the in­tro­duc­tion, we won’t gen­er­ate a source pack­age but we will work with its un­packed form which is any source tree con­tain­ing a debian/ di­rec­tory. In our ex­am­ples, we will start with a source tree con­tain­ing only a debian/ di­rec­tory but you are free to in­clude this debian/ di­rec­tory into an ex­ist­ing project.

As an ex­am­ple, we will pack­age mem­cached, a dis­trib­uted mem­ory cache. There are four files to create:

  • debian/compat;
  • debian/changelog;
  • debian/control; and
  • debian/rules.

The first one is easy. Put 9 in it:3

echo 9 > debian/compat

The sec­ond one has the fol­low­ing content:

memcached (0.0-0) UNRELEASED; urgency=medium

  * Fake entry

 -- Happy Packager <happy@example.com>  Tue, 19 Apr 2016 22:27:05 +0200

The only im­por­tant in­for­ma­tion is the name of the source pack­age, memcached, on the first line. Ev­ery­thing else can be left as is as it won’t in­flu­ence the gen­er­ated bi­nary packages.

The control file#

debian/control de­scribes the meta­data of both the source pack­age and the gen­er­ated bi­nary pack­ages. We have to write a block for each of them.

Source: memcached
Maintainer: Vincent Bernat <bernat@debian.org>

Package: memcached
Architecture: any
Description: high-performance memory object caching system

The source pack­age is called memcached. We have to use the same name as in debian/changelog.

We gen­er­ate only one bi­nary pack­age: memcached. In the re­main­ing of the ex­am­ple, when you see memcached, this is the name of a bi­nary pack­age. The Architecture field should be set to ei­ther any or all. Use all ex­clu­sively if the pack­age con­tains only arch-in­de­pen­dent files. In doubt, just stick to any.

The Description field con­tains a short de­scrip­tion of the bi­nary package.

The build recipe#

The last manda­tory file is debian/rules. It’s the recipe of the pack­age. We need to re­trieve mem­cached, build it and in­stall its file tree in debian/memcached/. It looks like this:

#!/usr/bin/make -f

DISTRIBUTION = $(shell lsb_release -sr)
VERSION = 1.4.25
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)

%:
    dh $@

override_dh_auto_clean:
override_dh_auto_test:
override_dh_auto_build:
override_dh_auto_install:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
    ./configure --prefix=/usr
    make
    make install DESTDIR=debian/memcached

override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)

The empty tar­gets override_dh_auto_clean, override_dh_auto_test and override_dh_auto_build keep deb­helper from be­ing too smart. The override_dh_gencontrol tar­get sets the pack­age ver­sion4 with­out up­dat­ing debian/changelog. If you ig­nore the slight boil­er­plate, the recipe is quite sim­i­lar to what you would have done with fpm:

DISTRIBUTION=$(lsb_release -sr)
VERSION=1.4.25
PACKAGEVERSION=${VERSION}-0~${DISTRIBUTION}0
TARBALL=memcached-${VERSION}.tar.gz
URL=http://www.memcached.org/files/${TARBALL}

wget -N --progress=dot:mega ${URL}
tar --strip-components=1 -xf ${TARBALL}
./configure --prefix=/usr
make
make install DESTDIR=/tmp/installdir

# Build the final package
fpm -s dir -t deb \
    -n memcached \
    -v ${PACKAGEVERSION} \
    -C /tmp/installdir \
    --description "high-performance memory object caching system"

You can re­view the whole pack­age tree on GitHub and build it with dpkg-buildpackage -us -uc -b.

Pragmatic packages 102#

At this point, we can it­er­ate and add sev­eral im­prove­ments to our mem­cached pack­age. None of them are manda­tory but they are usu­ally worth the ad­di­tional effort.

Build dependencies#

Our ini­tial build recipe only works when sev­eral pack­ages are in­stalled, like wget and libevent-dev. They are not present on all De­bian sys­tems. You can eas­ily ex­press that you need them by adding a Build-Depends sec­tion for the source pack­age in debian/control:

Source: memcached
Build-Depends: debhelper (>= 9),
               wget, ca-certificates, lsb-release,
               libevent-dev

Al­ways spec­ify the debhelper (>= 9) de­pen­dency as we heav­ily rely on it. We don’t re­quire make or a C com­piler be­cause it is as­sumed that the build-essential meta-pack­age is in­stalled and it pulls them. dpkg-buildpackage will com­plain if the de­pen­den­cies are not met. If you want to in­stall these pack­ages from your CI sys­tem, you can use the fol­low­ing command:5

mk-build-deps \
    -t 'apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -qqy' \
    -i -r debian/control

You may also want to in­ves­ti­gate pbuilder or sbuild, two tools to build De­bian pack­ages in a clean iso­lated en­vi­ronment.

Runtime dependencies#

If the re­sult­ing pack­age is in­stalled on a freshly in­stalled ma­chine, it won’t work be­cause it will be miss­ing libevent, a re­quired li­brary for mem­cached. You can ex­press the de­pen­den­cies needed by each bi­nary pack­age by adding a Depends field. More­over, for dy­namic li­braries, you can au­to­mat­i­cally get the right de­pen­den­cies by us­ing some sub­sti­tu­tion variables:

Package: memcached
Depends: ${misc:Depends}, ${shlibs:Depends}

The re­sult­ing pack­age will con­tain the fol­low­ing in­formation:

$ dpkg -I ../memcached_1.4.25-0\~unstable0_amd64.deb | grep Depends
 Depends: libc6 (>= 2.17), libevent-2.0-5 (>= 2.0.10-stable)

Integration with init system#

Most pack­aged dae­mons come with some in­te­gra­tion with the init sys­tem. This in­te­gra­tion en­sures the dae­mon will be started on boot and restarted on up­grade. For De­bian-based dis­tri­bu­tions, there are sev­eral init sys­tems avail­able. The most prom­i­nent ones are:

  • Sys­tem-V init is the his­tor­i­cal init sys­tem. More mod­ern inits are able to re­use scripts writ­ten for this init, so this is a safe com­mon de­nom­i­na­tor for pack­aged daemons.
  • Up­start is the less-his­tor­i­cal init sys­tem for Ubuntu (used in Ubuntu 14.10 and pre­vi­ous re­leases).
  • sys­temd is the de­fault init sys­tem for De­bian since Jessie and for Ubuntu since 15.04.

Writ­ing a cor­rect script for the Sys­tem-V init is er­ror-prone. There­fore, I usu­ally pre­fer to pro­vide a na­tive con­fig­u­ra­tion file for the de­fault init sys­tem of the tar­geted dis­tri­bu­tion (Up­start and systemd).

System-V#

If you want to pro­vide a Sys­tem-V init script, have a look at /etc/init.d/skeleton on the most an­cient dis­tri­bu­tion you want to tar­get and adapt it.6 Put the re­sult in debian/memcached.init. It will be in­stalled at the right place, in­voked on in­stall, up­grade and re­moval. On De­bian-based sys­tems, many init scripts al­low user cus­tomiza­tions by pro­vid­ing a /etc/default/memcached file. You can ship one by putting its con­tent in debian/memcached.default.

Upstart#

Pro­vid­ing an Up­start job is sim­i­lar: put it in debian/memcached.upstart. For example:

description "memcached daemon"

start on runlevel [2345]
stop on runlevel [!2345]
respawn
respawn limit 5 60
expect daemon

script
  . /etc/default/memcached
  exec memcached -d -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
end script

When writ­ing an Up­start job, the most im­por­tant di­rec­tive is ex­pect. Be sure to get it right. Here, we use expect daemon and mem­cached is started with the -d flag.

systemd#

Pro­vid­ing a sys­temd unit is a bit more com­plex. The con­tent of the file should go in debian/memcached.service. For example:

[Unit]
Description=memcached daemon
After=network.target

[Service]
Type=forking
EnvironmentFile=/etc/default/memcached
ExecStart=/usr/bin/memcached -d -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
Restart=on-failure

[Install]
WantedBy=multi-user.target

We re­use /etc/default/memcached even if it is not con­sid­ered a good prac­tice with sys­temd.7 Like for Up­start, the di­rec­tive Type is quite im­por­tant. We used forking as mem­cached is started with the -d flag.

Up­date (2017-11)

When tar­get­ing De­bian Stretch, Ubuntu Yaketty or more re­cent, put 10 in debian/compat and ig­nore the fol­low­ing in­structions.

You also need to add a build-de­pen­dency to dh-systemd in debian/control:

Source: memcached
Build-Depends: debhelper (>= 9),
               wget, ca-certificates, lsb-release,
               libevent-dev,
               dh-systemd

And you need to mod­ify the de­fault rule in debian/rules:

%:
    dh $@ --with systemd

The ex­tra com­plex­ity is a bit un­for­tu­nate but sys­temd in­te­gra­tion was not part of deb­helper.8 With­out these ad­di­tional mod­i­fi­ca­tions, the unit will get in­stalled but you won’t get a proper in­te­gra­tion and the ser­vice won’t be en­abled on in­stall or boot.

Dedicated user#

Many dae­mons don’t need to run as root and it is a good prac­tice to ship a ded­i­cated user. In the case of mem­cached, we can pro­vide a _memcached user.9

Up­date (2018-11)

If you only care about sys­temd, you may save time and en­ergy by us­ing its dy­namic users fea­ture instead.

Add a debian/memcached.postinst file with the fol­low­ing content:

#!/bin/sh

set -e

case "$1" in
    configure)
        adduser --system --disabled-password --disabled-login --home /var/empty \
                --no-create-home --quiet --force-badname --group _memcached
        ;;
esac

#DEBHELPER#

exit 0

There is no cleanup of the user when the pack­age is re­moved for two reasons:

  1. Less stuff to write.
  2. The user could still own some files.

The util­ity adduser will do the right thing what­ever the re­quested user al­ready ex­ists or not. You need to add it as a de­pen­dency in debian/control:

Package: memcached
Depends: ${misc:Depends}, ${shlibs:Depends}, adduser

The #DEBHELPER# marker is im­por­tant as it will be re­placed by some code to han­dle the ser­vice con­fig­u­ra­tion files (or some other stuff).

You can re­view the whole pack­age tree on GitHub and build it with dpkg-buildpackage -us -uc -b.

Pragmatic packages 103#

It is pos­si­ble to lever­age deb­helper to re­duce the recipe size and to make it more declar­a­tive. This sec­tion is quite op­tional and it re­quires un­der­stand­ing a bit more how a De­bian pack­age is built. Feel free to skip it.

The big picture#

There are four steps to build a reg­u­lar De­bian package:

  1. debian/rules clean should clean the source tree to make it pristine.

  2. debian/rules build should trig­ger the build. For an au­to­conf-based soft­ware, like mem­cached, this step should ex­e­cute some­thing like ./configure && make.

  3. debian/rules install should in­stall the file tree of each bi­nary pack­age. For an au­to­conf-based soft­ware, this step should ex­e­cute make install DESTDIR=debian/memcached.

  4. debian/rules binary will pack the dif­fer­ent file trees into bi­nary packages.

You don’t di­rectly write each of these tar­gets. In­stead, you let dh, a com­po­nent of deb­helper, do most of the work. The fol­low­ing debian/rules file should do al­most ev­ery­thing cor­rectly with many source packages:

#!/usr/bin/make -f
%:
    dh $@

For each of the four tar­gets de­scribed above, you can run dh with --no-act to see what it would do. For example:

$ dh build --no-act
   dh_testdir
   dh_update_autotools_config
   dh_auto_configure
   dh_auto_build
   dh_auto_test

Each of these helpers has a man­ual page. Helpers start­ing with dh_auto_ are a bit “magic.” For ex­am­ple, dh_auto_configure will try to au­to­mat­i­cally con­fig­ure a pack­age prior to build­ing: it will de­tect the build sys­tem and in­voke ./configure, cmake or Makefile.PL.

If one of the helpers does not do the “right” thing, you can re­place it by us­ing an over­ride target:

override_dh_auto_configure:
    ./configure --with-some-grog

These helpers are also con­fig­urable, so you can just al­ter a bit their be­hav­ior by in­vok­ing them with ad­di­tional options:

override_dh_auto_configure:
    dh_auto_configure -- --with-some-grog

This way, ./configure will be called with your cus­tom flag but also with a lot of de­fault flags like --prefix=/usr for bet­ter in­te­gration.

In the ini­tial mem­cached ex­am­ple, we over­rode all these “magic” tar­gets. dh_auto_clean, dh_auto_configure and dh_auto_build are con­verted to no-ops to avoid any un­ex­pected be­hav­ior. dh_auto_install is hi­jacked to do all the build process. Ad­di­tion­ally, we mod­i­fied the be­hav­ior of the dh_gencontrol helper by forc­ing the ver­sion num­ber in­stead of us­ing the one from debian/changelog.

Automatic builds#

As mem­cached is an au­to­conf-en­abled pack­age, dh knows how to build it: ./configure && make && make install. There­fore, we can let it han­dle most of the work with this debian/rules file:

#!/usr/bin/make -f

DISTRIBUTION = $(shell lsb_release -sr)
VERSION = 1.4.25
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)

%:
    dh $@ --with systemd

override_dh_auto_clean:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)

override_dh_auto_test:
    # Don't run the whitespace test
    rm t/whitespace.t
    dh_auto_test

override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)

The dh_auto_clean tar­get is hi­jacked to down­load and setup the source tree.10 We do not over­ride the dh_auto_configure step, so dh will ex­e­cute the ./configure script with the ap­pro­pri­ate op­tions. We do not over­ride the dh_auto_build step ei­ther: dh will ex­e­cute make. dh_auto_test is in­voked af­ter the build and it will run the mem­cached test suite. We need to over­ride it be­cause one of the tests is com­plain­ing about odd white­spaces in the debian/ di­rec­tory. We sup­press this rogue test and let dh_auto_test ex­e­cute the test suite. dh_auto_install is not over­rid­den ei­ther, so dh will ex­e­cute some vari­ant of make install.

To get a bet­ter sense of the dif­fer­ence, here is a diff:

--- memcached-intermediate/debian/rules 2016-04-30 14:02:37.425593362 +0200
+++ memcached/debian/rules  2016-05-01 14:55:15.815063835 +0200
@@ -12,10 +12,9 @@
 override_dh_auto_clean:
-override_dh_auto_test:
-override_dh_auto_build:
-override_dh_auto_install:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
-   ./configure --prefix=/usr
-   make
-   make install DESTDIR=debian/memcached
+
+override_dh_auto_test:
+   # Don't run the whitespace test
+   rm t/whitespace.t
+   dh_auto_test

It is up to you to de­cide if dh can do some work for you, but you could try to start from a min­i­mal debian/rules and only over­ride some targets.

Install additional files#

While make install in­stalled the es­sen­tial files for mem­cached, you may want to put ad­di­tional files in the bi­nary pack­age. You could use cp in your build recipe, but you can also de­clare them:

  • files listed in debian/memcached.docs will be copied to /usr/share/doc/memcached by dh_installdocs
  • files listed in debian/memcached.examples will be copied to /usr/share/doc/memcached/examples by dh_installexamples
  • files listed in debian/memcached.manpages will be copied to the ap­pro­pri­ate sub­di­rec­tory of /usr/share/man by dh_installman

Here is an ex­am­ple us­ing wild­cards for debian/memcached.docs:

doc/*.txt

If you need to copy some files to an ar­bi­trary lo­ca­tion, you can list them along with their des­ti­na­tion di­rec­to­ries in debian/memcached.install and dh_install will take care of the copy. Here is an example:

scripts/memcached-tool usr/bin

Us­ing these files make the build process more declar­a­tive. It is a mat­ter of taste and you are free to use cp in debian/rules in­stead. You can re­view the whole pack­age tree on GitHub.

Other examples#

The Git repos­i­tory con­tains some ad­di­tional ex­am­ples. They all fol­low the same scheme:

  • dh_auto_clean is hi­jacked to down­load and setup the source tree
  • dh_gencontrol is mod­i­fied to use a com­puted version

No­tably, you’ll find dae­mons in Java, Go, Python and Node.js. The goal of these ex­am­ples is to demon­strate that us­ing De­bian tools to build De­bian pack­ages can be straight­for­ward. Hope this helps.