Pragmatic Debian packaging

Vincent Bernat

Notice

This guide is an up­dated ver­sion of a pre­vi­ous edi­tion. If you need to tar­get dis­tri­bu­tions older than De­bian Stretch and Ubuntu Bionic, have a look at the older ver­sion instead.

While the cre­ation of De­bian pack­ages is abun­dantly doc­u­mented, most tu­to­ri­als are tar­geted to pack­ages im­ple­ment­ing the De­bian pol­icy. More­over, De­bian pack­ag­ing has a rep­u­ta­tion for be­ing un­nec­es­sar­ily dif­fi­cult1 and many peo­ple pre­fer to use less con­strained tools,2 like fpm or CheckInstall.

How­ever, build­ing De­bian pack­ages with the of­fi­cial tools can be­come straight­for­ward if you bend some rules:

  1. No source pack­age will be gen­er­ated. Pack­ages will be built di­rectly from a check­out of a VCS repository.

  2. Ad­di­tional de­pen­den­cies can be down­loaded dur­ing the build. Pack­ag­ing in­di­vid­u­ally each de­pen­dency is painstak­ing work, no­tably when you have to deal with some fast-paced ecosys­tems like Java, Java­Script, and Go.

  3. The pro­duced pack­ages may bun­dle de­pen­den­cies. This is likely to raise some con­cerns about se­cu­rity and long-term main­te­nance, but this is a com­mon trade-off in many ecosys­tems, no­tably Java, Java­Script and Go.

Pragmatic packages 101#

In the De­bian ar­chive, you have two kinds of pack­ages: the source pack­ages and the bi­nary pack­ages. Each bi­nary pack­age is built from a source pack­age. You need a name for each package.

As stated in the in­tro­duc­tion, we won’t gen­er­ate a source pack­age but we will work with its un­packed form which is any source tree con­tain­ing a debian/ di­rec­tory. In our ex­am­ples, we will start with a source tree con­tain­ing only a debian/ di­rec­tory but you are free to in­clude this debian/ di­rec­tory into an ex­ist­ing project.

As an ex­am­ple, we will pack­age mem­cached, a dis­trib­uted mem­ory cache. You need to cre­ate four files:

  • debian/compat;
  • debian/​changelog;
  • debian/​control; and
  • debian/rules.

The first one is easy. Put 11 in it:3

echo 11 > debian/compat

The sec­ond one has the fol­low­ing content:

memcached (0.0-0) UNRELEASED; urgency=medium

  * Fake entry

 -- Happy Packager <happy@example.com>  Tue, 19 Apr 2016 22:27:05 +0200

The only im­por­tant in­for­ma­tion is the name of the source pack­age, memcached, on the first line. Ev­ery­thing else can be left as is as it won’t in­flu­ence the gen­er­ated bi­nary packages.

The control file#

debian/​control de­scribes the meta­data of both the source pack­age and the gen­er­ated bi­nary pack­ages. We have to write a block for each of them.

Source: memcached
Maintainer: Vincent Bernat <bernat@debian.org>

Package: memcached
Architecture: any
Description: high-performance memory object caching system

The source pack­age is called memcached. We have to use the same name as in debian/​changelog.

We gen­er­ate only one bi­nary pack­age: memcached. In the re­main­der of the ex­am­ple, when you see memcached, this is the name of a bi­nary pack­age. The Architecture field should be set to ei­ther any or all. Use all ex­clu­sively if the pack­age con­tains only arch-in­de­pen­dent files. In doubt, just stick to any.

The Description field con­tains a short de­scrip­tion of the bi­nary package.

The build recipe#

The last manda­tory file is debian/​rules. It is the recipe for the pack­age. We need to re­trieve mem­cached, build it, and in­stall its file tree in debian/​memcached/. It looks like this:

#!/usr/bin/make -f

DISTRIBUTION = $(shell sed -n "s/^VERSION_CODENAME=//p" /etc/os-release)
VERSION = 1.6.6
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)

%:
    dh $@

override_dh_auto_clean:
override_dh_auto_test:
override_dh_auto_build:
override_dh_auto_install:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
    ./configure --prefix=/usr
    make
    make install DESTDIR=debian/memcached

override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)

The empty tar­gets override_​dh_​auto_​clean, override_​dh_​auto_​test and override_​dh_​auto_​build keep deb­helper from be­ing too smart. The override_​dh_​gencontrol tar­get sets the pack­age ver­sion4 with­out up­dat­ing debian/​changelog. If you ig­nore the slight boil­er­plate, the recipe is quite sim­i­lar to what you would have done with fpm:

DISTRIBUTION=$(sed -n "s/^VERSION_CODENAME=//p" /etc/os-release)
VERSION=1.6.6
PACKAGEVERSION=${VERSION}-0~${DISTRIBUTION}0
TARBALL=memcached-${VERSION}.tar.gz
URL=http://www.memcached.org/files/${TARBALL}

wget -N --progress=dot:mega ${URL}
tar --strip-components=1 -xf ${TARBALL}
./configure --prefix=/usr
make
make install DESTDIR=/tmp/installdir

# Build the final package
fpm -s dir -t deb \
    -n memcached \
    -v ${PACKAGEVERSION} \
    -C /tmp/installdir \
    --description "high-performance memory object caching system"

You can re­view the whole pack­age tree on GitHub and build it with the dpkg-buildpackage -us -uc -b com­mand or with GIT_​PBUILDER_​OPTIONS=--use-network=​yes DIST=​bionic git-pbuilder -us -uc -b if you al­ready set up a tool like pbuilder.

Pragmatic packages 102#

At this point, we can it­er­ate and add sev­eral im­prove­ments to our mem­cached pack­age. None of them are manda­tory but they are usu­ally worth the ad­di­tional effort.

Build dependencies#

Our ini­tial build recipe only works when sev­eral pack­ages are in­stalled, like wget and libevent-dev. They are not present on all De­bian sys­tems. You can eas­ily ex­press that you need them by adding a Build-Depends sec­tion for the source pack­age in debian/​control:

Source: memcached
Build-Depends: debhelper (>= 11),
               wget, ca-certificates,
               libevent-dev

Al­ways spec­ify the debhelper (>= 11) de­pen­dency as we heav­ily rely on it. We don’t re­quire make or a C com­piler be­cause it is as­sumed that the build-essential meta-pack­age is in­stalled and it pulls them. dpkg-buildpackage will com­plain if the de­pen­den­cies are not met. If you want to in­stall these pack­ages from your CI sys­tem, you can use the fol­low­ing command:5

mk-build-deps \
    -t 'apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -qqy' \
    -i -r debian/control

You may also want to in­ves­ti­gate pbuilder, sbuild or whale­builder, three tools to build De­bian pack­ages in a clean iso­lated en­vi­ronment.

Runtime dependencies#

If the re­sult­ing pack­age is in­stalled on a freshly in­stalled ma­chine, it won’t work be­cause it will be miss­ing libevent, a re­quired li­brary for mem­cached. You can ex­press the de­pen­den­cies needed by each bi­nary pack­age by adding a Depends field. More­over, for dy­namic li­braries, you can au­to­mat­i­cally get the right de­pen­den­cies by us­ing some sub­sti­tu­tion variables:

Package: memcached
Depends: ${misc:Depends}, ${shlibs:Depends}

The re­sult­ing pack­age will con­tain the fol­low­ing in­formation:

$ dpkg -I ../memcached_1.6.6-0\~buster0_amd64.deb | grep Depends
 Depends: libc6 (>= 2.17), libevent-2.1-6 (>= 2.1.8-stable)

Integration with the init system#

Most pack­aged dae­mons come with some in­te­gra­tion with the init sys­tem. This in­te­gra­tion en­sures the dae­mon will be started on boot and restarted on up­grade. While De­bian still sup­ports sev­eral init sys­tems, in my opin­ion, the most prag­matic one is systemd.

The con­tent of a sys­temd unit should go in debian/​memcached.​service. For example:

[Unit]
Description=memcached daemon
After=network.target

[Service]
Type=forking
Environment=PORT=11211
Environment=MAXCONN=1024
Environment=CACHESIZE=64
Environment=OPTIONS=
ExecStart=/usr/bin/memcached -d -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
Restart=on-failure
User=_memcached
DynamicUser=yes

[Install]
WantedBy=multi-user.target

The Type di­rec­tive is quite im­por­tant. We used forking as mem­cached is started with the -d flag and will fork when it is ready to ac­cept re­quests. If you use a non-fork­ing dae­mon, you can ei­ther use notify for a dae­mon with some sup­port for sys­temd or simple otherwise.

A user can cus­tom­ize one of the Environment di­rec­tives by us­ing systemctl edit memcached.​service to cre­ate an over­ride which will be in­stalled in /etc/​systemd/​system/​memcached.​service.​d/override.​conf. systemctl cat memcached.​service can be used to check the fi­nal def­i­ni­tion of the unit.

The DynamicUser di­rec­tive is also quite in­ter­est­ing. sys­temd au­to­mat­i­cally cre­ates a _memcached user6 and runs the dae­mon as this user. This frees us from man­ag­ing a sys­tem user ourselves!

You can re­view the whole pack­age tree on GitHub and build it with the dpkg-buildpackage -us -uc -b command.

Pragmatic packages 103#

It is pos­si­ble to lever­age deb­helper to re­duce the recipe size and to make it more declar­a­tive. This sec­tion is quite op­tional and it re­quires un­der­stand­ing a bit more how a De­bian pack­age is built. Feel free to skip it.

The big picture#

There are four steps to build a reg­u­lar De­bian package:

  1. debian/​rules clean cleans the source tree to make it pristine.

  2. debian/​rules build builds the code. For an au­to­conf-based soft­ware, like mem­cached, this step should ex­e­cute some­thing like ./configure && make.

  3. debian/​rules install in­stalls files at the cor­rect lo­ca­tion for each bi­nary pack­age. For an au­to­conf-based soft­ware, this step should ex­e­cute make install DESTDIR=​debian/​memcached.

  4. debian/​rules binary packs the dif­fer­ent file trees into bi­nary packages.

You don’t di­rectly write each of these tar­gets. In­stead, you let dh, a com­po­nent of deb­helper, do most of the work. The fol­low­ing debian/​rules file should do al­most ev­ery­thing cor­rectly with many source packages:

#!/usr/bin/make -f
%:
    dh $@

For each of the four tar­gets de­scribed above, you can run dh with --no-act to see what it would do. For example:

$ dh build --no-act
   dh_testdir
   dh_update_autotools_config
   dh_auto_configure
   dh_auto_build
   dh_auto_test

Each of these helpers has a man­ual page. Helpers start­ing with dh_​auto_ are a bit “magic.” For ex­am­ple, dh_​auto_​configure will try to au­to­mat­i­cally con­fig­ure a pack­age be­fore build­ing: it will de­tect the build sys­tem and in­voke ./configure, cmake, or Makefile.PL.

If one of the helpers does not do the “right” thing, you can re­place it by us­ing an over­ride target:

override_dh_auto_configure:
    ./configure --with-some-grog

These helpers are also con­fig­urable, so you can just al­ter a bit their be­hav­ior by in­vok­ing them with ad­di­tional options:

override_dh_auto_configure:
    dh_auto_configure -- --with-some-grog

This way, ./configure will be called with your cus­tom flag but also with a lot of de­fault flags like --prefix=/​usr for bet­ter in­te­gra­tion. A man­ual page is avail­able for each tool.

In the ini­tial mem­cached ex­am­ple, we over­rode all these “magic” tar­gets. dh_​auto_​clean, dh_​auto_​configure, and dh_​auto_​build are con­verted to no-ops to avoid any un­ex­pected be­hav­ior. dh_​auto_​install is hi­jacked to do all the build process. Ad­di­tion­ally, we mod­i­fied the be­hav­ior of the dh_​gencontrol helper by forc­ing the ver­sion num­ber in­stead of us­ing the one from debian/​changelog.

Automatic builds#

As mem­cached is an au­to­conf-en­abled pack­age, dh knows how to build it: ./configure && make && make install. There­fore, we can let it han­dle most of the work with this debian/​rules file:

#!/usr/bin/make -f

DISTRIBUTION = $(shell sed -n "s/^VERSION_CODENAME=//p" /etc/os-release)
VERSION = 1.6.6
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)

%:
    dh $@ --with systemd

override_dh_update_autotools_config:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)

override_dh_auto_test:
    # Don't run the whitespace test
    rm t/whitespace.t
    dh_auto_test

override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)

The dh_​update_​autotools_​config tar­get is hi­jacked to down­load and set up the source tree. We do not over­ride the dh_​auto_​configure step, so dh will ex­e­cute the ./configure script with the ap­pro­pri­ate op­tions. We do not over­ride the dh_​auto_​build step ei­ther: dh will ex­e­cute make. dh_​auto_​test is in­voked af­ter the build and it will run the mem­cached test suite. We need to over­ride it be­cause one of the tests is com­plain­ing about odd white­spaces in the debian/ di­rec­tory. We sup­press this rogue test and let dh_​auto_​test ex­e­cute the test suite. dh_​auto_​install is not over­rid­den ei­ther, so dh will ex­e­cute some vari­ant of make install.

To get a bet­ter sense of the dif­fer­ence, here is a diff:

--- memcached-intermediate/debian/rules 2019-05-31 07:52:40.908868035 +0200
+++ memcached/debian/rules      2019-05-31 07:28:17.404380064 +0200
@@ -9,15 +9,14 @@
 %:
    dh $@

-override_dh_auto_clean:
-override_dh_auto_test:
-override_dh_auto_build:
-override_dh_auto_install:
+override_dh_update_autotools_config:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
-   ./configure --prefix=/usr
-   make
-   make install DESTDIR=debian/memcached
+
+override_dh_auto_test:
+   # Don't run the whitespace test
+   rm t/whitespace.t
+   dh_auto_test

 override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)

It is up to you to de­cide if dh can do some work for you, but you could try to start from a min­i­mal debian/​rules and only over­ride some targets.

Install additional files#

While make install in­stalled the es­sen­tial files for mem­cached, you may want to put ad­di­tional files in the bi­nary pack­age. You could use cp in your build recipe, but you can also de­clare them:

  • files listed in debian/​memcached.​docs will be copied to /usr/​share/​doc/​memcached by dh_​installdocs
  • files listed in debian/​memcached.​examples will be copied to /usr/​share/​doc/​memcached/​examples by dh_​installexamples
  • files listed in debian/​memcached.​manpages will be copied to the ap­pro­pri­ate sub­di­rec­tory of /usr/​share/​man by dh_​installman

Here is an ex­am­ple us­ing wild­cards for debian/​memcached.docs:

doc/*.txt

If you need to copy some files to an ar­bi­trary lo­ca­tion, you can list them along with their des­ti­na­tion di­rec­to­ries in debian/​memcached.​install and dh_​install will take care of the copy. Here is an example:

scripts/memcached-tool usr/bin

Us­ing these files makes the build process more declar­a­tive. It is a mat­ter of taste, and you are free to use cp in debian/​rules in­stead. You can re­view the whole pack­age tree on GitHub.

Other examples#

The Git repos­i­tory con­tains some ad­di­tional ex­am­ples. They all fol­low the same scheme:

  • dh_​update_​autotools_​config is hi­jacked to down­load and set up the source tree
  • dh_​gencontrol is mod­i­fied to use a com­puted version

No­tably, you will find dae­mons in Java, Go, Python and Node.js. The goal of these ex­am­ples is to demon­strate that us­ing De­bian tools to build De­bian pack­ages can be straight­for­ward. Hope this helps.

Up­date (2019-05)

This guide has been up­dated to use override_​dh_​update_​autotools_​config to fetch source files in­stead of override_​dh_​auto_​clean as it in­ter­acts bet­ter with most build tools, like pbuilder and sbuild.