While the creation of Debian packages is abundantly documented, most tutorials are targeted to packages implementing the Debian policy. Moreover, Debian packaging has a reputation for being unnecessarily difficult¹ and many people prefer to use less constrained tools² like fpm or CheckInstall.

However, building Debian packages with the official tools can become straightforward if you bend some rules:

1. No source package will be generated. Packages will be built directly from a checkout of a VCS repository.

2. Additional dependencies can be downloaded during the build. Packaging individually each dependency is painstaking work, notably when you have to deal with some fast-paced ecosystems like Java, JavaScript, and Go.

3. The produced packages may bundle dependencies. This is likely to raise some concerns about security and long-term maintenance, but this is a common trade-off in many ecosystems, notably Java, JavaScript and Go.

Pragmatic packages 101#

In the Debian archive, you have two kinds of packages: the source packages and the binary packages. Each binary package is built from a source package. You need a name for each package.

As stated in the introduction, we won’t generate a source package but we will work with its unpacked form which is any source tree containing a debian/ directory. In our examples, we will start with a source tree containing only a debian/ directory but you are free to include this debian/ directory into an existing project.

As an example, we will package memcached, a distributed memory cache. You need to create four files:

• debian/compat;
• debian/changelog;
• debian/control; and
• debian/rules.

The first one is easy. Put 11 in it:³

echo 11 > debian/compat

The second one has the following content:

memcached (0.0-0) UNRELEASED; urgency=medium

  * Fake entry

 -- Happy Packager <happy@example.com>  Tue, 19 Apr 2016 22:27:05 +0200

The only important information is the name of the source package, memcached, on the first line. Everything else can be left as is as it won’t influence the generated binary packages.

The control file#

debian/control describes the metadata of both the source package and the generated binary packages. We have to write a block for each of them.

Source: memcached
Maintainer: Vincent Bernat <bernat@debian.org>

Package: memcached
Architecture: any
Description: high-performance memory object caching system

The source package is called memcached. We have to use the same name as in debian/changelog.

We generate only one binary package: memcached. In the remaining of the example, when you see memcached, this is the name of a binary package. The Architecture field should be set to either any or all. Use all exclusively if the package contains only arch-independent files. In doubt, just stick to any.

The Description field contains a short description of the binary package.

The build recipe#

The last mandatory file is debian/rules. It’s the recipe for the package. We need to retrieve memcached, build it, and install its file tree in debian/memcached/. It looks like this:

#!/usr/bin/make -f

DISTRIBUTION = $(shell sed -n "s/^VERSION_CODENAME=//p" /etc/os-release)
VERSION = 1.6.6
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)

%:
    dh $@

override_dh_auto_clean:
override_dh_auto_test:
override_dh_auto_build:
override_dh_auto_install:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
    ./configure --prefix=/usr
    make
    make install DESTDIR=debian/memcached

override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)

The empty targets override_dh_auto_clean, override_dh_auto_test and override_dh_auto_build keep debhelper from being too smart. The override_dh_gencontrol target sets the package version⁴ without updating debian/changelog. If you ignore the slight boilerplate, the recipe is quite similar to what you would have done with fpm:

DISTRIBUTION=$(sed -n "s/^VERSION_CODENAME=//p" /etc/os-release)
VERSION=1.6.6
PACKAGEVERSION=${VERSION}-0~${DISTRIBUTION}0
TARBALL=memcached-${VERSION}.tar.gz
URL=http://www.memcached.org/files/${TARBALL}

wget -N --progress=dot:mega ${URL}
tar --strip-components=1 -xf ${TARBALL}
./configure --prefix=/usr
make
make install DESTDIR=/tmp/installdir

# Build the final package
fpm -s dir -t deb \
    -n memcached \
    -v ${PACKAGEVERSION} \
    -C /tmp/installdir \
    --description "high-performance memory object caching system"

You can review the whole package tree on GitHub and build it with the dpkg-buildpackage -us -uc -b command or with GIT_PBUILDER_OPTIONS=--use-network=yes DIST=bionic git-pbuilder -us -uc -b if you already setup a tool like pbuilder.

Pragmatic packages 102#

At this point, we can iterate and add several improvements to our memcached package. None of them are mandatory but they are usually worth the additional effort.

Build dependencies#

Our initial build recipe only works when several packages are installed, like wget and libevent-dev. They are not present on all Debian systems. You can easily express that you need them by adding a Build-Depends section for the source package in debian/control:

Source: memcached
Build-Depends: debhelper (>= 11),
               wget, ca-certificates,
               libevent-dev

Always specify the debhelper (>= 11) dependency as we heavily rely on it. We don’t require make or a C compiler because it is assumed that the build-essential meta-package is installed and it pulls them. dpkg-buildpackage will complain if the dependencies are not met. If you want to install these packages from your CI system, you can use the following command:⁵

mk-build-deps \
    -t 'apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -qqy' \
    -i -r debian/control

You may also want to investigate pbuilder, sbuild or whalebuilder, three tools to build Debian packages in a clean isolated environment.

Runtime dependencies#

If the resulting package is installed on a freshly installed machine, it won’t work because it will be missing libevent, a required library for memcached. You can express the dependencies needed by each binary package by adding a Depends field. Moreover, for dynamic libraries, you can automatically get the right dependencies by using some substitution variables:

Package: memcached
Depends: ${misc:Depends}, ${shlibs:Depends}

The resulting package will contain the following information:

$ dpkg -I ../memcached_1.6.6-0\~buster0_amd64.deb | grep Depends
 Depends: libc6 (>= 2.17), libevent-2.1-6 (>= 2.1.8-stable)

Integration with the init system#

Most packaged daemons come with some integration with the init system. This integration ensures the daemon will be started on boot and restarted on upgrade. While Debian still supports several init systems, in my opinion, the most pragmatic one is systemd.

The content of a systemd unit should go in debian/memcached.service. For example:

[Unit]
Description=memcached daemon
After=network.target

[Service]
Type=forking
Environment=PORT=11211
Environment=MAXCONN=1024
Environment=CACHESIZE=64
Environment=OPTIONS=
ExecStart=/usr/bin/memcached -d -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
Restart=on-failure
User=_memcached
DynamicUser=yes

[Install]
WantedBy=multi-user.target

The Type directive is quite important. We used forking as memcached is started with the -d flag and will fork when it is ready to accept requests. If you use a non-forking daemon, you can either use notify for a daemon with some support for systemd or simple otherwise.

A user can customize one of the Environment directive by using systemctl edit memcached.service to create an override which will be installed in /etc/systemd/system/memcached.service.d/override.conf. systemctl cat memcached.service can be used to check the final definition of the unit.

The DynamicUser directive is also quite interesting. systemd will automatically create a _memcached user⁶ and will run the daemon as this user. This frees us from managing a system user ourselves!

You can review the whole package tree on GitHub and build it with the dpkg-buildpackage -us -uc -b command.

Pragmatic packages 103#

It is possible to leverage debhelper to reduce the recipe size and to make it more declarative. This section is quite optional and it requires understanding a bit more how a Debian package is built. Feel free to skip it.

The big picture#

There are four steps to build a regular Debian package:

1. debian/rules clean cleans the source tree to make it pristine.

2. debian/rules build builds the code. For an autoconf-based software, like memcached, this step should execute something like ./configure && make.

3. debian/rules install installs files at the correct location for each binary package. For an autoconf-based software, this step should execute make install DESTDIR=debian/memcached.

4. debian/rules binary packs the different file trees into binary packages.

You don’t directly write each of these targets. Instead, you let dh, a component of debhelper, do most of the work. The following debian/rules file should do almost everything correctly with many source packages:

#!/usr/bin/make -f
%:
    dh $@

For each of the four targets described above, you can run dh with --no-act to see what it would do. For example:

$ dh build --no-act
   dh_testdir
   dh_update_autotools_config
   dh_auto_configure
   dh_auto_build
   dh_auto_test

Each of these helpers has a manual page. Helpers starting with dh_auto_ are a bit “magic.” For example, dh_auto_configure will try to automatically configure a package before building: it will detect the build system and invoke ./configure, cmake, or Makefile.PL.

If one of the helpers do not do the “right” thing, you can replace it by using an override target:

override_dh_auto_configure:
    ./configure --with-some-grog

These helpers are also configurable, so you can just alter a bit their behavior by invoking them with additional options:

override_dh_auto_configure:
    dh_auto_configure -- --with-some-grog

This way, ./configure will be called with your custom flag but also with a lot of default flags like --prefix=/usr for better integration. A manual page is available for each tool.

In the initial memcached example, we overrode all these “magic” targets. dh_auto_clean, dh_auto_configure, and dh_auto_build are converted to no-ops to avoid any unexpected behavior. dh_auto_install is hijacked to do all the build process. Additionally, we modified the behavior of the dh_gencontrol helper by forcing the version number instead of using the one from debian/changelog.

Automatic builds#

As memcached is an autoconf-enabled package, dh knows how to build it: ./configure && make && make install. Therefore, we can let it handle most of the work with this debian/rules file:

#!/usr/bin/make -f

DISTRIBUTION = $(shell sed -n "s/^VERSION_CODENAME=//p" /etc/os-release)
VERSION = 1.6.6
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)

%:
    dh $@ --with systemd

override_dh_update_autotools_config:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)

override_dh_auto_test:
    # Don't run the whitespace test
    rm t/whitespace.t
    dh_auto_test

override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)

The dh_update_autotools_config target is hijacked to download and setup the source tree. We don’t override the dh_auto_configure step, so dh will execute the ./configure script with the appropriate options. We don’t override the dh_auto_build step either: dh will execute make. dh_auto_test is invoked after the build and it will run the memcached test suite. We need to override it because one of the tests is complaining about odd whitespaces in the debian/ directory. We suppress this rogue test and let dh_auto_test execute the test suite. dh_auto_install is not overridden either, so dh will execute some variant of make install.

To get a better sense of the difference, here is a diff:

--- memcached-intermediate/debian/rules 2019-05-31 07:52:40.908868035 +0200
+++ memcached/debian/rules      2019-05-31 07:28:17.404380064 +0200
@@ -9,15 +9,14 @@
 %:
    dh $@

-override_dh_auto_clean:
-override_dh_auto_test:
-override_dh_auto_build:
-override_dh_auto_install:
+override_dh_update_autotools_config:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
-   ./configure --prefix=/usr
-   make
-   make install DESTDIR=debian/memcached
+
+override_dh_auto_test:
+   # Don't run the whitespace test
+   rm t/whitespace.t
+   dh_auto_test

 override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)

It is up to you to decide if dh can do some work for you, but you could try to start from a minimal debian/rules and only override some targets.

Install additional files#

While make install installed the essential files for memcached, you may want to put additional files in the binary package. You could use cp in your build recipe, but you can also declare them:

• files listed in debian/memcached.docs will be copied to /usr/share/doc/memcached by dh_installdocs
• files listed in debian/memcached.examples will be copied to /usr/share/doc/memcached/examples by dh_installexamples
• files listed in debian/memcached.manpages will be copied to the appropriate subdirectory of /usr/share/man by dh_installman

Here is an example using wildcards for debian/memcached.docs:

doc/*.txt

If you need to copy some files to an arbitrary location, you can list them along with their destination directories in debian/memcached.install and dh_install will take care of the copy. Here is an example:

scripts/memcached-tool usr/bin

Using these files makes the build process more declarative. It is a matter of taste and you are free to use cp in debian/rules instead. You can review the whole package tree on GitHub.

Other examples#

The Git repository contains some additional examples. They all follow the same scheme:

• dh_update_autotools_config is hijacked to download and setup the source tree
• dh_gencontrol is modified to use a computed version

Notably, you’ll find daemons in Java, Go, Python and Node.js. The goal of these examples is to demonstrate that using Debian tools to build Debian packages can be straightforward. Hope this helps.