Testing network software with pytest and Linux namespaces
Started in 2008, lldpd is an implementation of IEEE 802.1AB-2005 (aka LLDP) written in C. While it contains some unit tests, like many other network-related software at the time, their coverage is pretty poor: they are hard to write because the code is written in an imperative style and tighly coupled with the system. It would require extensive mocking.1 While a rewrite (complete or iterative) would help to make the code more test-friendly, it would be quite an effort and it will likely introduce operational bugs along the way.
To get better test coverage, the major features of lldpd are now verified through integration tests. These tests leverage Linux network namespaces to setup a lightweight and isolated environment for each test. They run through pytest, a powerful testing tool.
pytest in a nutshell#
pytest is a Python testing tool whose primary use is to write tests for Python applications but is versatile enough for other creative usages. It is bundled with three killer features:
- you can directly use the
- you can inject fixtures in any test function; and
- you can parametrize tests.
With unittest, the unit testing framework included with Python, and many similar frameworks, unit tests have to be encapsulated into a class and use the provided assertion methods. For example:
class testArithmetics(unittest.TestCase): def test_addition(self): self.assertEqual(1 + 3, 4)
The equivalent with pytest is simpler and more readable:
def test_addition(): assert 1 + 3 == 4
pytest will analyze the AST and display useful error messages in case of failure. For further information, see Benjamin Peterson’s article.
A fixture is the set of actions performed in order to prepare the system to run some tests. With classic frameworks, you can only define one fixture for a set of tests:
class testInVM(unittest.TestCase): def setUp(self): self.vm = VM('Test-VM') self.vm.start() self.ssh = SSHClient() self.ssh.connect(self.vm.public_ip) def tearDown(self): self.ssh.close() self.vm.destroy() def test_hello(self): stdin, stdout, stderr = self.ssh.exec_command("echo hello") stdin.close() self.assertEqual(stderr.read(), b"") self.assertEqual(stdout.read(), b"hello\n")
In the example above, we want to test various commands on a remote
VM. The fixture launches a new VM and configure an SSH
connection. However, if the SSH connection cannot be established, the
fixture will fail and the
tearDown() method won’t be invoked. The VM
will be left running.
Instead, with pytest, we could do this:
@pytest.yield_fixture def vm(): r = VM('Test-VM') r.start() yield r r.destroy() @pytest.yield_fixture def ssh(vm): ssh = SSHClient() ssh.connect(vm.public_ip) yield ssh ssh.close() def test_hello(ssh): stdin, stdout, stderr = ssh.exec_command("echo hello") stdin.close() stderr.read() == b"" stdout.read() == b"hello\n"
The first fixture will provide a freshly booted VM. The second one will setup an SSH connection to the VM provided as an argument. Fixtures are used through dependency injection: just give their names in the signature of the test functions and fixtures that need them. Each fixture only handle the lifetime of one entity. Whatever a dependent test function or fixture succeeds or fails, the VM will always be finally destroyed.
If you want to run the same test several times with a varying parameter, you can dynamically create test functions or use one test function with a loop. With pytest, you can parametrize test functions and fixtures:
@pytest.mark.parametrize("n1, n2, expected", [ (1, 3, 4), (8, 20, 28), (-4, 0, -4)]) def test_addition(n1, n2, expected): assert n1 + n2 == expected
The general plan for to test a feature in lldpd is the following:
- Setup two namespaces.
- Create a virtual link between them.
- Spawn a
lldpdprocess in each namespace.
- Test the feature in one namespace.
- Check with
lldpcliwe get the expected result in the other.
Here is a typical test using the most interesting features of pytest:
@pytest.mark.skipif('LLDP-MED' not in pytest.config.lldpd.features, reason="LLDP-MED not supported") @pytest.mark.parametrize("classe, expected", [ (1, "Generic Endpoint (Class I)"), (2, "Media Endpoint (Class II)"), (3, "Communication Device Endpoint (Class III)"), (4, "Network Connectivity Device")]) def test_med_devicetype(lldpd, lldpcli, namespaces, links, classe, expected): links(namespaces(1), namespaces(2)) with namespaces(1): lldpd("-r") with namespaces(2): lldpd("-M", str(classe)) with namespaces(1): out = lldpcli("-f", "keyvalue", "show", "neighbors", "details") assert out['lldp.eth0.lldp-med.device-type'] == expected
First, the test will be executed only if
lldpd was compiled with
LLDP-MED support. Second, the test is parametrized. We will execute
four distinct tests, one for each role that
lldpd should be able to
take as an LLDP-MED-enabled endpoint.
The signature of the test has four parameters that are not covered by
links. They are fixtures. A lot of magic happen in them to keep
the actual tests short:
lldpdis a factory to spawn an instance of
lldpd. When called, it will setup the current namespace (setting up the chroot, creating the user and group for privilege separation, replacing some files to be distribution-agnostic, …), then call
lldpdwith the additional parameters provided. The output is recorded and added to the test report in case of failure. The module also contains the creation of the
pytest.config.lldpdobject that is used to record the features supported by
lldpdand skip non-matching tests. You can read
fixtures/programs.pyfor more details.
lldpcliis also a factory, but it spawns instances of
lldpcli, the client to query
lldpd. Moreover, it will parse the output in a dictionary to reduce boilerplate.
namespacesis one of the most interesting pieces. It is a factory for Linux namespaces. It will spawn a new namespace or refer to an existing one. It is possible to switch from one namespace to another (with
with) as they are contexts. Behind the scene, the factory maintains the appropriate file descriptors for each namespace and switch to them with
setns(). Once the test is done, everything is wipped out as the file descriptors are garbage collected. You can read
fixtures/namespaces.pyfor more details. It is quite reusable in other projects.2
linkscontains helpers to handle network interfaces: creation of virtual ethernet link between namespaces, creation of bridges, bonds and VLAN, etc. It relies on the pyroute2 module. You can read
fixtures/network.pyfor more details.
You can see an example of a test run on the
Travis build for 0.9.2. Since each test is correctly isolated,
it’s possible to run parallel tests with
pytest -n 10 --boxed. To
catch even more bugs, both the address sanitizer (ASAN) and the
undefined behavior sanitizer (UBSAN) are enabled. In case of a
problem, notably a memory leak, the faulty program will exit with a
non-zero exit code and the associated test will fail.
A project like cwrap would definitely help. However, it lacks support for Netlink and raw sockets that are essential in lldpd operations. ↩︎
There are three main limitations in the use of namespaces with this fixture. First, when creating a user namespace, only root is mapped to the current user. With lldpd, we have two users (
_lldpd). Therefore, the tests have to run as root. The second limitation is with the PID namespace. It’s not possible for a process to switch from one PID namespace to another. When you call
setns()on a PID namespace, only children of the current process will be in the new PID namespace. The PID namespace is convenient to ensure everyone gets killed once the tests are terminated but you must keep in mind that
/procmust be mounted in children only. The third limitation is that, for some namespaces (PID and user), all threads of a process must be part of the same namespace. Therefore, don’t use threads in tests. Use
multiprocessingmodule instead. ↩︎