With this post, I would like to start a series around contribution to open source. Each post will be an example of contribution to a project. There is no universal recipe but the steps are usually:

1. Spot a problem.
2. Check it is still present in recent versions.
3. Check it has not been already reported.
4. Fix the problem.
5. Send the fix upstream.

Let’s explore how to add a feature to RRDtool to ignore missing RRD files.

What is RRDtool?#

RRDtool stands for round-robin database tool. It stores time-series data like CPU usage or network bandwidth in a circular buffer to ensure that the size of each RRD file remains constant over time. Once stored, data can be graphed to get something like this:

The rrdtutorial manual page provides a complete introduction to RRDtool. To understand the remainder of this post, you need to have a basic knowledge of how RRD files work. Let’s do a very quick introduction:

• an RRD file defines a step interval, for example 30 seconds;
• an RRD file also defines one or several data sources (DS);
• a DS is composed of one primary data point (PDP) every step interval; those PDP are not stored in the RRD files: an external process will feed data points every step interval and RRDtool will interpolate them into PDP so that they exactly fit the step interval;
• PDP are transformed into consolidated data points (CDP) using a consolidation function (CF), like the average function;
• CDP are stored in a round-robin archive (RRA) inside the RRD file;
• an RRA is defined by a CF, the number of PDP to consolidate to get a CDP and the number of CDP to keep; an RRD file can have several RRA.

To create a new RRD file, you can issue the following command:

rrdtool create sensor1.rrd         \
    --step 30                      \
    DS:temperature:GAUGE:100:0:U   \
    RRA:AVERAGE:0.5:1:100          \
    RRA:AVERAGE:0.5:10:100

The step interval is 30. There is one data source named temperature. We have two RRA. The first one will take one PDP and turn it into a CDP and will store 100 of them. It will contain less than one hour of data (30×1×100). The second one will take 10 PDP, compute the average to get a CDP and store 100 of them. It covers a greater time span with less precision.

To feed the data source, you can use some script:

while true; do
   temp=$(get_sensor1_temperature)
   [ -z "$temp" ] || \
      rrdtool update sensor1.rrd N:$temp
   sleep 30
done

Suppose you have built three RRD files for three different sensors and you want to graph them:

rrdtool graph temperatures.png             \
  --start="now-1h" --end="now"             \
  --title="Temperatures for X, Y and Z"    \
  --vertical-label="K"                     \
  --lower-limit=0                          \
  DEF:t1=sensor1.rrd:temperature:AVERAGE   \
  DEF:t2=sensor2.rrd:temperature:AVERAGE   \
  DEF:t3=sensor3.rrd:temperature:AVERAGE   \
  LINE1:t1#ff0000:"Sensor 1"               \
  LINE1:t2#00ff00:"Sensor 2"               \
  LINE1:t3#0000ff:"Sensor 3"

The DEF lines allow you to name data sources to be used. RRDtool will select the most appropriate RRA to match the requested interval with the maximum precision.

The problem#

While you can store several different time-series in an RRD file, it is common practice to use several RRD files to build a graph, like in the example above.

If you have many graphs to generate for different systems, you can use a tool which comes with a templating system, like drraw. However, if it is not smart enough to only include definitions for RRD files that actually exist¹, rrdtool graph will complain about the missing RRD files and won’t output anything.

You could provide an empty RRD file, but this takes disk space. It would be convenient to let RRDtool consider data points from a missing RRD file as unknown values. A quick search about a solution wasn’t fruitful.

Reproduce the problem#

Before trying to fix the problem, we need to reproduce it with simpler data. It is likely that you run into the problem with a complex graph combining several RRD. We need to find a small test case that exhibits the problem and that we will be able to use until the resolution of the problem. This test case also allows you to work outside of your production environment.

So, let’s build three simple RRD files:

for i in $(seq 1 3); do
  rrdtool create sensor${i}.rrd        \
        --start=now-5h --step 30       \
        DS:temperature:GAUGE:100:U:U   \
        RRA:AVERAGE:0.5:1:300          \
        RRA:AVERAGE:0.5:10:100
  for t in $(seq 1 600); do
    t=$(( (601-$t) * -30))
    v=$(echo "c($t/5+$i)*$i*4 + 10" | bc -l)
    rrdtool update sensor${i}.rrd -- ${t}:${v}
  done
done

Let’s build the appropriate graph:

rrdtool graph out.png                          \
      --start="now-2h" --end="now-1h"          \
      --title="Temperatures for X, Y and Z"    \
      --vertical-label="°C"                    \
      --lower-limit=-10                        \
      DEF:t1=sensor1.rrd:temperature:AVERAGE   \
      DEF:t2=sensor2.rrd:temperature:AVERAGE   \
      DEF:t3=sensor3.rrd:temperature:AVERAGE   \
      LINE1:t1#ff0000:"Sensor 1"               \
      LINE1:t2#00ff00:"Sensor 2"               \
      LINE1:t3#0000ff:"Sensor 3"

We get the following graph:

Reproducing the problem is then quite easy. We just suppress one of the RRD files and the previous command now says: ERROR: opening 'sensor2.rrd': No such file or directory.

Reproduce the problem with the latest version#

The previous experiment was done with version 1.4.7. This is the latest stable version. However, maybe the problem has been already fixed in the development version. Moreover, developing a fix on the development branch is more effective than on a random stable version.

Development of RRDtool is now hosted on GitHub. Let’s grab the source tree and compile it:

$ git clone git@github.com:oetiker/rrdtool-1.x.git
Cloning into 'rrdtool-1.x'...
$ cd rrdtool-1.x
$ ./autogen.sh
[...]
$ ./configure CFLAGS="-O0 -g"
checking build system type... x86_64-unknown-linux-gnu
[...]
Config is DONE!
$ make
[...]

Once compiled, we can try again our test case. We just need to replace rrdtool with libtool execute src/rrdtool. We get the exact same error. A quick look at the updated manual page shows no new option that could help here.

The fix#

The first step is to find where the error happens:

$ git grep "ERROR: opening"
$ git grep "opening '"
src/rrd_cgi.c:            rrd_set_error("opening '%s': %s", file_name, rrd_strerror(errno));
src/rrd_open.c:        rrd_set_error("opening '%s': %s", file_name, rrd_strerror(errno));

The corresponding source code looks like this:

rrd_file_t *rrd_open(
    const char *const file_name,
    rrd_t *rrd,
    unsigned rdwr)
{
/* [...] */
    if ((rrd_simple_file->fd = open(file_name, flags, 0666)) < 0) {
        rrd_set_error("opening '%s': %s", file_name, rrd_strerror(errno));
        goto out_free;
    }
/* [...] */
}

If the file doesn’t exist, we could open a fake file. However, there may be a better place. We need to find where rrd_open() is called. Instead of using git grep, let’s use gdb and set a breakpoint when the error happens:

$ libtool execute gdb --args   \
rrdtool-1.x/src/rrdtool        \
      graph out.png            \
      --start="now-2h" --end="now-1h"        \
      --title="Temperatures for X, Y and Z"  \
      --vertical-label="°C"                  \
      --lower-limit=-10                      \
      DEF:t1=sensor1.rrd:temperature:AVERAGE \
      DEF:t2=sensor2.rrd:temperature:AVERAGE \
      DEF:t3=sensor3.rrd:temperature:AVERAGE \
      LINE1:t1#ff0000:"Sensor 1"             \
      LINE1:t2#00ff00:"Sensor 2"             \
      LINE1:t3#0000ff:"Sensor 3"

gdb$ b rrd_open.c:196
No source file named rrd_open.c.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (rrd_open.c:196) pending.

gdb$ run
Starting program: /home/bernat/tmp/rrdtool-1.x/src/.libs/lt-rrdtool graph out.png --start=now-2h --end=now-1h --title=Temperatures\ for\ X,\ Y\ and\ Z --vertical-label=°C --lower-limit=-10 DEF:t1=sensor1.rrd:temperature:AVERAGE DEF:t2=sensor2.rrd:temperature:AVERAGE DEF:t3=sensor3.rrd:temperature:AVERAGE LINE1:t1\#ff0000:Sensor\ 1 LINE1:t2\#00ff00:Sensor\ 2 LINE1:t3\#0000ff:Sensor\ 3

Breakpoint 1, rrd_open (file_name=0x425e30 "sensor2.rrd", rrd=0x7fffffffaea0, rdwr=1) at rrd_open.c:196
196             rrd_set_error("opening '%s': %s", file_name, rrd_strerror(errno));

gdb$ bt
#0  rrd_open (file_name=0x425e30 "sensor2.rrd", rrd=0x7fffffffaea0, rdwr=1) at rrd_open.c:196
#1  0x00007ffff7faa274 in rrd_fetch_fn (filename=0x425e30 "sensor2.rrd", cf_idx=CF_AVERAGE, start=0x4265b0, end=0x4265b8, step=0x7fffffffaff0, ds_cnt=0x4265e0, ds_namv=0x4265f0, data=0x4265f8) at rrd_fetch.c:249
#2  0x00007ffff7fafd4b in data_fetch (im=0x7fffffffb0c0) at rrd_graph.c:905
#3  0x00007ffff7fb9bc7 in graph_paint (im=0x7fffffffb0c0) at rrd_graph.c:3296
#4  0x00007ffff7fbdf67 in rrd_graph_v (argc=13, argv=0x7fffffffe5c0) at rrd_graph.c:4132
#5  0x00007ffff7fbd9b3 in rrd_graph (argc=13, argv=0x7fffffffe5c0, prdata=0x7fffffffe288, xsize=0x7fffffffe284, ysize=0x7fffffffe280, stream=0x0, ymin=0x7fffffffe278, ymax=0x7fffffffe270) at rrd_graph.c:4013
#6  0x0000000000403279 in HandleInputLine (argc=14, argv=0x7fffffffe5b8, out=0x7ffff7dd7880) at rrd_tool.c:740
#7  0x000000000040259a in main (argc=14, argv=0x7fffffffe5b8) at rrd_tool.c:523

So, we could modify rrd_fetch_fn() in rrd_fetch.c or data_fetch() in rrd_graph.c. Since we only want to modify the behaviour of rrdtool graph, let’s have a look at rrd_graph.c:

/* If connecting was successfull, use the daemon to query the data.
 * If there is no connection, for example because no daemon address
 * was specified, (try to) use the local file directly. */
if (rrdc_is_connected (rrd_daemon))
{
    status = rrdc_fetch (im->gdes[i].rrd,
            cf_to_string (im->gdes[i].cf),
            &im->gdes[i].start,
            &im->gdes[i].end,
            &ft_step,
            &im->gdes[i].ds_cnt,
            &im->gdes[i].ds_namv,
            &im->gdes[i].data);
    if (status != 0)
        return (status);
}
else
{
    if ((rrd_fetch_fn(im->gdes[i].rrd,
                    im->gdes[i].cf,
                    &im->gdes[i].start,
                    &im->gdes[i].end,
                    &ft_step,
                    &im->gdes[i].ds_cnt,
                    &im->gdes[i].ds_namv,
                    &im->gdes[i].data)) == -1) {
        return -1;                      
    }               
}

There are two cases that should be handled:

1. when the data is fetched from the RRD daemon and
2. when the data is fetched directly from an RRD file.

Let’s focus on the second case, but for the sake of coherency, the first case should be handled too. Before doing any modification, let’s switch to a dedicated topic branch:

$ git checkout -b feature/missing-rrd

rrd_fetch_fn() is expected to modify the start, end, ft_step to match what’s possible with the requested RRD file. It is also expected to give proper values to ds_cnt (number of data sources), ds_namv (names of data sources) and data (the CDP from the most appropriate RRA). We can try a simple fix: provide an empty RRD with no data source:

if ((rrd_fetch_fn(...)) == -1) {
    rrd_clear_error();
    im->gdes[i].ds_cnt = im->gdes[i].start = im->gdes[i].end = 0;
    ft_step = 1; 
}

If we try this modification, we get an error at the end of the data_fetch() function: ERROR: No DS called 'temperature' in 'sensor2.rrd. This is due to the following check:

/* lets see if the required data source is really there */
for (ii = 0; ii < (int) im->gdes[i].ds_cnt; ii++) {
    if (strcmp(im->gdes[i].ds_namv[ii], im->gdes[i].ds_nam) == 0) {
        im->gdes[i].ds = ii;
    }
}
if (im->gdes[i].ds == -1) {
    rrd_set_error("No DS called '%s' in '%s'",
                  im->gdes[i].ds_nam, im->gdes[i].rrd);
    return -1;
}

Removing the last check seems to do the trick. There are two problems:

1. Can the remaining code handle a graph description with ds set to -1 and ds_cnt set to 0?
2. We have removed perfectly valid errors about missing files and missing data sources.

For the first problem, the test case seems to run fine. In my opinion, you should not spend too much time checking that your patch doesn’t break anything: submit it to upstream, tell them that you have some doubts and, because they have far more knowledge of the codebase than you, they should be able to tell.

Let’s tackle the second issue. We will add an option to rrdtool graph allowing one to provide missing RRD files. We need to find a free letter (like -Z) and a long name (like --missing-ds). The image_desc_t structure passed to data_fetch() function has an extra_flags member. We define a new flag, ALLOW_MISSING_DS, that will be used when the user specifies our new option.

Adding this option is not really difficult. Have a look at the final patch².

Submitting the fix#

The last step is to forward the patch to the upstream project. Even if your patch is incomplete or buggy, it is still worth submitting it. There is no reason to be shy or ashamed: somebody may take the patch from here and enhance it or suggest improvements.

There are usually two possibilities:

1. Submit the patch to the mailing-list of the project: use git send-email master for this purpose. This can be cumbersome because you may have to subscribe to the mailing-list before being able to post.
2. Submit the patch through an issue tracker.

Directly submitting the patch to a private person is less likely to be useful since it won’t be available to anyone else until the person acts upon it.

RRDtool is hosted on GitHub which comes with a simple issue tracker. If you are not familiar with GitHub, there is a convenient way to open a new issue and attaching a branch directly to it. Follow those simple steps:

1. Fork the repository. The repository will be show in your account with a link to the original.
2. Commit the change into a branch of your copy of the repository.
3. Push the changes to GitHub.
4. Send a pull request.

If you have already cloned the upstream repository and do not want to start again after forking it on GitHub, you can add your fork as a remote:

# Add your fork as a remote repository
git remote add github git@github.com:vincentbernat/rrdtool-1.x
# Rename the origin remote to something more sensible
git remote rename origin upstream
# Sync with upstream (optional)
git pull upstream master
git rebase master
# Push the topic branch to GitHub
git push feature/missing-rrd github