In 2008, I landed my second job, in the network team at Orange Portails, the division behind the websites and search engine of the French telecom operator Orange. The place ran like clockwork: a comprehensive technical setup, a dedicated team for every part of the business, and room to focus on what I do best. A few years later, none of that mattered: thanks to an obsession with the numbers, we could no longer deliver new services on time.

The first years

During my first years, the department operated like a startup. Its cradle was the French company Echo. They built a search engine. France Télécom bought it and renamed it Voila. It was the most visited search engine in France in the early 2000s. France Télécom consolidated the portal activities into the Wanadoo Portails division, later renamed Orange Portails.

The technical environment was excellent. We had many internal tools:² a ticket system, an RRD-based graphing tool, an IPAM, a reporting tool, and an SNMP-based alerting tool.³ We deployed our Linux servers with CFEngine. We installed systems and applications from internal Debian repositories. We documented everything in a private MediaWiki instance. Supervision was performed with an ancestor of Xymon. The network architecture was clean and scalable with little legacy. We onboarded new people in a day.

It was a nurturing environment for me. I developed several tools: lldpd, an 802.1AB implementation, Snimpy, a pythonic binding for Net-SNMP, Wiremaps, a layer-2 discovery tool with a time machine to know which device is connected where, Kitérő, a tool to simulate network conditions, QCSS-3, a controller for load-balancers, and ipoo, a service available through a Jabber chatbot and a Greasemonkey script to expose IP-related information. I added SNMP support for Keepalived and Quagga. I also started this blog, with articles like “Anycast DNS,” TLS-related articles like “TLS computational DoS mitigation,” SNMP-related articles like “Integration of Net-SNMP into an event loop,” Linux-related articles like “Tuning Linux IPv4 route cache,” and an article about VXLAN long before it was cool.

The collapse

When we needed new servers, the on-site team would take a set from the inventory, install our base Linux distribution on them, put them in the datacenter, and cable them to the top-of-the-rack switches. We opened a ticket describing the servers we needed, and one week later, our servers were available. 💫

Orange wanted to know if this team was performing well, so they asked for KPIs. They decided to use the number of tickets completed in a year. They asked to double this number. So instead of one ticket for a new service, we would open six tickets—one per server. By the end of the year, the KPIs had more than doubled.

Everybody saw it as a success for performance management. So, they asked to do the same for the next year. Now, we needed to open a ticket per server and per step. Again, the KPIs doubled. Behind the scenes, the tickets went to different people and were no longer handled in order. So, for the next year, it was decided to have meta-tickets and meetings to follow the progress of these tickets. Of course, all these extra steps pushed the KPI even higher.

This performance management method spread to the other teams.⁴ Everything became slower. Instead of a couple of weeks, a new service now took six months. We built a Soviet nail factory. But the KPIs were good, and we stopped caring.

Let me give you another example. We had to estimate the impact of each night operation. We weren’t half bad: we declared most operations “without any expected impact.” Most of the time, there was no impact. One time out of five, there was a 5-second impact. We were told to try harder to meet our expected impact. What did we do? We started declaring a 5-second expected impact. One day, we got a 30-second impact and were told we failed to match the expected impact. In the end, we declared most operations with a 10-minute expected impact, and we stopped caring: instead of carefully shifting traffic around, we allowed ourselves a 5-minute impact. And our KPIs were never better.

KPIs are not bad, but they are easy to break. Use them carefully: let the people doing the work help choose the metrics, and tie those metrics to the quality of the service—for example, with service level objectives. Otherwise, even dedicated people stop caring, game the system, and eventually quit. 📊

AI slop is invading the web. A recent story about disallowing LLM-generated submissions on Lobsters triggered a lot of debate. My personal worst offenders are LinkedIn articles with AI-generated images and uninspired articles filled with emojis from people trying to masquerade as experts on a subject they don’t care enough to write themselves. While I am unhappy about this situation, I rely on LLMs for grammar, copyediting, and translation. I don’t see this as a contradiction.

I am a native French speaker, but I blog in both English and French. When I started writing this blog in 2011, I was composing in French and translating to English, but I found it was better to work in the reverse order to avoid unnatural and non-idiomatic constructions. One of my goals is to write “good” English but I never felt it was my strong point.¹ For example, verb tenses are often an issue, even if I mostly stick with the present tense. I learn the rules and forget them right away. I also don’t feel like hiring an editor for something I see as a hobby.

As an example, I have kept the history of the successive iterations when writing “Scaling Akvorado BMP RIB with sharding”:

1. the first draft, authored with the help of a thesaurus,²
2. the edited copy revised by the copyediting skill,
3. the translation to French generated with the translation skill, and
4. the human proofread of the French translation, with minor edits to the English version.

I know that LLMs may alter the author’s voice when editing, but the corrections in the second step are minor. The prompt asks to “apply light stylistic edits,” with some guidance around avoiding passive voice, long sentences, bland verbs, and filler words. It also defines the target audience: technical with a B2 level in English.

In the following excerpt, I used “long time” instead of “long-standing.” The former is missing a hyphen and applies to people—a long-time friend, while the later relates to a situation—a long-standing agreement. I had a hard time understanding the reason of the second change: the LLM prefers a defining relative clause to provide the definition of “RIB sharding.”

As the Internet routing table contains more than 1 million routes, Akvorado needs to scale to tens of millions of routes. This has been a long time long-standing challenge, but I expect this issue is now fixed by using RIB sharding, a method to split that splits the routing database into several parts to enable concurrent updates.

In the next modification, the LLM puts “device” instead of “equipment.” This is correct as “equipment” is an uncountable noun. I know that, but I still fall into this trap.

When Akvorado does not find a route from a specific device, it falls back to a route sent by another equipment device.

I ask the LLM to use “descriptive verbs” and it complies by replacing a multi-word predicate with a lexically rich verb:

The benchmarks demonstrate it has better performance than outperforms other packages, both packages for lookups, insertions, and memory usage.

It also fixes grammar errors. In the next excerpt, a “list of routes” is a singular expression. Moreover, “stored” is a state and I should not use “into” as it expresses a change.

The list of routes for each prefix are is not stored directly into in the prefix tree.

As a last example, consider the following snippet. The “require” verb accepts a noun or an object followed by a to-infinitive. I can’t use it with just a to-infinitive.

An alternative would be to have one prefix tree for each peer but it would require to configure configuring all routers to export their routes.

As someone who didn’t grow up speaking English, I struggle with these grammar rules despite reading a lot of English material.³ French is more complex to get started but more systematic. English is full of irregularities.

On each page, I disclose in the footer whether an AI modified the content. There are three levels:

• 🧠: no AI or almost no AI (e.g., grammar corrections)
• ✨: enhanced (e.g., copyediting)
• 🤖: generated (e.g., translated from another language, even if human-edited)

Hover or tap the icon to reveal the AI’s name and its role in the document.

The graph below shows which tool altered each post, year by year. Recently, I applied the grammar skill to past articles. Since 2018, French articles have been translated with the help of DeepL first, then of an LLM. Since 2024, English articles are copyedited.

If you are strongly against any usage of LLMs specifically for writing, I hope you accept my more nuanced position on the usage of these tools as a trade-off to provide clearer and more engaging articles. Years of literature on improving English told us it is important to choose the right word to keep the reader engaged.

[…] Good writing consists of mastering the fundamentals (vocabulary, grammar, the elements of style) and then filling the third level of your toolbox with the right instruments.

― Stephen King, On Writing

To associate routing information—like AS paths or BGP communities—to flows, Akvorado can import routes through the BGP Monitoring Protocol (BMP). As the Internet routing table contains more than 1 million routes, Akvorado needs to scale to tens of millions of routes.¹ This has been a long-standing challenge,² but I expect this issue is now fixed by using RIB sharding, a method that splits the routing database into several parts to enable concurrent updates.

Previous implementation

Akvorado connects 2 elements to build its RIB:

1. a prefix tree, and
2. a list of routes attached to each prefix.

In the diagram above, the RIB stores five IPv4 prefixes and two IPv6 prefixes. One of them, 2001:db8:1::/48, contains three routes:

• from peer 3, next hop 2001:db8::3:1, AS 65402, AS path 65402, community 65402:31,
• from peer 4, next hop 2001:db8::4:1, same ASN, AS path, and community,
• from peer 5, next hop 2001:db8::5:1, AS 65402, AS path 65401 65402, community 65402:31.

The rib structure is defined in Go as follows:

type rib struct {
    tree          *bart.Table[prefixIndex]
    routes        map[routeKey]route
    nlris         *intern.Pool[nlri]
    nextHops      *intern.Pool[nextHop]
    rtas          *intern.Pool[routeAttributes]
    nextPrefixID  prefixIndex
    freePrefixIDs []prefixIndex
}

The prefix tree uses the bart package, an adaptation of Donald Knuth’s ART algorithm. The benchmarks demonstrate it outperforms other packages for lookups, insertions, and memory usage.³ Plus, the author is quite helpful.

Storing routes in a map

The list of routes for each prefix is not stored directly in the prefix tree: it would put too much pressure on the garbage collector by allocating per-prefix arrays.

Instead, the RIB assigns a unique 32-bit prefix identifier for each prefix, either by picking the last available prefix identifier from the freePrefixIDs array if any, or using the nextPrefixID value before incrementing it. Then, the routes are stored in the routes map, leveraging the optimized Swiss table in Go. To retrieve routes attached to a prefix, we look them up one by one in the routes map with a 64-bit key combining the 32-bit prefix index with a 32-bit route index matching the position of the route in the list. Akvorado scans routes from the first to the last to find the best one.⁴ It knows there is no more route if the route key returns no result.

type prefixIndex uint32
type routeIndex uint32
type routeKey uint64

Interning routes

A route contains a BGP peer identifier, a partial NLRI⁵, the next hop, and the attributes.

type route struct {
    peer       uint32
    nlri       intern.Reference[nlri]
    nextHop    intern.Reference[nextHop]
    attributes intern.Reference[routeAttributes]
    prefixLen  uint8
}

type nlri struct {
    family bgp.Family
    path   uint32
    rd     RD
}
type nextHop netip.Addr
type routeAttributes struct {
    asn              uint32
    asPath           []uint32
    communities      []uint32
    largeCommunities []bgp.LargeCommunity
}

To save memory and allocations, NLRI, next hops, and route attributes are “interned”: a 32-bit integer replaces the real value. The mechanism predates the unique package introduced in Go 1.23. We keep it because it has different trade-offs:

• It uses explicit reference counting instead of relying on weak pointers.
• It works with non-comparable values implementing Hash() and Equal() methods.⁶
• It uses explicit pool instances. This will be useful for sharding.
• It has better performance. See for example this benchmark.
• It consumes half the memory thanks to unsigned 32-bit references instead of pointers.
• But it is not safe for concurrent use.

Why does it not scale?

The global read/write lock is a bottleneck in this implementation. But how? There are several users of the RIB, each with its own set of constraints:

• The Kafka workers look up the RIB to enrich flows with routing information. They are bound by the number of Kafka partitions.⁸ Akvorado also adjusts their number to ensure efficient batching to ClickHouse. On our setup, the number of workers oscillates between 8 and 16. As we want to observe the latest data, we cannot afford for the Kafka workers to lag too much.

• The monitored routers send route updates through the BMP protocol. When connecting, they can send millions of routes.⁹ After the initial synchronization, updates are sent continuously and may spike from time to time. The router detects a stuck BMP station when its TCP window is full and resets the session in this case. While Akvorado implements a large incoming buffer, it still needs to update the received routes with the write lock held fast enough to avoid being detected as stuck.

• When a remote BGP peer goes down, Akvorado flushes the associated routes by walking the RIB with the write lock held. When a monitored router goes down, Akvorado waits a bit but eventually flushes all the associated routes.

In short: on a busy setup, lock contention is high for both readers and writers, and neither can lag too much behind.

RIB sharding

First step: basic sharding

To remove the global lock, the RIB is split into several “shards,” each one handling a subset of the prefixes:

The prefix tree stays global and is protected by a single lock. Each shard gets its read/write lock, its route map, and its intern pools to store NLRIs, next hops, and route attributes, which would not have been possible with Go’s unique package. The prefix indexes are also sharded: the 8 most significant bits are the shard index and the 24 remaining bits are the local prefix index.

Gerhard confirmed that after this blind change, the BMP receiver chugged steadily. 🎉

Later, I wrote a concurrent benchmark over half a million synthetic but plausible routes¹⁰ partitioned over 0 to 8 writers, churning routes as fast as possible, while 1 to 16 readers continuously look up a set of 10,000 routes. I don’t know if this benchmark is realistic, but it confirms the improvements for both read and write latencies:

It also shows that a high number of writers degrades read latency.

Second step: lock-free reads

The single read/write lock protecting the prefix tree is the next target. The bart package provides alternative mutation methods returning an updated tree using copy-on-write. Readers don’t need the global lock any more, leaving it only to synchronize writers. The prefix tree is boxed in an atomic pointer.

Without a lock, readers can now fetch a stale prefix index when walking their copy of the tree if a concurrent writer removes the last route attached to this prefix index and recycles it for another prefix. To avoid this issue, we combine the prefix index with a generation number and store them in the tree:

type generation uint32
type prefixRef struct {
    idx prefixIndex
    gen generation
}
type rib struct {
    mu     sync.Mutex
    tree   atomic.Pointer[bart.Table[prefixRef]]
    shards []*ribShard
}

Each shard stores the generation number for each local prefix index. The generation number increases by one if the associated prefix index is freed. When looking up the routes attached to a prefix index, the reader checks if the generation number matches. Otherwise, it assumes the index was recycled and the list of routes is empty.¹¹ You can see this case in the diagram above for prefix index 5, stored with a generation index of 3, while the current value in the []generations array is 4. The generation number could overflow, but it is not a problem as lookups are quick.

Running the concurrent benchmark against this new implementation shows the improvements for the read latency as soon as the cost of the copy-on-write prefix tree is amortized.

Among the multiple attempts to optimize the BMP component, RIB sharding is one of the more satisfying. Akvorado 2.2 implements the first step. PR #2433, drafted while writing this blog post, implements the second step and was released with Akvorado 2.4. 🪓

Vertical rhythm aligns lines to a consistent spacing cadence down the page. It creates a predictable flow for the eye to follow. Thanks to the rlh CSS unit, vertical rhythm is now easier to implement for text.¹ But illustrations and tables can disrupt the layout. The amateur typographer in me wants to follow Bringhurst’s wisdom:

Headings, subheads, block quotations, footnotes, illustrations, captions and other intrusions into the text create syncopations and variations against the base rhythm of regularly leaded lines. These variations can and should add life to the page, but the main text should also return after each variation precisely on beat and in phase.

― Robert Bringhurst, The Elements of Typographic Style

Text

Three factors govern vertical rhythm: font size, line height and margin or padding. Let’s set our baseline with an 18-pixel font and a 1.5 line height:

html {
  font-size: 112.5%;
  line-height: 1.5;
}
h1, h2, h3, h4 {
  font-size: 100%;
}
html, body,
h1, h2, h3, h4,
p, blockquote,
dl, dt, dd, ol, ul, li {
  margin: 0;
  padding: 0;
}

CSS Values and Units Module Level 4 defines the rlh unit, equal to the computed line height of the root element. All browsers support it since 2023.² Use it to insert vertical spaces or to fix the line height when altering font size:³

h1, h2, h3, h4 {
  margin-top: 2rlh;
  margin-bottom: 1rlh;
}
h1 {
  font-size: 2.5rem;
  line-height: 2rlh;
}
h2 {
  font-size: 1.5rem;
  line-height: 1rlh;
}
h3 {
  font-size: 1.25rem;
  line-height: 1rlh;
}
p, blockquote, pre {
  margin-top: 1rlh;
}
aside {
  font-size: 0.875rem;
  line-height: 1rlh;
}

We can check the result by overlaying a grid⁴ on the content:

If a child element uses a font with taller intrinsic metrics, it may stretch the line’s box beyond the configured line height.⁵ A workaround is to reduce the line height to 1. The glyphs overflow but don’t push the line taller.

code, kbd {
  line-height: 1;
}

Responsive images

Responsive images are difficult to align on the grid because we don’t know their height. CSS Rhythmic Sizing Module Level 1 introduces the block-step property to adjust the height of an element to a multiple of a step unit. But most browsers don’t support it yet.

With JavaScript, we can add padding around the image so it does not disturb the vertical rhythm:

const targets = document.querySelectorAll(".lf-media-outer");
const adjust = (el, height) => {
  const rlh = parseFloat(getComputedStyle(document.documentElement).lineHeight);
  const padding = Math.ceil(height / rlh) * rlh - height;
  el.style.padding = `${padding / 2}px 0`;
};

targets.forEach((el) => adjust(el, el.clientHeight));

As the image is responsive, its height can change. We need to wrap a resize observer around the adjust() function:

const ro = new ResizeObserver((entries) => {
  for (const entry of entries) {
    const height = entry.contentBoxSize[0].blockSize;
    adjust(entry.target, height);
  }
});
for (const target of targets) {
  ro.observe(target);
}

Tables

Table cells could set 1rlh as their height but they would feel constricted. Using 2rlh wastes too much space. Instead, we use incremental leading: we align one in every five lines.

table {
  border-spacing: 2px 0;
  border-collapse: separate;
  th {
    padding: 0.4rlh 1em;
  }
  td {
    padding: 0.2rlh 0.5em;
  }
}

To align the elements after the table, we need to add some padding. We can either reuse the JavaScript code from images or use a few lines of CSS that count the regular rows and compute the missing vertical padding:

table:has(tbody tr:nth-child(5n):last-child)   { padding-bottom: 0.2rlh; }
table:has(tbody tr:nth-child(5n+1):last-child) { padding-bottom: 0.8rlh; }
table:has(tbody tr:nth-child(5n+2):last-child) { padding-bottom: 0.4rlh; }
table:has(tbody tr:nth-child(5n+3):last-child) { padding-bottom: 0 }
table:has(tbody tr:nth-child(5n+4):last-child) { padding-bottom: 0.6rlh; }

A header cell has twice the padding of a regular cell. With two regular rows, the total padding is 2×2×0.2+2×0.4=1.6. We need to add 0.4rlh to reach 2rlh of extra vertical padding across the table.

None of this is necessary. But once you start looking, you can’t unsee it. Until browsers implement CSS Rhythmic Sizing, a bit of CSS wizardry and a touch of JavaScript is enough to pull it off. The main text now returns after each intrusion “precisely on beat and in phase.” 🎼

I often need a quick calculation or a unit conversion. Rather than reaching for a separate tool, a few lines of Zsh configuration turn = into a calculator. Typing = 660km / (2/3)c * 2 -> ms gives me 6.60457 ms¹ without leaving my terminal, thanks to the Zsh line editor.

The equal alias

The main idea looks simple: define = as an alias to a calculator command. I prefer Numbat, a scientific calculator that supports unit conversions. Qalculate is a close second.² If neither is available, we fall back to Zsh’s built-in zcalc module.

As the alias built-in uses = as a separator for name and value, we need to alter the aliases associative array:

if (( $+commands[numbat] )); then
  aliases[=]='numbat -e'
elif (( $+commands[qalc] )); then
  aliases[=]='qalc'
else
  autoload -Uz zcalc
  aliases[=]='zcalc -f -e'
fi

With this in place, = 847/11 becomes numbat -e 847/11.

The quoting problem

The first problem surfaces quickly. Typing = 5 * 3 fails: Zsh expands the * character as a glob pattern before passing it to the calculator. The same issue applies to other characters that Zsh treats specially, such as > or |. You must quote the expression:

$ = '5 * 3'
15

We fix this by hooking into the Zsh line editor to quote the expression before executing it.

Automatic quoting with ZLE

Zsh calls the line-finish widget before submitting a command. We hook a function that detects the = prefix and quotes the expression:

_vbe_calc_quote() {
  case $BUFFER in
    "="*)
      typeset -g _vbe_calc_expr=$BUFFER # not used yet
      BUFFER="= ${(q-)${${BUFFER#=}# }}"
      ;;
  esac
}
add-zle-hook-widget line-finish _vbe_calc_quote

When you type = 5 * 3 and press ↲, _vbe_calc_quote strips the = prefix, quotes the remainder with the (q-) parameter expansion flag, and rewrites the buffer to = '5 * 3' before Zsh submits the command. As a bonus, you can save a few keystrokes with =5*3! 🚀

You can now compute math expressions and convert units directly from your shell. Zsh automatically quotes your expressions:

$ = '1 + 2'
3
$ = 'pi/3 + pi |> cos'
-0.5
$ = '17 USD -> EUR'
14.7122 €
$ = '180*500mg -> g'
90 g
$ = '5 gigabytes / (2 minutes + 17 seconds) -> megabits/s'
291.971 Mbit/s
$ = 'now() -> tz("Asia/Tokyo")'
2026-03-22 22:00:03 JST (UTC +09), Asia/Tokyo
$ = '1 / (40 rods / hogshead) -> L / 100km'
118548 × 0.01 l/km

Storing unquoted history

As is, Zsh records the quoted expression in history. You must unquote it before submitting it again. Otherwise, the ZLE widget quotes it a second time. Bart Schaefer provided a solution to store the original version:

_vbe_calc_history() {
  return ${+_vbe_calc_expr}
}
add-zsh-hook zshaddhistory _vbe_calc_history

_vbe_calc_preexec() {
  (( ${+_vbe_calc_expr} )) && print -s $_vbe_calc_expr
  unset _vbe_calc_expr
  return 0
}
add-zsh-hook preexec _vbe_calc_preexec

The zshaddhistory hook returns 1 if we are evaluating an expression, telling Zsh not to record the command. The preexec hook then adds the original, unquoted command with print -s.

The complete code is available in my zshrc. A common alternative is the noglob precommand modifier. If you stick with to instead of -> for unit conversion, it covers 90% of use cases. For a related Zsh line editor trick, see how I use auto-expanding aliases to fix common typos.

Akvorado, a network flow collector, relies on Traefik, a reverse HTTP proxy, to expose HTTP endpoints for its Docker Compose services. Docker labels attached to each service define the routing rules. Traefik picks them up automatically when a container starts. Instead of maintaining a static configuration file to collect Prometheus metrics, we apply the same approach with Grafana Alloy.

Traefik & Docker

Traefik listens for events on the Docker socket. Each service advertises its configuration through labels. For example, here is the Loki service in Akvorado:

services:
  loki:
    # …
    expose:
      - 3100/tcp
    labels:
      - traefik.enable=true
      - traefik.http.routers.loki.rule=PathPrefix(`/loki`)

Once the container is healthy, Traefik creates a router forwarding requests matching /loki to its first exposed port. Colocating Traefik configuration with the service definition is attractive. How do we achieve the same for Prometheus metrics?

Metrics discovery with Alloy

Grafana Alloy, a metrics collector that scrapes Prometheus endpoints, includes a discovery.docker component. Just like Traefik, it connects to the Docker socket.¹ With a few relabeling rules, we teach it to use Docker labels to locate and scrape metrics.

We define three labels on each service:

• metrics.enable set to true enables metrics collection,
• metrics.port specifies the port exposing the Prometheus endpoint, and
• metrics.path specifies the path to the metrics endpoint.

If a service exposes more than one port, metrics.port is mandatory. Otherwise, it defaults to the only exposed port. The default value for metrics.path is /metrics. The Loki service from earlier becomes:

services:
  loki:
    # …
    expose:
      - 3100/tcp
    labels:
      - traefik.enable=true
      - traefik.http.routers.loki.rule=PathPrefix(`/loki`)
      - metrics.enable=true
      - metrics.path=/loki/metrics

Alloy’s configuration is split into four parts:

1. discover containers through the Docker socket,
2. filter and relabel targets using Docker labels,
3. scrape the matching endpoints, and
4. forward the metrics to Prometheus.

Discovering Docker containers

The first building block discovers running containers:

discovery.docker "docker" {
  host             = "unix:///var/run/docker.sock"
  refresh_interval = "30s"
  filter {
    name   = "label"
    values = ["com.docker.compose.project=akvorado"]
  }
}

This connects to the Docker socket and lists containers every 30 seconds.² The filter block restricts discovery to containers belonging to the akvorado project, avoiding interference with unrelated containers on the same host. For each discovered container, Alloy produces a target with labels such as __meta_docker_container_label_metrics_port for the metrics.port Docker label.

Relabeling targets

The relabeling step filters and transforms raw targets from Docker discovery into scrape targets. The first stage keeps only targets with metrics.enable set to true:

discovery.relabel "prometheus" {
  targets = discovery.docker.docker.targets

  // Keep only targets with metrics.enable=true
  rule {
    source_labels = ["__meta_docker_container_label_metrics_enable"]
    regex         = `true`
    action        = "keep"
  }

  // …
}

The second stage overrides the discovered port when the service defines metrics.port:

// When metrics.port is set, override __address__.
rule {
  source_labels = ["__address__", "__meta_docker_container_label_metrics_port"]
  regex         = `(.+):\d+;(.+)`
  target_label  = "__address__"
  replacement   = "$1:$2"
}

Next, we handle containers in host network mode. When __meta_docker_network_name equals host, Alloy rewrites the address to host.docker.internal instead of localhost:³

// When host networking, override __address__ to host.docker.internal.
rule {
  source_labels = ["__meta_docker_container_label_metrics_port", "__meta_docker_network_name"]
  regex         = `(.+);host`
  target_label  = "__address__"
  replacement   = "host.docker.internal:$1"
}

The next stage derives the job name from the service name, stripping any numbered suffix. The instance label is the address without the port:

rule {
  source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
  regex         = `(.+)(?:-\d+)?`
  target_label  = "job"
}
rule {
  source_labels = ["__address__"]
  regex         = `(.+):\d+`
  target_label  = "instance"
}

If a container defines metrics.path, Alloy uses it. Otherwise, it defaults to /metrics:

rule {
  source_labels = ["__meta_docker_container_label_metrics_path"]
  regex         = `(.+)`
  target_label  = "__metrics_path__"
}
rule {
  source_labels = ["__metrics_path__"]
  regex         = ""
  target_label  = "__metrics_path__"
  replacement   = "/metrics"
}

Scraping and forwarding

With the targets properly relabeled, scraping and forwarding are straightforward:

prometheus.scrape "docker" {
  targets         = discovery.relabel.prometheus.output
  forward_to      = [prometheus.remote_write.default.receiver]
  scrape_interval = "30s"
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://prometheus:9090/api/v1/write"
  }
}

prometheus.scrape periodically fetches metrics from the discovered targets. prometheus.remote_write sends them to Prometheus.

Built-in exporters

Some services do not expose a Prometheus endpoint. Redis and Kafka are common examples. Alloy ships built-in Prometheus exporters that query these services and expose metrics on their behalf.

prometheus.exporter.redis "docker" {
  redis_addr = "redis:6379"
}
discovery.relabel "redis" {
  targets = prometheus.exporter.redis.docker.targets
  rule {
    target_label = "job"
    replacement  = "redis"
  }
}
prometheus.scrape "redis" {
  targets         = discovery.relabel.redis.output
  forward_to      = [prometheus.remote_write.default.receiver]
  scrape_interval = "30s"
}

The same pattern applies to Kafka:

prometheus.exporter.kafka "docker" {
  kafka_uris = ["kafka:9092"]
}
discovery.relabel "kafka" {
  targets = prometheus.exporter.kafka.docker.targets
  rule {
    target_label = "job"
    replacement  = "kafka"
  }
}
prometheus.scrape "kafka" {
  targets         = discovery.relabel.kafka.output
  forward_to      = [prometheus.remote_write.default.receiver]
  scrape_interval = "30s"
}

Each exporter is a separate component with its own relabeling and scrape configuration. We set the job label explicitly since no Docker metadata can provide it.

With this setup, adding metrics to a new service with a Prometheus endpoint requires only a few labels in docker-compose.yml, just like adding a Traefik route. Alloy picks it up automatically. You can apply the same pattern with another discovery method, like discovery.kubernetes, discovery.scaleway, or discovery.http. 🩺