Compressing embedded files in Go

Vincent Bernat

Go’s em­bed fea­ture lets you bun­dle static as­sets into an ex­e­cutable, but it stores them un­com­pressed. This wastes space: a web in­ter­face with doc­u­men­ta­tion can bloat your bi­nary by dozens of megabytes. A propo­si­tion to op­tion­ally en­able com­pres­sion was de­clined be­cause it is dif­fi­cult to han­dle all use cases. One so­lu­tion? Put all the as­sets into a ZIP archive! 🗜️

Code#

The Go stan­dard li­brary in­cludes a mod­ule to read and write ZIP ar­chives. It con­tains a func­tion that turns a ZIP ar­chive into an io/​fs.FS struc­ture that can re­place embed.FS in most contexts.1

package embed

import (
  "archive/zip"
  "bytes"
  _ "embed"
  "fmt"
  "io/fs"
  "sync"
)

//go:embed data/embed.zip
var embeddedZip []byte

var dataOnce = sync.OnceValue(func() *zip.Reader {
  r, err := zip.NewReader(bytes.NewReader(embeddedZip), int64(len(embeddedZip)))
  if err != nil {
    panic(fmt.Sprintf("cannot read embedded archive: %s", err))
  }
  return r
})

func Data() fs.FS {
  return dataOnce()
}

We can build the embed.​zip ar­chive with a rule in a Makefile. We spec­ify the files to em­bed as de­pen­den­cies to en­sure changes are detected.

common/embed/data/embed.zip: console/data/frontend console/data/docs
common/embed/data/embed.zip: orchestrator/clickhouse/data/protocols.csv 
common/embed/data/embed.zip: orchestrator/clickhouse/data/icmp.csv
common/embed/data/embed.zip: orchestrator/clickhouse/data/asns.csv
common/embed/data/embed.zip:
    mkdir -p common/embed/data && zip --quiet --recurse-paths --filesync $@ $^

The au­to­matic vari­able $@ is the rule tar­get, while $^ ex­pands to all the de­pen­den­cies, mod­i­fied or not.

Space gain#

Akvo­rado, a flow col­lec­tor writ­ten in Go, em­beds sev­eral static assets:

  • CSV files to trans­late port num­bers, pro­to­cols, or AS num­bers, and
  • HTML, CSS, JS, and im­age files for the web in­ter­face, and
  • the doc­u­mentation.
Breakdown of space used by each package before and after introducing
embed.zip. It is displayed as a treemap and we can see many embedded files
replaced by a bigger one.
Breakdown of the space used by each component before (left) and after (right) the introduction of embed.​zip.

Em­bed­ding these as­sets into a ZIP ar­chive re­duced the size of the Akvo­rado ex­e­cutable by more than 4 MiB:

$ unzip -p common/embed/data/embed.zip | wc -c | numfmt --to=iec
7.3M
$ ll common/embed/data/embed.zip
-rw-r--r-- 1 bernat users 2.9M Dec  7 17:17 common/embed/data/embed.zip

Performance loss#

Read­ing from a com­pressed ar­chive is not as fast as read­ing a flat file. A sim­ple bench­mark shows it is more than 4× slower. It also al­lo­cates some memory.2

goos: linux
goarch: amd64
pkg: akvorado/common/embed
cpu: AMD Ryzen 5 5600X 6-Core Processor
BenchmarkData/compressed-12     2262   526553 ns/op   610 B/op   10 allocs/op
BenchmarkData/uncompressed-12   9482   123175 ns/op     0 B/op    0 allocs/op

Each ac­cess to an as­set re­quires a de­com­pres­sion step, as seen in this flame graph:

🖼 Flame graph when reading data from embed.zip compared to reading data directly
CPU flame graph comparing the time spent on CPU when reading data from embed.zip (left) versus reading data directly (right). Because the Go testing framework executes the benchmark for uncompressed data 4 times more often, it uses the same horizontal space as the benchmark for compressed data. The graph is interactive.

While a ZIP ar­chive has an in­dex to quickly find the re­quested file, seek­ing in­side a com­pressed file is cur­rently not pos­si­ble.3 There­fore, the files from a com­pressed ar­chive do not im­ple­ment the io.​ReaderAt or io.​Seeker in­ter­faces, un­like di­rectly em­bed­ded files. This pre­vents some fea­tures, like serv­ing par­tial files or de­tect­ing MIME types when serv­ing files over HTTP.


For Akvo­rado, this is an ac­cept­able com­pro­mise to save a few mebibytes from an ex­e­cutable of al­most 100 MiB. Next week, I will con­tinue this fu­tile ad­ven­ture by ex­plain­ing how I pre­vented Go from dis­abling dead code elim­i­nation! 🦥