A Makefile for your Go project (2017)

Vincent Bernat February 5, 2017

Update (2019-07)

Go 1.11 introduces modules to manage dependencies without using GOPATH. While some aspects of the Makefile presented here are still relevant (fetching tools, running tests), the main point becomes moot. See the updated version of this guide.

My most loathed feature of Go is the mandatory use of GOPATH: I do not want to put my own code next to its dependencies. Hopefully, this issue is slowly starting to be accepted by the main authors. In the meantime, you can workaround this problem with more opinionated tools (like gb) or by crafting your own Makefile.

For the latter, you can have a look at Filippo Valsorda’s example or my own take which I describe in more detail here. This is not meant to be a universal Makefile but a relatively short one with some batteries included. It comes with a simple “Hello World!” application.

Project structure
Down the rabbit hole
Final result

Project structure#

For a standalone project, vendoring is a must-have¹ as you cannot rely on your dependencies to not introduce backward-incompatible changes. Some packages are using versioned URLs but most of them aren’t. There is currently no standard tool to handle vendoring. My personal take is to vendor all dependencies with Glide.²

It is a good practice to split an application into different packages while the main one stays fairly small. In the hellogopher example, the CLI is handled in the cmd package while the application logic for printing greetings is in the hello package:

.
├── cmd/
│   ├── hello.go
│   ├── root.go
│   └── version.go
├── glide.lock (generated)
├── glide.yaml
├── vendor/ (dependencies will go there)
├── hello/
│   ├── root.go
│   └── root_test.go
├── main.go
├── Makefile
└── README.md

Down the rabbit hole#

Let’s take a look at the various “features” of the Makefile.

`GOPATH` handling#

Since all dependencies are vendored, only our own project needs to be in the GOPATH:

PACKAGE  = hellogopher
GOPATH   = $(CURDIR)/.gopath
BASE     = $(GOPATH)/src/$(PACKAGE)

$(BASE):
    @mkdir -p $(dir $@)
    @ln -sf $(CURDIR) $@

The base import path is hellogopher, not github.com/vincentbernat/hellogopher: this shortens imports and makes them easily distinguishable from imports of dependency packages. However, your application won’t be go get-able. This is a personal choice and can be adjusted with the $(PACKAGE) variable.

We just create a symlink from .gopath/src/hellogopher to our root directory. The GOPATH environment variable is automatically exported to the shell commands of the recipes. Any tool should work fine after changing the current directory to $(BASE). For example, this snippet builds the executable:

.PHONY: all
all: | $(BASE)
    cd $(BASE) && $(GO) build -o bin/$(PACKAGE) main.go

Vendoring dependencies#

Glide is a bit like Ruby’s Bundler. In glide.yaml, you specify what packages you need and the constraints you want on them. Glide computes a glide.lock file containing the exact versions for each dependency (including recursive dependencies) and download them in the vendor/ folder. I choose to check into the VCS both glide.yaml and glide.lock files. It’s also possible to only check in the first one or to also check in the vendor/ directory. A work-in-progress is currently ongoing to provide a standard dependency management tool with a similar workflow.

We define two rules:³

GLIDE = glide

glide.lock: glide.yaml | $(BASE)
    cd $(BASE) && $(GLIDE) update
    @touch $@
vendor: glide.lock | $(BASE)
    cd $(BASE) && $(GLIDE) --quiet install
    @ln -sf . vendor/src
    @touch $@

We use a variable to invoke glide. This enables a user to easily override it (for example, with make GLIDE=$GOPATH/bin/glide).

Using third-party tools#

Most projects need some third-party tools. We can either expect them to be already installed or compile them in our private GOPATH. For example, here is the lint rule:

BIN    = $(GOPATH)/bin
GOLINT = $(BIN)/golint

$(BIN)/golint: | $(BASE) # ❶
    go get github.com/golang/lint/golint

.PHONY: lint
lint: vendor | $(BASE) $(GOLINT) # ❷
    @cd $(BASE) && ret=0 && for pkg in $(PKGS); do \
        test -z "$$($(GOLINT) $$pkg | tee /dev/stderr)" || ret=1 ; \
     done ; exit $$ret

As for glide, we let the user a chance to override which golint executable to use. By default, it uses a private copy. But a user can use its own copy with make GOLINT=/usr/bin/golint.

In ❶, we have the recipe to build the private copy. We issue go get⁴ to download and build golint. In ❷, the lint rule executes golint on each package contained in the $(PKGS) variable. We’ll explain this variable in the next section.

Working with non-vendored packages only#

Some commands need to be provided with a list of packages. Because we use a vendor/ directory, the shortcut ./... is not what we expect as we don’t want to run tests on our dependencies.⁵ Therefore, we compose a list of packages we care about:

PKGS = $(or $(PKG), $(shell cd $(BASE) && \
    env GOPATH=$(GOPATH) $(GO) list ./... | grep -v "^$(PACKAGE)/vendor/"))

If the user has provided the $(PKG) variable, we use it. For example, if they want to lint only the cmd package, they can invoke make lint PKG=hellogopher/cmd which is more intuitive than specifying PKGS.

Otherwise, we just execute go list ./... but we remove anything from the vendor/ directory.

Update (2018-03)

Since Go 1.9, ./... no longer matches packages in the vendor/ directory.

Tests#

Here are some rules to run tests:

TIMEOUT = 20
TEST_TARGETS := test-default test-bench test-short test-verbose test-race
.PHONY: $(TEST_TARGETS) check test tests
test-bench:   ARGS=-run=__absolutelynothing__ -bench=.
test-short:   ARGS=-short
test-verbose: ARGS=-v
test-race:    ARGS=-race
$(TEST_TARGETS): test

check test tests: fmt lint vendor | $(BASE)
    @cd $(BASE) && $(GO) test -timeout $(TIMEOUT)s $(ARGS) $(PKGS)

A user can invoke tests in different ways:

make test runs all tests;
make test TIMEOUT=10 runs all tests with a timeout of 10 seconds;
make test PKG=hellogopher/cmd only runs tests for the cmd package;
make test ARGS="-v -short" runs tests with the specified arguments;
make test-race runs tests with race detector enabled.

Tests coverage#

go test includes a test coverage tool. Unfortunately, it only handles one package at a time and you have to explicitly list the packages to be instrumented, otherwise the instrumentation is limited to the currently tested package. If you provide too many packages, the compilation time will skyrocket. Moreover, if you want an output compatible with Jenkins, you’ll need some additional tools.

COVERAGE_MODE    = atomic
COVERAGE_PROFILE = $(COVERAGE_DIR)/profile.out
COVERAGE_XML     = $(COVERAGE_DIR)/coverage.xml
COVERAGE_HTML    = $(COVERAGE_DIR)/index.html

.PHONY: test-coverage test-coverage-tools
test-coverage-tools: | $(GOCOVMERGE) $(GOCOV) $(GOCOVXML) # ❸
test-coverage: COVERAGE_DIR := $(CURDIR)/test/coverage.$(shell date -Iseconds)
test-coverage: fmt lint vendor test-coverage-tools | $(BASE)
    @mkdir -p $(COVERAGE_DIR)/coverage
    @cd $(BASE) && for pkg in $(PKGS); do \ # ❹
        $(GO) test \
            -coverpkg=$$($(GO) list -f '{{ join .Deps "\n" }}' $$pkg | \
                    grep '^$(PACKAGE)/' | grep -v '^$(PACKAGE)/vendor/' | \
                    tr '\n' ',')$$pkg \
            -covermode=$(COVERAGE_MODE) \
            -coverprofile="$(COVERAGE_DIR)/coverage/`echo $$pkg | tr "/" "-"`.cover" $$pkg ;\
     done
    @$(GOCOVMERGE) $(COVERAGE_DIR)/coverage/*.cover > $(COVERAGE_PROFILE)
    @$(GO) tool cover -html=$(COVERAGE_PROFILE) -o $(COVERAGE_HTML)
    @$(GOCOV) convert $(COVERAGE_PROFILE) | $(GOCOVXML) > $(COVERAGE_XML)

First, we define some variables to let the user override them. We also require the following tools (in ❸):

gocovmerge merges profiles from different runs into a single one;
gocov-xml converts a coverage profile to the Cobertura format;
gocov is needed to convert a coverage profile to a format handled by gocov-xml.

The rules to build these tools are similar to the rule for golint described a few sections ago.

In ❹, for each package to test, we run go test with the -coverprofile argument. We also explicitly provide the list of packages to instrument to -coverpkg by using go list to get a list of dependencies for the tested package and keeping only our own.

Final result#

While the main goal of using a Makefile was to work around GOPATH, it is also a good place to hide the complexity of some operations, notably around test coverage.

The excerpts provided in this post are a bit simplified. Have a look at the final result for more perks!