Staves – A Container Image Builder based on Gentoo Linux

Michael Seifert, 2019-02-26, Updated 2020-11-17

The problem

Ameto (opens new window), the project I am working on, is mostly written in Python. Python has a great ecosystem of libraries and tools that make it a pleasure to use, but the language often prioritizes readability and simplicity over raw performance. Therefore, many packages come with native libraries and make function calls via ctypes (opens new window), Cython (opens new window), or cffi (opens new window). This may solve the performance situation, but it creates problems regarding the portability and packaging of the Python package: Whenever a user wants to install such a Python package, the native library has to be compiled for the user's architecture. Hence, they need a working compiler toolchain. Sometimes the native library link against additional libraries that need to be installed on the system, as is the case with Pillow (opens new window), for example. Runtime-dependencies like these cannot be expressed using a Python package manager like pip. The packaging situation in Python has improved a lot over the years, but this issue persists.

Existing tools

Linux Distribution images

Many existing Docker images are based off a Linux Distribution. Installing a package and its dependencies is as easy as using the distribution's package manager. Ubuntu or Debian base-image come in rather big, which is why most packagers also offer a variant based on Alpine Linux. I, too, started out assembling my Docker images based on Alpine Linux but I found the packages to be somewhat dated. I then tried to install more recent versions, but this is where things got really complicated. Let's have a look at the steps required to install Pillow (taken from this SO answer (opens new window)):

FROM python:alpine
RUN apk --update add libxml2-dev libxslt-dev libffi-dev gcc musl-dev libgcc openssl-dev curl
RUN apk add jpeg-dev zlib-dev freetype-dev lcms2-dev openjpeg-dev tiff-dev tk-dev tcl-dev
RUN pip install Pillow

The Dockerfile uses the official Python image based on Alpine Linux. It uses Alpine's package manager apk to set up the compilation toolchain and install runtime dependencies of Pillow. This can be refactored to a multi-stage build in order to exclude the toolchain and save space in the resulting image, but this is not the point here. What bothered me was the need to take care of all the dependencies. Since Pillow was not the only dependency that required native libraries, things were about to get more and more complicated. And I moved on.

Dgr

During my search I came across dgr (opens new window). Dgr is developed by BlaBlaCar who claim to use it in production. The dgr code base is maintained and provides automated tests. However, it only supports images in the ACI format which can be run by rkt (opens new window).[2] They planned to add support for OCI images, which could be used by Docker, but it has never happened.

Kubler

I also tried Kubler (opens new window), an image builder based on Gentoo Linux. As a long-time Gentoo user I like the idea, because it makes Kubler very flexible. However, the project did not live up to my expectations due to a number of reasons: Kubler's use case seems to be focused on systems administration, whereas I am looking for a solution for application development. It has no packaging process and does not provide automated tests, nor versioned releases. This means it can only be installed via git clone. I tried to make changes to the project, some of which were accepted upstream. Eventually, I got frustrated and started working on my own solution.

Staves to the rescue

Staves[3] is a Gentoo-based container image builder. It is a Python application that can be installed as a system tool and uses a TOML configuration file. Here is a minimal example a of a ./staves.toml config:

name = "test/bash"
packages = ['=app-shells/bash-4.4_p23-r1']
command = ['/bin/bash']

This Staves configuration will create an image named test/bash containing the Gentoo package for Bash in version 4.4_p23-r1. The image's entrypoint is set to /bin/bash so that we will receive a shell prompt when starting the image. We can start the build using the Staves CLI:

$ staves build --builder "staves/x86_64-glibc:0.7.0" "latest"

This will start a Gentoo-based Docker container that creates the image *test/bash:latest".

Builders

As you can see by the CLI command, Staves uses the notion of builders. A builder is a container image that provides a specific toolchain or build environment. The final image on the other hand contains only the runtime dependencies resulting in small image sizes. The idea is to have one builder per platform. You might want to have a builder that contains a Go compiler or a specific version of Ruby. The staves/x86_64-glibc image represents a default builder and provides GCC and Glibc for x86_64. There is also an experimental builder with a Musl toolchain. Let's see how we would create a custom builder for Python 3.6 packages:

name = 'ameto/builder-python3.6'
packages = ['dev-util/staves', '@system']
command = ['python3.6', '-m', 'staves']

[env]
PYTHON_TARGETS="python3_6"
PYTHON_SINGLE_TARGET="python3_6"

We want the new builder to have Staves installed for obvious reasons. We are also installing @system which contains the most basic tools we need to install packages.[4] We are talking about things like tar or wget. A Python interpreter is available by default in the builder, so there is no need to explicitly add it to the package list. We also added an env section that specifies environment variables which go into /etc/portage/make.conf.[5] A builder with this environment will only have Python 3.6 available. This also ensures that the builder only installs the Python 3.6 version of a packages, if the packages supports multiple Python versions. Custom builders are derived from the default builders provided by Staves. Therefore, the build command is almost the same with the slight addition of the --create-builder option:

$ staves build --build-cache builder-python3.6_cache \
    --builder "staves/x86_64-glibc:0.7.0" \
    --create-builder "0.7.0"

Note that we also specified a build cache. Since the builder is Gentoo-based and Gentoo is a source-based distribution that compiles everything from scratch, builds can take a long time. Enabling the build cache will create binary packages after the first run which can be used to speed up subsequent build calls.

Per-package configuration and Package repositories

But there is more to be configured. Ameto's packages are currently not publicly available. They reside in a private package repository. The following configuration is used by Ameto's user registration service:

name = 'ameto/user_registration'
packages = ['=services/user-registration-9999']
command = ['python3', '-m', "user_registration.app"]

[env.no-cache]
FEATURES="-buildpkg"

['=services/user-registration-9999']
env = ['no-cache']

[[repositories]]
name = 'ameto'
type = 'git'
uri = '<redacted>'

This staves.toml configures a Git repository named ameto, which is where the package "services/user-registration" can be found. The ameto repository requires authentication, but staves forwards the user's ssh keys to the builder by default.[6] Therefore, if the user executing Staves has access to the repository, the Staves builder will have access as well.

Also note that the package version is 9999, which denotes that the package is built using the latest VCS sources. Since we do not want to cache the resulting package, we define a named environment that disables caching. The environment is then applied it to the user registration service. That way, we can leverage a build cache for the dependencies while using the same Staves configuration in the Continuous Integration pipeline. Just like package-specific environment variables, it is possible to specify per-package USE Flags[7] or keywords[8].

Staves builds are highly configurable and produce very small images, because they contain only the bare minimum required to run the installed software. The two-stage build process completely isolates build dependencies from the host system and theoretically allows cross compilation for other architectures. I am using Staves productively, but it does not come without drawbacks. I found the build times to be very long (10 – 45 min), even with caching enabled. Although a lot of effort was put into Staves, I still consider it experimental and I am currently gauging public interest in this project. If you would like to see Staves publicly available as free open-source software, please reach out to me.


  1. The Docker developers have started to respond to the ongoing criticism and started to modularize Docker. BuildKit (opens new window), which is responsible for creating Docker images, has made the Dockerfile syntax pluggable. It therefore opens the way for syntax extensions, such as RUN --mount (opens new window) for mounting SSH secrets during build. I recommend the blog post on Build secrets and SSH forwarding in Docker 18.09 (opens new window) by Tõnis Tiigi for more information. ↩︎

  2. I always found rkt to be architecturally superior to Docker. However, the tooling around Docker is so good that it was hard to decide not to use docker at the time. Right now, I see rkt being succeeded by podman (opens new window) or systemd which is adding OCI support for systemd-nspawn (opens new window). ↩︎

  3. A stave is a wooden stake used to form a barrel. You obviously need staves to make a barrel, which is a type of container in the broader sense. I am open for name suggestions 😃 ↩︎

  4. @system is a so-called set of packages defined by Gentoo's package manager. ↩︎

  5. Configration in make.conf affects all package builds. Gentoo supports multiple Python installations in parallel and these two variables make sure that only Python 3.6 is installed. ↩︎

  6. Shameless plug: I have previously written about Fetching packages from authenticated HTTP URIs with Gentoo Portage in general ↩︎

  7. USE Flags are feature switches for Gentoo Packages. For example, disabling the svn USE Flag for the package dev-vcs/git will not install support for git svn commands. ↩︎

  8. Keywords are part of the stabilization process of Gentoo Linux. There is a keyword for each platform, e.g. amd64 denotes a package that is stable on x86_64, whereas ~amd64 denotes a more recent package version that has not been stabilized. ↩︎