buildah: quick deep dive

README.md

buildah - deep dive

Buildah is a daemonless container build tool to build OCI images. It supports building from Dockerfile, as well as subcommands that map to Dockerfile commands.

View the riveting screencast of demoing this!

Overview

Intended to be suitable as root and not-root usage (with userns). As root, layers and state default to /var/lib/containers/storage/. As non-root, layers and state default to ~/.local/share/containers/storage/.

Building from Dockerfile like docker build -t $USER/myapp . is intended to be a drop-in replacement with buildah bud -t $USER/myapp .. Ultimately container execution is a child process that calls runc. This has the benefit of inheriting cgroups, selinux contexts, etc. Non-root execution opens many use-cases for restricted environments and building in limited privilege containers. With shared libraries for layer storage, it has the benefits of using images and making available images by cri-o and podman. Using the containers/image backend lets the push subcommand copy straight to a registry, a docker daemon, an OCI image layout, or others.

Drawbacks may be:

• it currently doesn't cache the intermediary steps update from Tom Sweeney "add --layers parameter or the envvar BUILDAH_LAYERS"
• The configs are potentially in a couple files/places and a mix of YAML and TOML
• And surely some bugs too ;-)

Share libraries

• logic for fetching and pushing container images is in common library https://github.com/containers/image
• logic for storing and managing the local graph is in common library https://github.com/containers/storage
• logic for container execution is in common library https://github.com/containers/libpod

Examples

A Dockerfile

Let's reproduce a popular base image from its Dockerfile. If you've ever seen FROM golang here is the Dockerfile at the time of writing this. See Dockerfile:

[vbatts@chacha] {master *} ~$ whoami
vbatts
[vbatts@chacha] {master *} ~$ buildah bud -t vbatts/golang -f ./Dockerfile .
STEP 1: FROM buildpack-deps:stretch-scm
Getting image source signatures
Copying blob sha256:55cbf04beb7001d222c71bfdeae780bda19d5cb37b8dbd65ff0d3e6a0b9b74e6
 43.21 MiB / 43.21 MiB [====================================================] 1s
Copying blob sha256:1607093a898cc241de8712e4361dcd907898fff35b945adca42db3963f3827b3
 10.24 MiB / 10.24 MiB [====================================================] 0s
Copying blob sha256:9a8ea045c9261c180a34df19cfc9bb3c3f28f29b279bf964ee801536e8244f2f
 4.14 MiB / 4.14 MiB [======================================================] 0s
Copying blob sha256:d4eee24d4dacb41c21411e0477a741655303cdc48b18a948632c31f0f3a70bb8
 47.75 MiB / 47.75 MiB [====================================================] 1s
Copying config sha256:14385e94c226dd226b368a0fa2cb933b720c7dd26f84c657e650ddff0ae42a9c
 2.58 KiB / 2.58 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
STEP 2: RUN apt-get update && apt-get install -y --no-install-recommends 		g++ 		gcc 		libc6-dev 		make 		pkg-config 	&& rm -rf /var/lib/apt/lists/*
Get:1 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]
Ign:2 http://deb.debian.org/debian stretch InRelease      
[...]
+ go version
go version go1.10.3 linux/amd64
STEP 5: ENV GOPATH /go
STEP 6: ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH
STEP 7: RUN mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"
STEP 8: WORKDIR $GOPATH
STEP 9: COMMIT containers-storage:[vfs@/home/vbatts/.local/share/containers/storage+/run/user/1000/run]localhost/vbatts/golang:latest
Getting image source signatures
Skipping fetch of repeat blob sha256:3b10514a95bec77489a57d6e2fbfddb7ddfdb643907470ce5de0f1b05c603706
Skipping fetch of repeat blob sha256:719d45669b35360e3ca0d53d159c42ca9985eb925a6b28ac38d0ded39a1314e8
Skipping fetch of repeat blob sha256:ce6466f43b110e66d7d3a72600262a1f1b015d9a64aad5f133f081868d4825fc
Skipping fetch of repeat blob sha256:fa0c3f992cbd10a0569ed212414b50f1c35d97521f7e4a9e55a9abcf47ca77e2
Copying blob sha256:83920660763eecdfd7560e6e6e5a1550bd3e7e54340ee6f4ff30e638ba1d761f
 181.06 MiB / 181.06 MiB [==================================================] 6s
Copying config sha256:f56fa07caaa31faab275fd69729caf25b76fc59aad09811edff27acb49dc1d23
 1.72 KiB / 1.72 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
--> f56fa07caaa31faab275fd69729caf25b76fc59aad09811edff27acb49dc1d23
[vbatts@chacha] {master *} ~$ buildah push --tls-verify=false vbatts/golang localhost:5000/vbatts/golang
Getting image source signatures
Copying blob sha256:3b10514a95bec77489a57d6e2fbfddb7ddfdb643907470ce5de0f1b05c603706
 100.64 MiB / 100.64 MiB [==================================================] 6s
Copying blob sha256:719d45669b35360e3ca0d53d159c42ca9985eb925a6b28ac38d0ded39a1314e8
 22.88 MiB / 22.88 MiB [====================================================] 1s
Copying blob sha256:ce6466f43b110e66d7d3a72600262a1f1b015d9a64aad5f133f081868d4825fc
 7.62 MiB / 7.62 MiB [======================================================] 0s
Copying blob sha256:fa0c3f992cbd10a0569ed212414b50f1c35d97521f7e4a9e55a9abcf47ca77e2
 139.60 MiB / 139.60 MiB [==================================================] 9s
Copying blob sha256:33114e88c6e12ca4599c1d45a635fb33139c8c5af0a34b98601f3a6ea38ee060
 504.57 MiB / 504.57 MiB [=================================================] 28s
Copying config sha256:f56fa07caaa31faab275fd69729caf25b76fc59aad09811edff27acb49dc1d23
 1.72 KiB / 1.72 KiB [======================================================] 0s
Writing manifest to image destination
Copying config sha256:f56fa07caaa31faab275fd69729caf25b76fc59aad09811edff27acb49dc1d23
 0 B / 1.72 KiB [-----------------------------------------------------------] 0s
Writing manifest to image destination
Storing signatures

Here we've just rebuilt the base golang:latest image for ourselves, as a non-root user, and then pushed it to a local registry as digest @sha256:f56fa07caaa31faab275fd69729caf25b76fc59aad09811edff27acb49dc1d23.

A build.sh script

Like was mentioned, the subcommands of buildah are largely mapped to the Dockerfile commands, with a few extras. Special configuration commands are rolled up into buildah config i.e. (ENV, WORKDIR, EXPOSE, and others), so check the buildah config --help. I've made the above golang:latest Dockerfile into its equivalent build.sh for the sake of comparison and example.

Note: the environment variables inside the RUN sections need to be escaped so they are not evaluated on the host, but only once evaluated inside the container. And the variables in the config --env PATH...

See ./buildah-build.sh:

[vbatts@chacha] {master *} ~$ sh buildah-build.sh 
++ buildah from buildpack-deps:stretch-scm
Getting image source signatures
Copying blob sha256:55cbf04beb7001d222c71bfdeae780bda19d5cb37b8dbd65ff0d3e6a0b9b74e6
 43.21 MiB / 43.21 MiB [====================================================] 1s
Copying blob sha256:1607093a898cc241de8712e4361dcd907898fff35b945adca42db3963f3827b3
 10.24 MiB / 10.24 MiB [====================================================] 0s
Copying blob sha256:9a8ea045c9261c180a34df19cfc9bb3c3f28f29b279bf964ee801536e8244f2f
 4.14 MiB / 4.14 MiB [======================================================] 0s
Copying blob sha256:d4eee24d4dacb41c21411e0477a741655303cdc48b18a948632c31f0f3a70bb8
 47.75 MiB / 47.75 MiB [====================================================] 1s
Copying config sha256:14385e94c226dd226b368a0fa2cb933b720c7dd26f84c657e650ddff0ae42a9c
 2.58 KiB / 2.58 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
+ from=buildpack-deps-working-container
+ buildah run buildpack-deps-working-container sh
Ign:1 http://deb.debian.org/debian stretch InRelease
[...]
+ buildah config --env PATH=/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin buildpack-deps-working-container-2
+ buildah run buildpack-deps-working-container-2 sh
+ buildah config --workingdir '$GOPATH' buildpack-deps-working-container-2
+ buildah commit buildpack-deps-working-container-2 vbatts/golang
Getting image source signatures
Skipping fetch of repeat blob sha256:3b10514a95bec77489a57d6e2fbfddb7ddfdb643907470ce5de0f1b05c603706
Skipping fetch of repeat blob sha256:719d45669b35360e3ca0d53d159c42ca9985eb925a6b28ac38d0ded39a1314e8
Skipping fetch of repeat blob sha256:ce6466f43b110e66d7d3a72600262a1f1b015d9a64aad5f133f081868d4825fc
Skipping fetch of repeat blob sha256:fa0c3f992cbd10a0569ed212414b50f1c35d97521f7e4a9e55a9abcf47ca77e2
Copying blob sha256:3d8a5823dc7a6395abb6abca48b98d910159f5391fc56364c70a00eacaa07d58
 181.06 MiB / 181.06 MiB [==================================================] 6s
Copying config sha256:ac5b465c725ef496ee582bc2112ac624cf1308ba77984fc76f93b3552b3825ed
 1.71 KiB / 1.71 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
ac5b465c725ef496ee582bc2112ac624cf1308ba77984fc76f93b3552b3825ed
+ buildah push --tls-verify=false vbatts/golang docker.usersys.redhat.com/vbatts/golang
Getting image source signatures
Copying blob sha256:3b10514a95bec77489a57d6e2fbfddb7ddfdb643907470ce5de0f1b05c603706
 100.64 MiB / 100.64 MiB [==================================================] 6s
Copying blob sha256:719d45669b35360e3ca0d53d159c42ca9985eb925a6b28ac38d0ded39a1314e8
 22.88 MiB / 22.88 MiB [====================================================] 1s
Copying blob sha256:ce6466f43b110e66d7d3a72600262a1f1b015d9a64aad5f133f081868d4825fc
 7.62 MiB / 7.62 MiB [======================================================] 0s
Copying blob sha256:fa0c3f992cbd10a0569ed212414b50f1c35d97521f7e4a9e55a9abcf47ca77e2
 139.60 MiB / 139.60 MiB [==================================================] 8s
Copying blob sha256:f0da44eed0a3a25d1700cfa5da7a28b0e2a0c03d25b686d463946d1c7241779c
 504.57 MiB / 504.57 MiB [=================================================] 26s
Copying config sha256:ac5b465c725ef496ee582bc2112ac624cf1308ba77984fc76f93b3552b3825ed
 1.71 KiB / 1.71 KiB [======================================================] 0s
Writing manifest to image destination
Copying config sha256:ac5b465c725ef496ee582bc2112ac624cf1308ba77984fc76f93b3552b3825ed
 0 B / 1.71 KiB [-----------------------------------------------------------] 0s
Writing manifest to image destination
Storing signatures

A more advanced build.sh script

When building a minimal container image, or building the bootstrap base image, there ends up being a trick somewhere that produces the initial rootfs that's extracted into a FROM scratch image layer. The discouragement for this approach is that it relies tightly to the host environment that is used to build it.

See ./buildah-rootfs.sh:

[vbatts@chacha] {master *} ~$ grep NAME /etc/*release
/etc/os-release:NAME=Fedora
/etc/os-release:PRETTY_NAME="Fedora 28 (Workstation Edition)"
/etc/os-release:CPE_NAME="cpe:/o:fedoraproject:fedora:28"
[vbatts@chacha] {master *} ~$ sudo sh buildah-rootfs.sh 
++ buildah from scratch
+ from=working-container-2
++ buildah mount working-container-2
+ scratchmnt=/var/lib/containers/storage/overlay/befab1d1b0301c3b569a379faba21a3c5e0ac6f8e5cca57a5302ec7dbba9923c/merged
+ dnf install --installroot /var/lib/containers/storage/overlay/befab1d1b0301c3b569a379faba21a3c5e0ac6f8e5cca57a5302ec7dbba9923c/merged --release 28 --setopt install_weak_deps=false -y bash coreutils
Copr repo for bazel owned by vbatts                                                                                                   5.8 kB/s | 1.8 kB     00:00    
Copr repo for dnf-update-timer owned by vbatts                                                                                        3.5 kB/s | 1.1 kB     00:00    
Copr repo for envoy owned by vbatts                                                                                                    14 kB/s | 4.1 kB     00:00    
Copr repo for go-fonts owned by vbatts                                                                                                6.3 kB/s | 1.8 kB     00:00    
Fedora 28 - x86_64 - Updates                                                                                                          5.0 MB/s |  22 MB     00:04    
Fedora 28 - x86_64                                                                                                                    1.3 MB/s |  60 MB     00:46    
[...]
+ dnf clean --installroot /var/lib/containers/storage/overlay/bf46a643c60352abeca04dba9dc6128e38589af59ce7a21030234157e3582aa5/merged all
86 files removed
+ buildah unmount working-container-3
771f0aed1c8bb569672fca809addac47c93e03d37e9469e96111f96c916a202b
+ buildah config --cmd /bin/bash --created-by vbatts@chacha working-container-3
+ buildah commit working-container-3 vbatts/bash
Getting image source signatures
Copying blob sha256:bf92ee31dd0772c8ba762a07d29564ae171c66a63e32ea60e7dbc23a44c9c465
 81.80 MiB / 81.80 MiB [====================================================] 3s
Copying config sha256:1fc80722dca3c1312211b9e4808b3cb6917f438b86492b2dcef0f28259834808
 347 B / 347 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
1fc80722dca3c1312211b9e4808b3cb6917f438b86492b2dcef0f28259834808
[vbatts@chacha] {master *} ~$ sudo podman run -it --rm localhost/vbatts/bash echo "farts"
farts

Here you can see dnf repos shown from the host, since it using the dnf from the host to bootstrap the container rootfs. This approach may require root, as commands on the host like dnf have constraints. If you were copying a single static binary into the container, this may not need privilege.

More information

• (blog) Getting Started with Buildah
• (blog) Building buildah container images for kubernetes
• ./buildah/demos/
• Red Hat docs on building containers

buildah-build.sh

#!/bin/bash
# adapted from https://github.com/docker-library/golang/blob/fa0223eaa427188e0c70025f557d515129a9973f/1.10/stretch/Dockerfile

set -ex

from=$(buildah from buildpack-deps:stretch-scm)
buildah run ${from} sh <<EOM
        apt-get update && apt-get install -y --no-install-recommends \
		g++ \
		gcc \
		libc6-dev \
		make \
		pkg-config \
	&& rm -rf /var/lib/apt/lists/*
EOM
buildah config --env GOLANG_VERSION=1.10.3 ${from}
buildah run ${from} sh <<EOM
    set -eux; \
	\
# this "case" statement is generated via "update.sh"
	dpkgArch="\$(dpkg --print-architecture)"; \
	case "\${dpkgArch##*-}" in \
		amd64) goRelArch='linux-amd64'; goRelSha256='fa1b0e45d3b647c252f51f5e1204aba049cde4af177ef9f2181f43004f901035' ;; \
		armhf) goRelArch='linux-armv6l'; goRelSha256='d3df3fa3d153e81041af24f31a82f86a21cb7b92c1b5552fb621bad0320f06b6' ;; \
		arm64) goRelArch='linux-arm64'; goRelSha256='355128a05b456c9e68792143801ad18e0431510a53857f640f7b30ba92624ed2' ;; \
		i386) goRelArch='linux-386'; goRelSha256='3d5fe1932c904a01acb13dae07a5835bffafef38bef9e5a05450c52948ebdeb4' ;; \
		ppc64el) goRelArch='linux-ppc64le'; goRelSha256='f3640b2f0990a9617c937775f669ee18f10a82e424e5f87a8ce794a6407b8347' ;; \
		s390x) goRelArch='linux-s390x'; goRelSha256='34385f64651f82fbc11dc43bdc410c2abda237bdef87f3a430d35a508ec3ce0d' ;; \
		*) goRelArch='src'; goRelSha256='567b1cc66c9704d1c019c50bef946272e911ec6baf244310f87f4e678be155f2'; \
			echo >&2; echo >&2 "warning: current architecture (\$dpkgArch) does not have a corresponding Go binary release; will be building from source"; echo >&2 ;; \
	esac; \
	\
	url="https://golang.org/dl/go\${GOLANG_VERSION}.\${goRelArch}.tar.gz"; \
	wget -O go.tgz "\$url"; \
	echo "\${goRelSha256} *go.tgz" | sha256sum -c -; \
	tar -C /usr/local -xzf go.tgz; \
	rm go.tgz; \
	\
	if [ "\$goRelArch" = 'src' ]; then \
		echo >&2; \
		echo >&2 'error: UNIMPLEMENTED'; \
		echo >&2 'TODO install golang-any from jessie-backports for GOROOT_BOOTSTRAP (and uninstall after build)'; \
		echo >&2; \
		exit 1; \
	fi; \
	\
	export PATH="/usr/local/go/bin:\$PATH"; \
	go version
EOM
buildah config --env GOPATH=/go ${from}
buildah config --env PATH='$GOPATH/bin:/usr/local/go/bin:${PATH}' ${from}
# found a fun bug on variable expansion ... https://github.com/projectatomic/buildah/issues/929
# here is a work around if you're not using master (or presumably a newer than 1.3 release)
#buildah config --env PATH="$(buildah run -t ${from} sh -c 'echo -n $GOPATH')/bin:$(buildah run -t ${from} sh -c 'echo -n $PATH')" ${from}

buildah run ${from} sh <<EOM
mkdir -p "\$GOPATH/src" "\$GOPATH/bin" && chmod -R 777 "\$GOPATH"
EOM
buildah config --workingdir \$GOPATH ${from}
buildah commit ${from} ${USER}/golang
buildah push --tls-verify=false ${USER}/golang localhost:5000/${USER}/golang

buildah-rootfs.sh

#!/bin/bash

set -ex

from=$(buildah from scratch)
scratchmnt=$(buildah mount ${from})

dnf install \
    --installroot ${scratchmnt} \
    --release 28 \
    --setopt install_weak_deps=false -y \
    bash coreutils
dnf clean \
    --installroot ${scratchmnt} \
    all
buildah unmount ${from}
buildah config \
    --cmd /bin/bash \
    --created-by "${USER}@${HOSTNAME}" \
    ${from}
buildah commit ${from} ${USER}/bash
buildah push --tls-verify=false ${USER}/bash localhost:5000/${USER}/bash

Dockerfile

FROM buildpack-deps:stretch-scm

# gcc for cgo
RUN apt-get update && apt-get install -y --no-install-recommends \
		g++ \
		gcc \
		libc6-dev \
		make \
		pkg-config \
	&& rm -rf /var/lib/apt/lists/*

ENV GOLANG_VERSION 1.10.3

RUN set -eux; \
	\
# this "case" statement is generated via "update.sh"
	dpkgArch="$(dpkg --print-architecture)"; \
	case "${dpkgArch##*-}" in \
		amd64) goRelArch='linux-amd64'; goRelSha256='fa1b0e45d3b647c252f51f5e1204aba049cde4af177ef9f2181f43004f901035' ;; \
		armhf) goRelArch='linux-armv6l'; goRelSha256='d3df3fa3d153e81041af24f31a82f86a21cb7b92c1b5552fb621bad0320f06b6' ;; \
		arm64) goRelArch='linux-arm64'; goRelSha256='355128a05b456c9e68792143801ad18e0431510a53857f640f7b30ba92624ed2' ;; \
		i386) goRelArch='linux-386'; goRelSha256='3d5fe1932c904a01acb13dae07a5835bffafef38bef9e5a05450c52948ebdeb4' ;; \
		ppc64el) goRelArch='linux-ppc64le'; goRelSha256='f3640b2f0990a9617c937775f669ee18f10a82e424e5f87a8ce794a6407b8347' ;; \
		s390x) goRelArch='linux-s390x'; goRelSha256='34385f64651f82fbc11dc43bdc410c2abda237bdef87f3a430d35a508ec3ce0d' ;; \
		*) goRelArch='src'; goRelSha256='567b1cc66c9704d1c019c50bef946272e911ec6baf244310f87f4e678be155f2'; \
			echo >&2; echo >&2 "warning: current architecture ($dpkgArch) does not have a corresponding Go binary release; will be building from source"; echo >&2 ;; \
	esac; \
	\
	url="https://golang.org/dl/go${GOLANG_VERSION}.${goRelArch}.tar.gz"; \
	wget -O go.tgz "$url"; \
	echo "${goRelSha256} *go.tgz" | sha256sum -c -; \
	tar -C /usr/local -xzf go.tgz; \
	rm go.tgz; \
	\
	if [ "$goRelArch" = 'src' ]; then \
		echo >&2; \
		echo >&2 'error: UNIMPLEMENTED'; \
		echo >&2 'TODO install golang-any from jessie-backports for GOROOT_BOOTSTRAP (and uninstall after build)'; \
		echo >&2; \
		exit 1; \
	fi; \
	\
	export PATH="/usr/local/go/bin:$PATH"; \
	go version

ENV GOPATH /go
ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH

RUN mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"
WORKDIR $GOPATH