BuildKit is a new and improved tool for building Docker images: it’s faster, has critical features missing from traditional Dockerfiles like build secrets, plus additionally useful features like cache mounting. So if you’re building Docker images, using BuildKit is in general a good idea.

And then there’s Podman: Podman is a reimplemented, compatible version of the Docker CLI and API. It does not however implement all the BuildKit Dockerfile extensions. On its own, then, Podman isn’t as good as Docker at building images.

There is another option, however: BuildKit has its own build tool, which is distinct from the traditional docker build, and this build tool can work with Podman.

Let’s see where Podman currently is as far as BuildKit features, and how to use BuildKit with Podman if that is not sufficient.

Build secrets: supported by Podman

Probably the most useful feature added by Buildkit is support for build secrets; standard Docker builds basically had no good way to securely use something like a package repository password.

The following Dockerfile uses the BuildKit secrets feature:

# syntax = docker/dockerfile:1.3
FROM python:3.9-slim-bullseye
RUN --mount=type=secret,id=mysecret echo "Secret is" && cat /run/secrets/mysecret

With normal Docker, we can build it and pass in a secret like so:

$ export DOCKER_BUILDKIT=1
$ echo iamverysecret > secret.txt
$ docker build --secret id=mysecret,src=secret.txt --progress=plain .
...
#8 [stage-0 2/2] RUN --mount=type=secret,id=mysecret echo "Secret is" && cat /run/secrets/mysecret
#8 sha256:78c75c636e72bba25ea3d82e3c7245f68f060a50b40f99be4573db7d2a0318e3
#8 0.361 Secret is
#8 0.375 iamverysecret#8 DONE 0.4s
...

As of Podman v3.3, this works with Podman as well:

$ docker build --secret id=mysecret,src=secret.txt .
...
STEP 2/2: RUN --mount=type=secret,id=mysecret echo "Secret is" && cat /run/secrets/mysecret
WARN[0000] Failed to decode the keys ["storage.options.ostree_repo"] from "/home/itamarst/.config/containers/storage.conf". 
Secret is
iam verysecret
...

As of version 3.3, Podman does not yet support the ability to read secrets from environment variables, which is supported by Docker v20.10 and later. But the basic support exists, and that’s good enough.

Caching mounts: not supported by Podman

Another useful BuildKit feature is the ability to cache certain directories across builds, which can for example cache pip’s download cache, speeding up rebuilds dramatically when dependencies change.

# syntax = docker/dockerfile:1.3
FROM python:3.9-slim-bullseye
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache pip install -r requirements.txt

Unfortunately, Podman does not yet support this:

$ podman build .
...
STEP 3/3: RUN --mount=type=cache,target=/root/.cache pip install -r requirements.txt
Error: error building at STEP "RUN --mount=type=cache,target=/root/.cache pip install -r requirements.txt": error resolving mountpoints for container "51aeadd15ca2e518ca94a0f866b87a66aea50b5bdc69610bec29fc743338474c": invalid filesystem type "cache"
...

Strictly speaking this feature isn’t necessary, but it is quite nice to have, especially during local development. So how can we access this and other BuildKit features while using Podman?

Using BuildKit with Podman

While BuildKit is built-in to newer versions of Docker, it is also distributed as a separate daemon and command-line tool. And these can run on top of Podman.

Having downloaded the client buildctl from the link above, we can start the daemon in Podman:

$ podman run -d --name buildkitd --privileged \
     docker.io/moby/buildkit:latest

And then we can build images with buildctl:

$ buildctl --addr=podman-container://buildkitd build \
   --frontend dockerfile.v0 \
   --local context=. \
   --local dockerfile=. \
   --export-cache type=inline \
   --output type=docker,name=mynewimage | podman load
...
Loaded image(s): localhost/latest:latest

If you were to run this with docker load, you would have an image called mynewimage visible in docker image ls. With Podman v3.3.1, however, the image ends up being called localhost/latest, which is… not what you’d expect.

The localhost part is just Podman’s way of saying “I don’t know what registry this uses”, so that’s fine, but the latest part is just wrong. I filed a bug.

The image does get loaded though, albeit with the wrong name, and you can run it with podman run. There are likely other ways to solve this by introducing a third tool, but that seems like even more work; if you know of another solution, please email me and I’ll update the article.

Unlike traditional ways of running image builds, the build cache is not stored in Podman’s image registry, it’s stored by the buildkit daemon, which in this case runs inside another Podman container. So if you restart that daemon the cache goes away, unless you’ve made sure to store it in a volume.

BuildKit can also push newly built images directly to a registry, and can use images in the registry as a source for the cache; see the BuildKit docs for details.

Should you use BuildKit with Podman?

In general, the buildctl documentation is pretty lacking. For example, there is no documentation for how you can add labels with buildctl, though I was able to figure it out by analogy with other commands (--opt label:labelname=labelvalue). There is also no complete list that I could find of all options. Possibly it doesn’t exist.

As such, using BuildKit outside of docker build or the newer docker buildx can be a frustrating experience.

If you are a Podman user, directly building with Podman supports the most critical feature BuildKit added: build secrets. So unless you need some of BuildKit’s fancier options, at the moment I would suggest just using Podman directly.