SBOM with signed Apptainer containers on BeeGFS

Introduction

To give credit where credit is due - this post is inspired by a recent update from Singularity in which they noted that Syft has added support for the SIF format used by Singularity (and Apptainer).

I thought to give this workflow a try and write a post that outlines additional advantages of Apptainer containers and I got some help from an Apptainer expert which helped me save time.

As most readers know, NetApp has an E-Series BeeGFS solution which I often write about.

Although E-Series arrays have very strong security features, and NetApp’s BeeGFS solution is based on BeeGFS with enterprise features and ACLs are available, BeeGFS itself currently has no encryption. This may pose challenges in some environments.

Many NetApp users keep highly secure data on ONTAP systems or StorageGRID (Object Storage), but there are cases (performance, cost, etc.) where keeping such data on BeeGFS is desired.

The other challenge is the security and portability of applications, where Singularity-type containers shine. SBOM-features of Syft only add to those strengths and now that Syft supports SIF (Singularity) archives, we can do this:

• Create Singularity/Apptainer containers on BeeGFS
• Package data with applications if you want to create an easy user experience or ship application with data
• Cryptographically sign (clear-sign) such containers
• Create SBOM for such containers, so that users can inspect them without scanning those containers if they want to
• Check containers for vulnerabilities
• Encrypt and sign Singularity containers on BeeGFS (which makes data secure, but you can no longer scan them, obviously)
• Do other neat stuff

Build data-app container and check its inventory

Download or copy data (logo.png) your application will use and prepare a definition file that combines your app and data.

In our case that file is caddy.def:

Bootstrap: docker
From: caddy:latest

%files
  logo.png /srv/view-me.png

%startscript
  caddy file-server --browse --listen :8080 --root /srv

Build:

apptainer build caddy.sif caddy.def

Use apptainer key sub-commands on this new image:

• create (a new PGP key)
• sign caddy.sif
• verify caddy.sif

Repeat the build with a different container file name for the encrypted version:

sudo apptainer build --passphrase caddy-encrypted.sif caddy.def

Create a software BOM for the first application-data image:

syft singularity:caddy.sif -o json=caddy.sbom.cdx.json

Check the SBOM file for known and published vulnerabilities:

This works for signed Apptainer containers, too:

$ grype sbom:caddy-signed.sbom.cdx.json
 ✔ Vulnerability DB        [no update available]
 ✔ Scanned image           [2 vulnerabilities]

NAME                        INSTALLED  FIXED-IN  TYPE       VULNERABILITY   SEVERITY 
google.golang.org/protobuf  v1.28.0              go-module  CVE-2015-5237   High      
google.golang.org/protobuf  v1.28.0              go-module  CVE-2021-22570  High   

Our app exposes data in a read-only fashion so we’ll proceed anyway. We could build a newer Caddy container to fix this, of course.

Once container is cryptographically signed, application and data in it are implicitly trusted for trusted PGP keys.

$ apptainer verify caddy-encrypted.sif
Verifying image: caddy-encrypted.sif
[LOCAL]   Signing entity: Sean XXXXXXX <[email protected]>
[LOCAL]   Fingerprint: 9D1AC07D2BA49D2BB70B3E9D59624FF646415296
Objects verified:
ID  |GROUP   |LINK    |TYPE
------------------------------------------------
1   |1       |NONE    |Def.FILE
2   |1       |NONE    |JSON.Generic
3   |1       |NONE    |JSON.Generic
4   |1       |NONE    |FS
Container verified: caddy-encrypted.sif

Use it

Now that we got our app+data container, we can move it around without the fear of tampering.

The container itself could be uploaded to container registry, but I don’t expect that large containers that pack data would be commonly stored in container registry. But sometimes you may want to do that. Random thoughts:

• Signed containers without data can of course be stored in container registry (see this example of using AWS Elastic Container Registry to store Apptainer images)
• Tamper-proof (i.e. signed) containers that include data (which can be very large!) are suitable for keeping on BeeGFS with properly configured ACLs if data is not very sensitive
• Encrypted signed containers that package data can be kept on BeeGFS even if data in the container is sensitive
  • Apptainer SBOM must be generated before the container is encrypted. SBOM reports can be clear-signed with PGP and left on BeeGFS
  • Vulnerability scan result can also be included with the image; otherwise output file would have to be signed as well (so that it cannot be tampered with)
• You must be careful about storing passwords in containers (best approach: don’t do it). For example, even in caddy-encrypted.sif it’s possible to see startup command. If there was a password in that line, that could be bad - especially for signed-only data+app containers:

#!/usr/bin/env run-singularity
SIF_MAGIC-TON-OF-BINARY-STUFF-HERE
c02Bootstrap: docker-TON-OF-BINARY-STUFF-HERE
From: caddy:latest

%files
  logo.png /srv/view-me.png

%startscript
  caddy file-server --browse --listen :8080 --root /srv

Once our app+data container is ready, we can keep it on BeeGFS and run it when we need to.

Starting and stopping these instances (one at a time, if you haven’t changed application port for different builds) is easy:

$ apptainer instance start --passphrase caddy-encrypted.sif web1
$ apptainer instance stop web1

Common steps in one screenshot:

Knowing that our app runs at *:8080, we can access it in the browser:

Access application data:

Conclusion

Syft support for the SIF format adds valuable features to Apptainer containers - not just “app+data containers”, but all Apptainer containers.

NetApp BeeGFS users benefit from other Apptainer features (see this post, for example), but signing and encryption not only prevents tampering, but it also provides a simple way to bundle data and application SBOM files and vulnerability reports within Apptainer containers.

While SBOM and vulnerability data should probably be easy to access via an API and in an automated fashion, nothing prevents us from also storing SBOM files and vulnerability reports elsewhere. Today many users still do not have a formal way to store and query SBOMs and vulnerability scan results, so being able to include that in data and app containers and archives can be useful.

The approach of packaging data with applications is useful for other reasons as well. Aside from the obvious (reproducible, tamper-free “unified” containers), this approach can be valuable to hybrid cloud users.

Data on encrypted filesystem normally loses protection as soon as it moves to another location (there are ways to solve that, but they can be expensive, complex, or inconvenient). Once a file is copied to a remote location it needs to be checked for integrity (and maybe not just while copying, but before each and every run) and then accessed by application which the destination may not have available in the convenient container format. Signed containers are portable, tamper proof (even from bit rot), easy to verify and use, while encrypted signed containers offer additional security to users who need it.