Building OCI images with sourcehut builds

[GunnarGrop] - - 11 mins read

Abstract

More and more I find myself needing custom built OCI images, for various purposes. Most recently has been this blog, in fact. Since I’m hosting this blog in Kubernetes, one of the most straight forward ways of keeping it updated and version controlled was to build an OCI image and include the relevant Hugo files in it.

While I’m under no illusions that it’s the best way to deploy a Hugo blog, it is very easily maintained and version controlled. I’ve also quite recently migrated all of my git repositories over to sourcehut, and this combined with my, maybe non-traditional, way of deploying this blog has forced me to have a look as sourcehut’s CI/CD system. Simply called “builds”.

I might do a more in-depth review of sourcehut builds at some point, but for now I thought it might be interesting to share how I’ve gone about building, publishing and signing OCI images using it.

What I’ll be doing in this guide is the following:

  • Create a sourehut build manifest.
  • Building a multi-arch OCI image with podman.
  • Publishing the OCI image to a registry.
  • Signing the manifest digest using cosign.

If you want to take a look at a real life example, just take a look at the source code for this blog.

Prerequisites

There are a few things you’ll need to follow this guide:

  • A sourcehut account (of course)
  • An OCI registry to push images to (Docker Hub, etc)
  • A sigstore key pair (for signing images)

sigstore key pair

If you do not yet have a sigstore keypair, you can simply create one using cosign.

cosign generate-key-pair

You’ll be prompted for a password, and then the following files will be created in you current directory

  • cosign.pub (your public key)
  • cosign.key (your private key)

To sign anything with your private key, you’ll need the password you supplied, of course. And this should go without saying, but NEVER share your private key or password with anyone.

Registry authfile

To effectively, and more safely, handle our OCI registry credentials, we should create an “authfile”.

An authfile looks like this:

{
        "auths": {
                "registry-1.docker.io": {
                        "auth": "dXNlcm5hbWU6cGFzc3dvcmQ="
                }
        }
}

Where a top-level block “auths” contains a dictionary of registry URIs, and each of those registry elements has an “auth” field, containing a base64 encoded representation of the string “username:password”.

You can generate the auth string by just piping your credentials to the base64 command (on UNIX systems):

echo "my-user@example.com:my-secret-password" | base64 --wrap 0

Another handy thing of using an authfile is that every tool we’re going to use in this tutorial can use it. That includes:

  • podman (or docker, if you want)
  • cosign
  • skopeo

For this tutorial I’ll juse assume we’re dealing with Docker Hub, like in the example above.

Creating sourcehut Secrets

Now that we have our sigstore keys and out authfile, we need to create three different secrets in sourcehut, that we’ll use in our pipeline.

If you go to Manage secrets in sourcehut, you’ll have the option to create three types of secrets.

  • SSH Key
  • PGP Key
  • File

We’ll create three secrets of the “File” type.

Secret Name Secret Content File Path File Mode
DOCKER_HUB_AUTHFILE <your authfile> ~/.auth 600
COSIGN_KEY <your cosign private key> ~/.cosign.key 600
COSIGN_PASSWORD <your cosign private key password> ~/.cosign.password 600

When we’re using these secrets in our pipeline later, they will be mounted as files in our build environment, at the specified paths and with the specified file permissions.

Task: Setup

Now that we have our secrets we just need to create a build manifest for sourcehut builds to use. You can do this either by creating a files called “.build.yml” in the root in one of your git repositories. Or you can created a directory called “.builds” and place any “.yml” files in it.

According to the documentation, you can effectively just have up to 4 different build manifests (files) under “.builds”. If you have more than 4, then a random set of 4 builds will be selected when you’re triggering the CI.

For this tutorial though, just one build manifest is plenty. We’ll create a file in our git repository that looks like this. We’ll create a file in our git repository called “.builds/publish.yml”

Now we’ll start actually writing the build manifest. First of all, we’ll use the alpine/latest image to run our CI. But you can choose any of the available images here.

---
image: alpine/latest

Secondly we’ll install some packages in the build environment. Most of them are obvious, but iptables it needed for podman to be able to function correctly, and jq is just a handy tool to have.

packages:
  - iptables
  - podman
  - cosign
  - skopeo
  - jq

Then, we’ll specify which secrets we want to use. This is done by specifying a list of UUIDs, then ones you see when you go to the “Manage secrets” page.

secrets:
  - 06965d05-4554-4832-8a60-628e1a219a39  # DOCKER_HUB_AUTHFILE
  - f3e67338-52fb-4d8d-aa4d-7529bbf74cef  # COSIGN_KEY
  - a99495b8-32a3-4b37-9c9f-7e6772dbfba2  # COSIGN_PASSWORD

Then we’ll specify some environment variables for our build environment. Note that the PROJECT variable is just the full name of the git repository, which is what the directory containing the source code will be called.

environment:
  PROJECT: grop.dev                      # Name of the git repo (will also be the OCI image name)
  OCI_PLATFORM: linux/amd64,linux/arm64  # Platforms to build for
  OCI_REGISTRY: registry-1.docker.io     # Which registry to upload the image to
  OCI_REPOSITORY: gunnargrop             # Which repository to upload the image to

Lastly, we’ll write the tasks list. I usually like to have a setup task to take care of some basic stuff that are needed for the rest of the tasks.

Here you can see that I’ve included a task that exits if it’s not triggered buy a git tag (as opposed to a commit, etc). You can do this however you like, but I like running my builds only when I’m pushing a new tag to the repository. The build system gives us a handy function called “complete-build” that we can call, cleanly exiting the CI.

Note also that when setting environment variables for other tasks to use, we need to write them to the “~/.buildenv” file. This is a special file that is sourced by all tasks.

tasks:
  - setup: |
      # Exit pipeline if NOT triggered by tag
      if [[ "$GIT_REF" != refs/tags/* ]]; then
          complete-build
      fi

      # Set environment variables
      set +x
      cd $PROJECT
      # Let the VERSION variable be the name of the latest git tag
      echo "VERSION=$(git tag --sort=-v:refname | awk '/^\d/' | head -n 1)" >> ~/.buildenv
      set -x

      # Setup for podman, which requires cgroups to function
      sudo rc-update add cgroups
      sudo rc-service cgroups start

Note here that I’ve encapsulated the setting of environment variables by set +x and set -x. This isn’t strictly necessary here, it’s just something I do in every build manifest, in case I want to handle secrets here. And you should always encapsulate your handling of secrets like this, since this prevents the commands to be written to stdout.

set +x
# For example
MY_SECRET_VAR="$(cat ~/.my-secret-password-file)"
set -x

So, here is the full build manifest thus far.

---
image: alpine/latest

packages:
  - iptables  # Needed for podman
  - podman
  - cosign
  - skopeo
  - jq

secrets:
  - 06965d05-4554-4832-8a60-628e1a219a39  # DOCKER_HUB_AUTHFILE
  - f3e67338-52fb-4d8d-aa4d-7529bbf74cef  # COSIGN_KEY
  - a99495b8-32a3-4b37-9c9f-7e6772dbfba2  # COSIGN_PASSWORD

environment:
  PROJECT: grop.dev
  OCI_PLATFORM: linux/amd64,linux/arm64
  OCI_REGISTRY: registry-1.docker.io
  OCI_REPOSITORY: gunnargrop

tasks:
  - setup: |
      # Exit pipeline if NOT triggered by tag
      if [[ "$GIT_REF" != refs/tags/* ]]; then
      complete-build
      fi

      # Set environment variables
      set +x
      cd $PROJECT
      # Let the VERSION variable be the name of the latest git tag
      echo "VERSION=$(git tag --sort=-v:refname | awk '/^\d/' | head -n 1)" >> ~/.buildenv
      set -x

      # Setup for podman, which requires cgroups to function
      sudo rc-update add cgroups
      sudo rc-service cgroups start

Task: Build

Now we need a task to actually build our OCI image. I’d recommend always creating a manifest with podman, instead of just building images. Creating a manifest simplifies building for multiple architectures, as well as making it possible to push every image in the manifest with just one command.

Assuming your Dockerfile is in the root of the repository, we can just cd into the directory and run podman build.

- build_oci: |
    sudo podman manifest create "$PROJECT:$VERSION"

    cd $PROJECT
    sudo podman build \
    --file "Dockerfile" \
    --platform "$OCI_PLATFORM" \
    --manifest "$PROJECT:$VERSION" \
    .

Task: Publish

Now we need to publish the OCI image to our registry. Since we’re only running this CI when pushing a git tag, we can assume the git tag is named after the version we want to set, and because of the setup step we have a handy variable called VERSION that we can use.

Note the use of the --authfile flag here. We’re using the authfile mounted at “~/.auth”, from the secret that we created earlier.

Whenever I push a new version of an image, I also like to push it to the :latest tag, but you can do this however you like.

- publish_oci: |
    # Push version
    sudo podman manifest push \
    --authfile ~/.auth \
    "$PROJECT:$VERSION" \
    "$OCI_REGISTRY/$OCI_REPOSITORY/$PROJECT:$VERSION"

    # Push latest
    sudo podman manifest push \
    --authfile ~/.auth \
    "$PROJECT:$VERSION" \
    "$OCI_REGISTRY/$OCI_REPOSITORY/$PROJECT:latest"

Task: Sign

Now, after we’ve pushed the images, we’ll sign the index using cosign, using the private key we created earlier.

First we need to copy our authfile to “$HOME/.docker/config.json”, since cosign is hardcoded to read the authfile from this location. We’ll also set the environment variable COSIGN_PASSWORD to contain the password to our private key, from the secret that we created earlier. This is a special environment variable that cosign uses, so we don’t have to supply cosign commands with the password. These tasks contains sensitive values, so we’ll encapsulate them in an set +x set -x block.

Then we’ll also use skopeo to get the digest of the OCI index where we pushed our images in the last task. We can use our regular authfile here since skopeo inspect takes the --authfile flag. We’ll name the variable DIGEST. This does not need to be done for both the :$VERSION and the :latest image, since they contain the same images, and have the same digests.

Lastly, we’ll sign the images with cosign, using the digest we got from skopeo and the private key we created earlier. This will push a signature file to the OCI repository, which we can use to verify with our public key later.

- sign_oci: |
    set +x
    mkdir ~/.docker
    cp ~/.auth ~/.docker/config.json
    export COSIGN_PASSWORD=$(cat ~/cosign.password)
    set -x

    DIGEST=$(skopeo inspect --authfile ~/.auth docker://$OCI_REGISTRY/$OCI_REPOSITORY/$PROJECT:$VERSION | jq -r ".Digest")

    cosign sign --yes \
    --key ~/cosign.key \
    "$OCI_REGISTRY/$OCI_REPOSITORY/$PROJECT@$DIGEST"

Complete Build Manifest

Here is the complete file that we’ve written.

---
image: alpine/latest

packages:
  - iptables  # Needed for podman build
  - hugo
  - podman
  - cosign
  - skopeo
  - jq

secrets:
  - 06965d05-4554-4832-8a60-628e1a219a39  # DOCKER_HUB_AUTHFILE
  - f3e67338-52fb-4d8d-aa4d-7529bbf74cef  # COSIGN_KEY
  - a99495b8-32a3-4b37-9c9f-7e6772dbfba2  # COSIGN_PASSWORD

environment:
  PROJECT: grop.dev
  OCI_PLATFORM: linux/amd64,linux/arm64
  OCI_REGISTRY: registry-1.docker.io
  OCI_REPOSITORY: gunnargrop

tasks:
  - setup: |
      # Exit pipeline if NOT triggered by tag
      if [[ "$GIT_REF" != refs/tags/* ]]; then
          complete-build
      fi

      # Set environment variables
      set +x
      cd $PROJECT
      # Let the VERSION variable be the name of the latest git tag
      echo "VERSION=$(git tag --sort=-v:refname | awk '/^\d/' | head -n 1)" >> ~/.buildenv
      set -x

      # Setup for podman, which requires cgroups to function
      sudo rc-update add cgroups
      sudo rc-service cgroups start

  - build_oci: |
      sudo podman manifest create "$PROJECT:$VERSION"

      cd $PROJECT
      sudo podman build \
      --file "Dockerfile" \
      --platform "$OCI_PLATFORM" \
      --manifest "$PROJECT:$VERSION" \
      .

  - publish_oci: |
      # Push version
      sudo podman manifest push \
      --authfile ~/.auth \
      "$PROJECT:$VERSION" \
      "$OCI_REGISTRY/$OCI_REPOSITORY/$PROJECT:$VERSION"

      # Push latest
      sudo podman manifest push \
      --authfile ~/.auth \
      "$PROJECT:$VERSION" \
      "$OCI_REGISTRY/$OCI_REPOSITORY/$PROJECT:latest"

  - sign_oci: |
      set +x
      mkdir ~/.docker
      cp ~/.auth ~/.docker/config.json
      export COSIGN_PASSWORD=$(cat ~/cosign.password)
      set -x

      DIGEST=$(skopeo inspect --authfile ~/.auth docker://$OCI_REGISTRY/$OCI_REPOSITORY/$PROJECT:$VERSION | jq -r ".Digest")

      cosign sign --yes \
      --key ~/cosign.key \
      "$OCI_REGISTRY/$OCI_REPOSITORY/$PROJECT@$DIGEST"

Conclusion

We’re all done! Now when you push a new tag to your git repository, this CI will run, build your images, publish them, and sign them.

Once it’s completed, you can use cosign with your public key to verify that it’s signed by a trusted source. In my case, it would look like this:

cosign verify --key cosign.pub docker.io/gunnargrop/grop.dev:latest

Where I see this output:

Verification for index.docker.io/gunnargrop/grop.dev:latest – The following checks were performed on each of these signatures:

  • The cosign claims were validated
  • Existence of the claims in the transparency log was verified offline
  • The signatures were verified against the specified public key

[ { “critical”: { “identity”: { “docker-reference”: “registry-1.docker.io/gunnargrop/grop.dev” }, “image”: { “docker-manifest-digest”: “sha256:91caaa17007095a67be411215c97d989b127c2c9179df6d72258698361797988” }, “type”: “cosign container image signature” }, “optional”: { “Bundle”: { “SignedEntryTimestamp”: “MEYCIQC0sy5/hKB2+lmJEHOLsyzoDuxG/B9P0sxiOs3wWkozbgIhAMfqqwn/0erKYDc0jRo2x3Cmzd7aNC8oOWOnfUzCaEeb”, “Payload”: { “body”: “eyJhcGlWZXJzaW9uIjoiMC4wLjEiLCJraW5kIjoiaGFzaGVkcmVrb3JkIiwic3BlYyI6eyJkYXRhIjp7Imhhc2giOnsiYWxnb3JpdGhtIjoic2hhMjU2IiwidmFsdWUiOiJiNWU2ZTQyYjViZWQ2ZmJlMjFlMDZiYzk1ZGFhZGU4MmE4Mjg4MWEyNGVhN2U1NjRmNDk1OTY4MzNkZTUyMzQyIn19LCJzaWduYXR1cmUiOnsiY29udGVudCI6Ik1FVUNJUUQrSVFMZzhBTjVJRmh5S09PSEYvcXladWttTHRWZzZTZFR5WlhlaWROcFV3SWdHd2FaS3RHaWJKenVMNi9zQ294c0t0elVPb0h4cC8rYWpFRjAxR0gyV0g4PSIsInB1YmxpY0tleSI6eyJjb250ZW50IjoiTFMwdExTMUNSVWRKVGlCUVZVSk1TVU1nUzBWWkxTMHRMUzBLVFVacmQwVjNXVWhMYjFwSmVtb3dRMEZSV1VsTGIxcEplbW93UkVGUlkwUlJaMEZGUm1wNVZIVlZaVUYxV1VSVFdXZHJUVWwwT1ZCNk5GWlZVMDFUVFFwTlYydHhRbTV3ZVhCblNXUTNOMUZLUlN0U05FNUNVVGMwWWpReWMzQlVja3hZYWxsV2FGcHJXakl6WVRrdlQyWnpSVzl3UjFGMWVuUjNQVDBLTFMwdExTMUZUa1FnVUZWQ1RFbERJRXRGV1MwdExTMHRDZz09In19fX0=”, “integratedTime”: 1760886624, “logIndex”: 622102447, “logID”: “c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d” } } } } ]

Yay! It works!

If it doesn’t work, if there is no signature at all, or if the signature is by an untrusted source, the output will look like this:

Error: no matching signatures: invalid signature when validating ASN.1 encoded signature error during command execution: no matching signatures: invalid signature when validating ASN.1 encoded signature