Our open source release process—five years in

Carl Tashian

Follow Smallstep

We’ve been publishing releases of our open source step-ca Certificate Authority and step cryptography CLI tool for over five years now, so our release process is fairly mature at this point.

In this post, I'll share our current process and some of the real-world challenges of publishing open source packages in 2024, and explore opportunities for improvement.

And there’s plenty of room for improvement. I’m not here to brag about how smooth and amazing our release process is. A slightly awkward and lumbering 80% solution that includes a few manual steps is perfect for us. We publish open source releases once a month, at most. Optimizations here don’t have huge payoffs at the moment, so we choose to keep things simpler.

GitHub Actions: effective yet painful

GitHub Actions do most of the heavy lifting for us, as they do for many open- and closed-source projects. And Actions have been great for us overall. The YAML files are pretty easy for most people to follow. GitHub’s documentation and examples are great. And GitHub’s marketplace of Actions reduces toil for us. There's a lot of pieces we can just plug in.

But, Actions remind me of a story I once read about a programmer in the 1970s who—without access to a computer—had to write and mail a letter containing their source code to a computer operator in another region, who would then type it in, run the program, print out the result (or, just as often, a cryptic error), and mail it back to the programmer. True story!

While my example is hyperbolic, latency is a big issue for GitHub Actions. Changing our workflows and testing them is so painful. It can take 20-40 minutes to see the result of any change. And when we are working on a release workflow, it usually involves creating a lot of release candidates for testing purposes.

We also reuse workflows (splitting out small jobs into their own workflows), which is nice but adds coupling that makes it hard to understand when a small change in one workflow will have unintended consequences in another. GitHub Actions encourages this reuse, but doesn’t seem to provide a higher level organization for workflows.

And, unfortunately, GitHub Actions is an environment that discourages local testing. Instead, we use an SSH debugging Action that offers breakpoints and makes the workflow container or VM available via SSH.

So, our overall feeling about GitHub Actions? Muted rapture.

We love GoReleaser

GoReleaser is a useful tool because it knows how to build and package and sign Go projects for several operating systems, platforms, and package managers. It takes a lot of the work out of building a release. And we’re proud to sponsors of Carlos Becker, GoReleaser’s maintainer.

GoReleaser Pro covers the basics of building and publishing for us:

• Creating a GitHub tagged release, with release notes, a changelog, and assets, including:
  • Binary archives for 14 targets (OS + arch pairs)
  • Signed .deb and .rpm files for Linux amd64 (using nFPM)
  • Source code archive (tar)
  • Signatures (via cosign) and checksums for all assets
• Scoop and Winget release for Windows

Release artifacts: What and how we build them

Beyond that, there’s a longer list of tasks GoReleaser doesn’t cover:

• Building & pushing our open source projects in the proper order, to avoid dependency issues (maual)
• Exporting Markdown versions of our CLI help text, and pushing it to our docs website (GitHub Actions)
• Updated version numbers pushed to our install documentation (manual)
• Building Alpine-based Docker images for several Linux targets (GitHub Actions + Docker buildx)
• Building Debian-based, HSM-enabled Docker images with CGO enabled (GitHub Actions + Docker buildx)
• Publishing Helm charts for Kubernetes deployments (manual)
• Pushing binaries to an S3 bucket for further distribution to customers (GitHub Actions)
• Announcing new features via Discord, our newsletter, etc. (manual)

Right now, that’s everything we have to do to release a new version of step or step-ca.

Third-party packagers

All of the above happens before any third-party packagers (Linux distro maintainers, Microsoft, Homebrew, etc) get involved. The packagers—especially the Linux distro maintainers—handle a lot of concerns that we don’t have the capacity or expertise to handle.

Take a step-ca release, for example. Because step-ca is a server, a packager needs to answer questions like:

• How and where should step-ca be installed on a system? What package dependencies must exist? What capabilities should the binary be given?
• What, if any, default config should exist (or be generated)?
• What directories should be used for config, runtime data, autocompletions, secrets, etc?
• What should be in the default systemd service unit for step-ca? (if systemd is in use)
• What kind of security hardening should step-ca have? If a security module like AppArmor or SELinux is in use, what should the security policy be for step-ca?
• What if users want to use step-ca with a Hardware Security Module (HSM)? Should that binary be shipped by default, or shipped in a separate package?
• How should step plugins like step-kms-plugin be packaged?

The packagers—who are often volunteering their time—create and maintain automations that look for new releases from us and repackage our stuff in the hours and days after we publish a release. Third-party packagers will almost always build their own version of step-ca, using their own internal automations and tooling. This way, they can customize the build, they can ensure it runs within their ecosystem properly. And to help mitigate software supply chain attacks, they can commit a reference to the exact commit of our source code that they built, rather than pointing to a prebuilt binary. This way, anyone can potentially verify the build.

So, here are some takeaways so far:

• Building and maintaining and distributing a package for a specific OS or distro is a craft.
• Building a reproducible package for an operating system is difficult!
• Different distros, different crafts, different motivations.
• Announcing and accommodating deprecations and breaking changes is non-trivial
• Accommodating historical documentation is difficult (currently, we only publish documentation for the latest version)

Ideas for the future

Here are some possible areas where we could improve our process. No promises here—just some ideas for the future.

Release more often and reduce the change latency

One way to improve this process would be to publish releases more often. This would put pressure on us to automate what isn’t yet automated and perhaps to decrease our dependence on GitHub Actions for the sake of lower change latency.

Achieve package inclusion in the major distros

Our packages still aren’t officially included in the major Linux distributions like Debian or Fedora—that’s a longer road.

For example, with Fedora, in order to confidently sign a package containing one of our tools, someone needs to create a source package that builds a binary package for distribution. Each distro needs to be able to build our package from a source package. As a result, every go dependency in our project must have its own source package for Fedora. Many of these don’t already exist, so we will need to package those dependencies and be prepared to keep those packages updated as needed.

That’s just one piece of the technical side of the work for one distro. We also have to get these packages through various package review processes, which of course take a bit of time and energy.

Or... maintain our own package repository

Given that package inclusion is such a major effort, and given the number of Linux distros we’d like to be conveniently available for, it may make sense to maintain own our package repository. We haven’t set one up yet.

• Fedora Copr can build packages for distros when new releases pop up on Git
• Gitea has built-in package repositories for various distros

Move to date-based versioning

Major vs. minor vs. incremental semantic versioning is too squishy. The distinctions here are unclear. There’s an assumption that a new major version may have breaking changes and deprecations, but that information needs to be communicated in release notes anyway.

The real thing someone wants to know is: What is the age of this package? Date-based versions convey the age, and they signal effort put into the project. If I’m using a two year old version of a package, I will look assume a lot has changed and I’ll look more closely at the release notes. I’ll also be sure to update before filing any issues.

Maybe for libraries semantic versioning makes sense, but date-based makes more sense for a CLI tool or an operating system.

Produce verifiable builds

Right now we release reproducible builds, in the sense that anyone can check out the build's tag in GitHub, run make, and they should get the same binary that we did.

We're looking into providing verifiable builds so that anyone can examine the binary, see a list of included modules, and verify those against a public module checksum database.

Join the discussion: What are your experiences and tips on managing open source project releases? How do you tackle the challenges and optimize your processes? Find us on Twitter or Discord.