Automatic TLS in Kubernetes The Hard Way

Carl Tashian

Follow Smallstep

The whole team here at Smallstep loves Kelsey Hightower's excellent and hugely popular tutorial, Kubernetes The Hard Way. It's a great way to learn about Kubernetes architecture and get a minimalist cluster off the ground. Case in point: During a pandemic lockdown, a friend of Smallstep scavenged Mac Minis and old laptops from their office's conference rooms and used Kubernetes The Hard Way to set up a 22-node "frankencluster" in their apartment!

In Kubernetes The Hard Way, Kelsey gives instructions for creating TLS certificates, which are needed to secure communication between Kubernetes components. Now, Kubernetes uses mutual TLS. While typical web browsing uses server TLS certificates, with mutual TLS, both client and server have to identify themselves to each other using certificates. It's one of the only ways to enable secure, mutually authenticated communications in a cluster.

But something is missing from Kelsey's TLS setup: All of the certificates created in Kubernetes The Hard Way expire in one year. After a year, your cluster will fall down, and all of your certificates will need to be reissued manually. That’s totally fine if you’re using the project to learn Kubernetes, but it’s not okay if you want to set up a long-lived cluster.

As the creators of the step-ca open source, online Certificate Authority, one thing we know a lot about is certificate automation. So, we added automatic TLS certificate renewal to Kubernetes The Hard Way. Here's our pull request.

Central to this is the Certificate Authority (CA). The CA is the authoritative entity that everyone else in the cluster trusts, and it has the power to bind client and server identities (eg. usernames or hostnames) to public encryption keys using a digital signature that anyone can verify. When establishing a connection in a Kubernetes cluster, clients and servers alike must offer a TLS certificate that's digitally signed by the CA server that Kubernetes is configured to trust. They must also prove that they hold the private key associated with that certificate. Only then can a connection be established.

If you're deploying Kubernetes, your cluster will need several certificates (or dozens, or hundreds). And you'll need to know a few things about Public Key Infrastructure (PKI) and certificate management. So, we've augmented Kubernetes The Hard Way with tutorials for setting up your own internal CA server, issuing certificates for your cluster, and configuring automated renewal.

Added benefits of certificate automation

Our pull request adds automated renewal. But it also unlocks some new possibilities.

With Kubernetes The Hard Way, if someone steals a TLS private key from your cluster, there is no way for you to revoke the associated certificate. You have to create a new Certificate Authority and, by extension, create and deploy new certificates for everything in the cluster.

Once you have automation set up for your certificates, you can drop the validity period of your certificates down from a year, to three weeks or even a day. With short-lived certificates, a private key is only valuable for a limited time. If the key is stolen for any reason, the certificate can be revoked at the Certificate Authority, and renewal of the certificate is blocked.

Having an online CA in the mix improves your security posture, and it will keep your cluster running ongoing.