MySQL TLS — Practical Zero Trust

We've created a systemd-based certificate renewal timer that works with step. Check out our documentation on Renewal using systemd timers for background on how these timers work.

To install the certificate renewal unit files, run:

The renewal timer will check your certificate files every five minutes and renew them after two-thirds of their lifetime has elapsed.

We've created a systemd renewal timer for renewing certificates with a Smallstep CA (see non-ACME Linux instructions). However, we haven't yet investigated how to modify that timer for ACME use cases. We're working on it, but feel free to contribute this content directly on GitHub. At this point, you will need to manually create the cert-renewer@.service and cert-renewer@.timer template files.

To renew and hot-reload the MySQL server certificate, we will need a MySQL-specific systemd override file that can tell MySQL to refresh its certificates. To install the override, run:

You'll need a database user (eg. reloader@localhost) that has the CONNECTION_ADMIN privilege, for reloading the certificate:

Replace password with a secure password. Finally, create the reloader@localhost.cnf credential file for the mysql client, which will look like:

When the certificate is renewed, the renewal service will use the Unix domain socket to connect as reloader@localhost. TLS will not be enabled for that connection, because it is already considered secure. That's good because we want our certificate renewer to reach the server even if the server certificate is expired.

To start the renewal timer, run:

You'll see that the timer is active, by checking the output of systemctl list-timers.

When working with certificate in containers, you'll need to decide how and where and when to handle each of these tasks:

There are several approaches one could take for full certificate automation. Here are a few:

For an in-depth exploration of these approaches, with examples and tradeoffs, see our blog post Automating TLS Certificate Management in Docker.

To support the broadest range of environments where containers are run, we prefer the self-contained approach. While it requires building a custom Docker image, it solves the problems of root distribution, enrollment, renewal, and deployment without requiring volume mounts or changes to the host environment.

See our Docker image for MySQL with TLS for a complete, self-contained example that can be tailored to your needs.

See our Docker image for MySQL with TLS for a complete, self-contained example that can be tailored to your needs.

There are various approaches for configuring TLS certificate renewal in Kubernetes. For this tutorial, we will assume you are using Helm and will manage certificate renewal using the open source cert-manager project. You may repurpose various pieces for your own configuration by following the cert-manager or MySQL documentation.

First, install cert-manager in your cluster with Helm:

In cert-manager, a custom resource type Certificate represents each TLS certififcate and specifies how to store it.

Create a new Certificate resource db-certificate.yaml for your MySQL server. Make sure you specify the same DNS name configured above, and keep track of the Kubernetes secret name where cert-manager will store your certificate and private key.

Customize the Certificate name, secretName, and issuer name to suit your own naming style in your Kubernetes cluster, then apply it with kubectl.

This configuration specifies that cert-manager should issue and renew a TLS certificate with the DNS name db.example.net and store the certificate and private key in a Kubernetes secret named db-tls. The certificate is valid for 24 hours, and cert-manager will automatically renew it before expiration and update the db-tls secret.

You'll also notice the above Certificate resource has an issuerRef specifying a cert-manager StepIssuer resource named my-ca. Before cert-manager knows how to use your CA to issue and renew your certificate, we'll need to create that StepIssuer resource and configure it to point to your private CA.

First, let's grab the name and key kid from the JWK provisioner we created earlier.

Next, get a base64-encoded version of your CA root certificate:

Create my-ca-issuer.yaml and fill in the CA URL and provisioner details with your own. caBundle refers to the base64-encoded version of your root certificate.

You'll notice the passwordRef configuration above, which declares a secret where step-issuer can expect to find your JWK provisioner password. You can name this whatever you'd like. Create the secret in your cluster, then apply your StepIssuer resource.

If all went as expected now that both your Certificate and your StepIssuer are configured, cert-manager should have reached out to your Smallstep CA, issued (or begun issuing) the certificate, and created the db-tls secret with your certificate and private key.

You'll also notice the above Certificate resource has an issuerRef specifying a cert-manager Issuer resource named my-ca. Before cert-manager knows how to use your CA to issue and renew your certificate, we'll need to create that Issuer resource and configure it to point to your private ACME CA (or RA if you've configured one). Create my-ca-issuer.yaml and fill in the CA URL with your own. The Issuer will keep an account key in a Kubernetes secret which is configurable here as well.

Then apply:

To verify that the requester of the certificate actually controls the DNS name specified in your ACME request (from dnsName in your Certificate), the ACME CA will send an HTTP challenge request to that DNS name and expects to receive particular response. For this reason, your Kubernetes cluster must have some HTTP ingress configured, and your DNS name must resolve to that ingress. In the above example, we assume your ingress has a class name nginx, and cert-manager will automatically intercept ACME requests to that ingress to solve the HTTP01 ACME challenge sent by your CA. This effectively "authenticates" your ACME client with the CA, and the CA will then sign the certificate.

If all went as expected now that both your Certificate and your Issuer are configured, cert-manager should have issued (or begun issuing) the certificate and created the db-tls secret with your certificate and private key. We'll reference this secret later when we configure MySQL to use this certificate and private key for TLS.

Now that your certificate is issued (and kept updated) in the db-tls secret, you'll need to configure your MySQL deployment to mount that secret to disk, then to reload the certificate (or restart its process) each time the certificate is updated on disk. A general approach to certificate reloading and secret rotation in Kubernetes is to use Reloader or a similar tool to watch the db-tls secret, and run a rolling upgrade of your deployment when it changes.