Introducing Smallstep ACME RA: Automating internal TLS with ACME + Google CAS
We're excited to announce our new HSM-backed cloud ACME server, the Smallstep ACME Registration Authority (RA) for Google CA Services (CAS). This product aims to make your internal PKI easier to use, more secure, and simpler to scale:
An ACME interface to Google CAS. Our ACME server makes internal automated certificate enrollment and renewal simpler, by bringing the ACME protocol (used by Let's Encrypt; RFC8555) to your internal Google CAS environment. It supports all of the challenge types that Let's Encrypt supports, so it's straightforward to add ACME support to existing services.
For internal services, ACME is a great way to automate TLS certificate enrollment and renewal, due to its broad support across languages and platforms.
Leverage Google Security & Scalability. Our ACME server connects to your Google CAS instance, which acts as your CA and issues Google-signed certificates. In this scenario, our ACME server acts narrowly as a registration authority, sitting between Google CAS and your ACME clients.
Google CAS is a highly-available, scalable private CA that is backed by Hardware Security Modules (HSM). It uses FIPS 140-2 Level 3 validated HSMs.
Easy automated enrollment and renewal. Automation with ACME clients obviates the need for manual SSL certificate renewal, while preventing outages related to certificate expiry.
If you want internal ACME support, consider running Google CAS with Smallstep's ACME RA. Together you can think of the two as an HSM-backed cloud ACME server.
We plan to add support for more backends soon: Microsoft Active Directory Certificate Services (AD CS), Hashicorp Vault, and AWS Certificate Manager Private CA (ACM PCA).
ACME is a JSON API that runs mostly over HTTPS. To get a certificate issued by an ACME server, a client must prove that it controls the requested domain name(s). It does this by responding to ACME challenges from the server.
A typical ACME challenge flow looks like this:
- The ACME client generates a Certificate Signing Request (CSR) and a private key. It contacts the ACME server and requests a certificate for the intended domain name.
- To verify that the client owns the domain name, the ACME server responds with one or more challenges. The challenges are just random values. There are three challenge types that work at increasing levels of intimacy with the service that will ultimately use the certificate:
http-01— the challenge value is placed at a well-known URL on an HTTP server at a domain named in the certificate request.
dns-01— a DNS
TXTrecord must be created that matches the challenge value, confirming that the entity requesting the certificate has control over DNS for a domain named in the certificate request.
http-01challenge type, you need routability between the ACME server and the enrolling host. With
dns-01, all you need is for everyone to share the same DNS servers, and you can use public or private DNS. The downside of
dns-01is that you have to trust DNS—which is notoriously difficult to secure.
tls-alpn-01— the challenge value is added to the initial TLS handshake (using the Application-Layer Protocol Negotiation (ALPN) TLS extension) of a server answering at a domain named in the certificate request.
http-01uses HTTP and runs on port 80. If you believe in running HTTPS everywhere (and you should), then it's annoying to have to run an HTTP server just to get a certificate. So
tls-alpn-01helps solve for this because it only uses HTTPS.
tls-alpn-01also solves for multi-tenant web servers. You could have wildcard DNS for a bunch of different web server names, and still get certificates for those servers.
Once challenges have been met for each DNS name listed on the certificate, the client can retrieve its signed certificate from the server.
Later, the client returns to the ACME server to renew its certificate using the same approach.
In these tutorials we'll configure ACME clients for use with Google CAS and the Smallstep ACME server.
These examples require some setup. You will need:
- A Google Cloud Platform account with a project you can use for this tutorial.
- A Google CAS CA instance, and a Smallstep ACME RA for CAS server that connects to your CAS CA. See our complete setup instructions.
- You may want to create a new GCP project for testing.
This example uses the ACME
dns-01 challenge type, so you will need a Google Cloud DNS zone inside your GCP project. You can use a public or private zone. Unlike Let's Encrypt, private ACME works just fine within private DNS zones.
For this tutorial, I created a Google Compute Engine VM running a kind cluster:
$ kind cluster create
I'm using kind for testing, but pretty much any Kubernetes cluster will do as long as the cluster's DNS resolver can resolve names in your Cloud DNS zone.
Let's install Kubernetes cert-manager and patch it so that it will trust your internal ACME CA. First, install cert-manager:
$ kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.0.4/cert-manager.yaml
Next, create a
ConfigMap that contains your ACME server's CA certificate. To find your certificate's PEM file, select your CA in the Google Cloud CAS Console, and view your CA certificate under the Actions menu.
Create a file called
internal-ca.yaml, replacing the certificate shown here with your own:
[REPLACE with your CA certificate]
$ kubectl apply -f internal-ca.yaml
To inject this
ConfigMap into cert-manager, we need to patch the
cert-manager Deployment to add the CA certificate as a container volume mount.
Create a file called
- name: ca-pemstore
- name: ca-pemstore
# Provide the name of the ConfigMap containing the files you want
# to add to the container
Apply the patch:
$ kubectl patch deployment cert-manager -n cert-manager --patch "$(cat cm-ca-patch.yaml)"
Let's create a Google Cloud Platform service account with the
roles/dns.admin role. Replace the
PROJECT_ID here with your own:
$ export PROJECT_ID=step-cas-test
$ gcloud iam service-accounts create dns01-solver \
--project $PROJECT_ID --display-name "dns01-solver"
$ gcloud projects add-iam-policy-binding $PROJECT_ID \
--member serviceAccount:dns01-solver@$PROJECT_ID.iam.gserviceaccount.com \
Now import the service account's credentials as a Kubernetes secret:
$ gcloud iam service-accounts keys create key.json \
$ kubectl create secret generic clouddns-dns01-solver-svc-acct \
Finally, let's create an cert-manager Issuer to perform
dns-01 ACME challenges. Make a new file called
# Your Google Cloud Platform project ID
# Your Google CloudDNS zone name we will use for DNS01 challenges
Replace the values for
hostedZoneName with your own. Your Smallstep ACME endpoint will always take the form of
https://[ACME RA hostname]/acme/acme/directory.
$ kubectl apply -f acme-issuer.yaml
You now have an automated ACME certificate manager running inside your Kubernetes cluster.
Let's get a test certificate from our ACME CA, using a Certificate object. Create a file called
dnsNames with a DNS name that's inside your zone.
$ kubectl apply -f tls-certificate.yaml
You can check the status with
kubectl get certificaterequest or
kubectl describe certificate:
$ kubectl get certificaterequest
NAME READY AGE
k8s-internal-nzbnm True 7s
$ kubectl describe certificate k8s-internal
Creation Timestamp: 2020-11-03T23:06:46Z
Secret Name: k8s-internal-tls
Last Transition Time: 2020-11-03T23:11:01Z
Message: Certificate is up to date and has not expired
Not After: 2020-11-04T23:11:01Z
Not Before: 2020-11-03T23:11:01Z
Renewal Time: 2020-11-04T15:11:01Z
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 10m cert-manager Issuing certificate as Secret does not exist
Normal Generated 10m cert-manager Stored new private key in temporary Secret resource "k8s-internal-g79jq"
Normal Requested 10m cert-manager Created new CertificateRequest resource "k8s-internal-nzbnm"
Normal Issuing 9m33s cert-manager The certificate has been successfully issued
As you can see, cert-manager will automatically renew the certificate when approximately 2/3 of its lifetime has elapsed.
That's it! You now have automated, short-lived certificates for your Kubernetes cluster. There are many use cases for X.509 certificates issued through cert-manager.
Certbot is an ACME client that can run anywhere, using a job scheduler or renewal daemon to renew certificate automatically. Let's say we have a database server
db.smallstep.internal and we want to get a TLS certificate for it using an ACME
sudo REQUESTS_CA_BUNDLE=root_ca.crt certbot certonly \
--server https://ca.smallstep.internal/acme/acme/directory \
--agree-tos --email firstname.lastname@example.org \
root_ca.crt should contain your CA certificate. The
--server URL will always take the form of
https://[ACME RA hostname]/acme/acme/directory.
certbot renew can be scheduled as a job to keep certificates fresh. The caveat here is that Certbot defaults to renewing certificates 30 days before they expire. This is great for the open web, where certificates typically have ≥ 90 day lifetimes, but for our internal infrastructure, we'll want certificates to have much shorter lifetimes.
By default, the Smallstep ACME server will issue certificates valid for 24 hours. Let's aim to renew our
db.smallstep.internal certificate about 8 hours before it expires. Add
renew_before_expiry = "8 hours" to each of the renewal configuration files in
/etc/letsencryp/renew, and set your job scheduler to run
certbot renew every 5 or 10 minutes.
Notifying Certificate-Dependent Services
Many services that depend on certificates will only read the certificate files when they start up. So when you renew a certificate, a server process that depends on it may need to be restarted or sent a
SIGHUP signal to reload its certificates. To address this, Certbot can be configured to run a hook script before or after renewal to restart servers.
You could use Certbot to deploy certificates to remote devices. For example, for an SSL virtual server on a Citrix ADC load balancer, your Certbot hook script should do the following to deploy the certificate upon renewal:
Copy the new certificate and key into
/nsconfig/ssl/on the Citrix ADC appliance
Run the following:
add ssl certKey example_key_20201004 -cert certificate.crt -key certificate.key -passcrypt random -expiryMonitor DISABLED unbind ssl vserver [virtual server name] -certkeyName example_key_20201004 bind ssl vserver [virtual server name] -certkeyName example_key_20201005
In high availability environments, you should be aware that there is a gap in service between the
For enterprise environments, ACME support is already available in a wide variety of contexts, eg:
- In a F5 BIG-IP carrier-grade internal load balancer
- For IoT devices, in ESP32 SoC microcontroller firmware
- In Java applications, using acme4j
Smallstep ACME Registration Authority (RA) brings ACME protocol support to GCP CAS, allowing you to automate certificate enrollment and renewal using ACME-compliant clients like certbot, Terraform, Caddy, and Kubernetes cert-manager. The Smallstep RA does not sign certificates itself. Instead, certificate requests are passed to GCP CAS to sign and catalog delivering a number of benefits including:
- Issued certificates are trusted by anything that trusts your GCP CAS root certificate.
- Issued certificates appear in your GCP CAS console and audit logs.
- Security-sensitive signing keys are managed by GCP CAS and never seen by Smallstep ACME RA.
Smallstep ACME RA is built and supported by Smallstep, the company behind the open-source
step-ca certificate management toolchain. It builds on the open-source
step-ca project, adding click-to-deploy integration with GCP CAS, updates, and support. Head over to the GCP marketplace and give it a try today.
Carl Tashian (Website, LinkedIn) is an engineer, writer, exec coach, and startup all-rounder. He's currently an Offroad Engineer at Smallstep. He co-founded and built the engineering team at Trove, and he wrote the code that opens your Zipcar. He lives in San Francisco with his wife Siobhan and he loves to play the modular synthesizer 🎛️🎚️