Introducing Smallstep ACME RA: Automating internal TLS with ACME + Google CAS
Carl Tashian
We're excited to announce our new HSM-backed cloud ACME server, the Smallstep ACME Registration Authority (RA) for Google CA Services (CAS). This product aims to make your internal PKI easier to use, more secure, and simpler to scale:
-
An ACME interface to Google CAS. Our ACME server makes internal automated certificate enrollment and renewal simpler, by bringing the ACME protocol (used by Let's Encrypt; RFC8555) to your internal Google CAS environment. It supports all of the challenge types that Let's Encrypt supports, so it's straightforward to add ACME support to existing services.
For internal services, ACME is a great way to automate TLS certificate enrollment and renewal, due to its broad support across languages and platforms.
-
Leverage Google Security & Scalability. Our ACME server connects to your Google CAS instance, which acts as your CA and issues Google-signed certificates. In this scenario, our ACME server acts narrowly as a registration authority, sitting between Google CAS and your ACME clients.
Google CAS is a highly-available, scalable private CA that is backed by Hardware Security Modules (HSM). It uses FIPS 140-2 Level 3 validated HSMs.
-
Easy automated enrollment and renewal. Automation with ACME clients obviates the need for manual SSL certificate renewal, while preventing outages related to certificate expiry.
If you want internal ACME support, consider running Google CAS with Smallstep's ACME RA. Together you can think of the two as an HSM-backed cloud ACME server.
Our click-to-deploy image is available in the Google cloud marketplace.
We plan to add support for more backends soon: Microsoft Active Directory Certificate Services (AD CS), Hashicorp Vault, and AWS Certificate Manager Private CA (ACM PCA).
How ACME works
ACME is a JSON API that runs mostly over HTTPS. To get a certificate issued by an ACME server, a client must prove that it controls the requested domain name(s). It does this by responding to ACME challenges from the server.
A typical ACME challenge flow looks like this:
- The ACME client generates a Certificate Signing Request (CSR) and a private key. It contacts the ACME server and requests a certificate for the intended domain name.
- To verify that the client owns the domain name, the ACME server responds with one or more challenges. The challenges are just random values. There are three challenge types that work at increasing levels of intimacy with the service that will ultimately use the certificate:
-
http-01— the challenge value is placed at a well-known URL on an HTTP server at a domain named in the certificate request. -
dns-01— a DNSTXTrecord must be created that matches the challenge value, confirming that the entity requesting the certificate has control over DNS for a domain named in the certificate request.With
http-01challenge type, you need routability between the ACME server and the enrolling host. Withdns-01, all you need is for everyone to share the same DNS servers, and you can use public or private DNS. The downside ofdns-01is that you have to trust DNS—which is notoriously difficult to secure. -
tls-alpn-01— the challenge value is added to the initial TLS handshake (using the Application-Layer Protocol Negotiation (ALPN) TLS extension) of a server answering at a domain named in the certificate request.http-01uses HTTP and runs on port 80. If you believe in running HTTPS everywhere (and you should), then it's annoying to have to run an HTTP server just to get a certificate. Sotls-alpn-01helps solve for this because it only uses HTTPS.tls-alpn-01also solves for multi-tenant web servers. You could have wildcard DNS for a bunch of different web server names, and still get certificates for those servers.
-
Once challenges have been met for each DNS name listed on the certificate, the client can retrieve its signed certificate from the server.
Later, the client returns to the ACME server to renew its certificate using the same approach.
Let's try it out
In these tutorials we'll configure ACME clients for use with Google CAS and the Smallstep ACME server.
Prerequisites
These examples require some setup. You will need:
- A Google Cloud Platform account with a project you can use for this tutorial.
- A Google CAS CA instance, and a Smallstep ACME RA for CAS server that connects to your CAS CA. See our complete setup instructions.
- You may want to create a new GCP project for testing.
ACME + Kubernetes cert-manager
In this example, we'll configure Kubernetes cert-manager to get a certificate from an internal ACME server, using cert-manager's ACME issuer.
Before you begin
This example uses the ACME dns-01 challenge type, so you will need a Google Cloud DNS zone inside your GCP project. You can use a public or private zone. Unlike Let's Encrypt, private ACME works just fine within private DNS zones.
1. Create a cluster
For this tutorial, I created a Google Compute Engine VM running a kind cluster:
I'm using kind for testing, but pretty much any Kubernetes cluster will do as long as the cluster's DNS resolver can resolve names in your Cloud DNS zone.
2. Set up cert-manager to trust your internal CA
Let's install Kubernetes cert-manager and patch it so that it will trust your internal ACME CA. First, install cert-manager:
Next, create a ConfigMap that contains your ACME server's CA certificate. To find your certificate's PEM file, select your CA in the Google Cloud CAS Console, and view your CA certificate under the Actions menu.
Create a file called internal-ca.yaml, replacing the certificate shown here with your own:
apiVersion: v1 data: internal-ca.pem: | -----BEGIN CERTIFICATE----- [REPLACE with your CA certificate] -----END CERTIFICATE----- kind: ConfigMap metadata: name: ca-pemstore namespace: cert-manager resourceVersion: "9978" selfLink: /api/v1/namespaces/cert-manager/configmaps/ca-pemstore -
Apply it:
$ kubectl apply -f internal-ca.yaml
To inject this ConfigMap into cert-manager, we need to patch the cert-manager Deployment to add the CA certificate as a container volume mount.
Create a file called cm-ca-patch.yaml:
spec: template: spec: containers: - args: - --v=2 - --cluster-resource-namespace=$(POD_NAMESPACE) - --leader-election-namespace=kube-system name: cert-manager volumeMounts: - name: ca-pemstore mountPath: /etc/ssl/certs/internal-ca.pem subPath: internal-ca.pem readOnly: false resources: {} volumes: - name: ca-pemstore configMap: # Provide the name of the ConfigMap containing the files you want # to add to the container name: ca-pemstore
Apply the patch:
3. Create a GCP service account and import its credentials
Let's create a Google Cloud Platform service account with the roles/dns.admin role. Replace the PROJECT_ID here with your own:
Now import the service account's credentials as a Kubernetes secret:
Finally, let's create an cert-manager Issuer to perform dns-01 ACME challenges. Make a new file called acme-issuer.yaml:
apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: acme-issuer spec: acme: <strong>email: carl@smallstep.com</strong> <strong>server: https://ca.smallstep.internal/acme/acme/directory</strong> privateKeySecretRef: name: acme-issuer-account-key solvers: - dns01: cloudDNS: # Your Google Cloud Platform project ID <strong>project: step-cas-test</strong> # Your Google CloudDNS zone name we will use for DNS01 challenges <strong>hostedZoneName: step-cas-internal</strong> serviceAccountSecretRef: name: clouddns-dns01-solver-svc-acct key: key.json
Replace the values for email, server URL, project and hostedZoneName with your own. Your Smallstep ACME endpoint will always take the form of https://[ACME RA hostname]/acme/acme/directory.
Apply it:
$ kubectl apply -f acme-issuer.yaml
You now have an automated ACME certificate manager running inside your Kubernetes cluster.
4. Issue a test certificate
Let's get a test certificate from our ACME CA, using a Certificate object. Create a file called tls-certificate.yaml:
apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: k8s-internal namespace: default spec: secretName: k8s-internal-tls issuerRef: name: acme-issuer dnsNames: - k8s.smallstep.internal
Replace the dnsNames with a DNS name that's inside your zone.
Apply it:
$ kubectl apply -f tls-certificate.yaml
You can check the status with kubectl get certificaterequest or kubectl describe certificate:
As you can see, cert-manager will automatically renew the certificate when approximately 2/3 of its lifetime has elapsed.
That's it! You now have automated, short-lived certificates for your Kubernetes cluster. There are many use cases for X.509 certificates issued through cert-manager.
ACME + Certbot
Certbot is an ACME client that can run anywhere, using a job scheduler or renewal daemon to renew certificate automatically. Let's say we have a database server db.smallstep.internal and we want to get a TLS certificate for it using an ACME http-01 challenge:
The file root_ca.crt should contain your CA certificate. The --server URL will always take the form of https://[ACME RA hostname]/acme/acme/directory.
After enrollment, certbot renew can be scheduled as a job to keep certificates fresh. The caveat here is that Certbot defaults to renewing certificates 30 days before they expire. This is great for the open web, where certificates typically have ≥ 90 day lifetimes, but for our internal infrastructure, we'll want certificates to have much shorter lifetimes.
By default, the Smallstep ACME server will issue certificates valid for 24 hours. Let's aim to renew our db.smallstep.internal certificate about 8 hours before it expires. Add renew_before_expiry = "8 hours" to each of the renewal configuration files in /etc/letsencryp/renew, and set your job scheduler to run certbot renew every 5 or 10 minutes.
Notifying Certificate-Dependent Services
Many services that depend on certificates will only read the certificate files when they start up. So when you renew a certificate, a server process that depends on it may need to be restarted or sent a SIGHUP signal to reload its certificates. To address this, Certbot can be configured to run a hook script before or after renewal to restart servers.
ACME + Certbot + Citrix ADC
You could use Certbot to deploy certificates to remote devices. For example, for an SSL virtual server on a Citrix ADC load balancer, your Certbot hook script should do the following to deploy the certificate upon renewal:
-
Copy the new certificate and key into
/nsconfig/ssl/on the Citrix ADC appliance -
Run the following:
add ssl certKey example_key_20201004 -cert certificate.crt -key certificate.key -passcrypt random -expiryMonitor DISABLED unbind ssl vserver [virtual server name] -certkeyName example_key_20201004 bind ssl vserver [virtual server name] -certkeyName example_key_20201005
In high availability environments, you should be aware that there is a gap in service between the unbind and bind commands.
What else runs ACME?
For enterprise environments, ACME support is already available in a wide variety of contexts, eg:
- In a F5 BIG-IP carrier-grade internal load balancer
- For IoT devices, in ESP32 SoC microcontroller firmware
- In Java applications, using acme4j
Getting started with Google CAS and ACME
Smallstep ACME Registration Authority (RA) brings ACME protocol support to GCP CAS, allowing you to automate certificate enrollment and renewal using ACME-compliant clients like certbot, Terraform, Caddy, and Kubernetes cert-manager. The Smallstep RA does not sign certificates itself. Instead, certificate requests are passed to GCP CAS to sign and catalog delivering a number of benefits including:
- Issued certificates are trusted by anything that trusts your GCP CAS root certificate.
- Issued certificates appear in your GCP CAS console and audit logs.
- Security-sensitive signing keys are managed by GCP CAS and never seen by Smallstep ACME RA.
Smallstep ACME RA is built and supported by Smallstep, the company behind the open-source step-ca certificate management toolchain. It builds on the open-source step-ca project, adding click-to-deploy integration with GCP CAS, updates, and support. Head over to the GCP marketplace and give it a try today.
Subscribe to updates
Unsubscribe anytime, see Privacy Policy
Carl Tashian (Website, LinkedIn) is an engineer, writer, exec coach, and startup all-rounder. He's currently an Offroad Engineer at Smallstep. He co-founded and built the engineering team at Trove, and he wrote the code that opens your Zipcar. He lives in San Francisco with his wife Siobhan and he loves to play the modular synthesizer 🎛️🎚️
