Smallstep Certificate Manager How It Works
Certificates power modern software. Kubernetes, service meshes, containers, microservices, distributed databases & queues, configuration management & orchestration systems, BeyondCorp proxies, and secure software supply chains all use certificates. Organizations running modern infrastructure make extensive use of certificates, whether they're aware of it or not.
For many organizations, this is a blind spot, with long-lived certificates issued manually or from poorly secured deploy pipelines. Secure, reliable certificate management requires automation. Smallstep Certificate Manager is a flexible, extensible toolchain with all of the server and client-side components you'll need to solve this problem, carefully designed to be operationally simple, easy to use, and hard to misuse.
Certificate Manager builds on two open source projects, maintained by smallstep:
step-ca
: a private online certificate authority for secure automated certificate management.step
: a general-purpose cryptography toolkit and the client-side counterpart tostep-ca
.
Core Concepts & Components
Certificates, CAs, Subjects, Endpoints, and Relying Parties
A certificate is a sort of credential. Concretely, a certificate is a data structure that contains a name, public key, validity period (a.k.a., lifetime), and additional metadata. Certificates are signed by a trusted certificate authority (CA) and issued to endpoints. A relying party validates a certificate and extracts the subject name to authenticate the identity of an endpoint.
Endpoints
An Endpoint is an entity (person, device, workload) that is issued a certificate. Certificate Manager attempts to group certificate renewals for an Endpoint for alerting, audit, and billing purposes.
Certificate Manager can be configured to alert you when an Endpoint’s last certificate is approaching expiry and has not been renewed.
The Certificate Manager dashboard provides an Endpoint detail view that makes it easy for administrators to view the historical certificate lineage for an Endpoint. This aggregated view simplifies operations, auditing, reporting, and compliance inquiries.
Certificate Manager billing is also metered on Endpoint-months.
- Billing starts when a certificate is issued for a new Endpoint
- Billing ends when an Endpoint’s last certificate expires or is revoked
Billing per-certificate would penalize deployments that use short-lived certificates and automated renewal. Endpoint billing is designed to encourage this best practice.
Two Endpoint examples:
- A single device with one 30-day certificate would be billed at the same rate as,
- A single device with 60 one-day certificates renewed every 12-hours.
Endpoint grouping is automatic and intuitive for most use cases:
- For provisioners with renewal enabled:
- Certificates issued using
step ca certificate
(or any other method that uses the/sign
API) create a new Endpoint - Certificates issued using
step ca renew
(or any other method that uses the/renew
API) are associated with the existing Endpoint of the certificate that’s being renewed
- Certificates issued using
- For provisioners with renewal disabled, common with ACME and OIDC, certificates with identical subjects (common name and SANs), ignoring order and capitalization, belong to the same Endpoint
For billing purposes, there is a limit of three active certificates per Endpoint. Each active certificate above three is billed as an additional Endpoint. To avoid being charged for multiple Endpoints you can revoke unused certificates after they’ve been renewed.
If you’d like to opt-in to certificate-based billing, inquire about high-volume fixed pricing, or have any other questions about Endpoint-based billing for your use case please contact us.
Authorities
Certificate Manager authorities are trusted services that manage certificate issuance and revocation. Several authority types are supported. Authorities are highly configurable, and multiple authorities can be used together to meet complex requirements.
Issuing Authorities
An issuing authority is an online certificate authority that authenticates and authorizes certificate requests and issues certificates. Issuing authorities sign certificates themselves. To do so, they are provisioned with a special CA certificate and private key.
There are two types of issuing authority:
- Hosted: run by smallstep on your behalf as part of your Certificate Manager account.
- Linked: an instance of
step-ca
you run that connects to your Certificate Manager account for reporting, alerting, revocation, and other managed services.
If an issuing authority's private key is compromised, it can be used to
maliciously issue certificates that will be trusted by the rest of your
infrastructure. To protect these keys, step-ca
integrates with hardware and
software key managers, including PKCS#11 Hardware Security Modules (HSMs),
YubiKeys, and cloud key management systems (KMSs) from AWS, GCP, and Azure. For
hosted authorities, smallstep secures these keys for you in GCP's CloudKMS, with
options for software or FIPS 140-2 Level 3 hardware protection levels.
Root vs. Intermediate Certificate Authorities
A root CA is directly trusted by relying parties. The root CA's certificate must be deployed to VMs, devices, and containers, then software and systems must be configured to trust it. We'll see how this is done later when we discuss bootstrapping trust.
An intermediate CA (also called a subordinate CA) is issued a special certificate that authorizes it to sign other certificates. Endpoint certificates can be signed directly by the root or, more typically, by an intermediate CA that recursively chains up to the root CA. An intermediate CA allows you to keep your root signing keys offline, which improves security.
For most scenarios, this is a nuance you won't have to worry about. By default, Certificate Manager will create a non-issuing root CA and a separate issuing authority for you automatically. For advanced scenarios, you can use your own offline root CA, provision multiple issuing authorities from a single root, configure multiple levels of intermediates, and revoke and re-issue intermediate authority certificates as necessary.
Registration Authorities
Like an issuing authority, a registration authority is an online service that accepts and authenticates certificate requests. But, instead of issuing certificates itself, a registration authority passes authentic requests to an issuing authority to sign. Registration authorities support all Provisioner types and are an optional component. They deliver most of the benefits of a linked issuing authority with less operational complexity since there's no signing key to manage.
Validation Authorities
A validation authority distributes certificate revocation status. Validation authorities implement two open standards to support active revocation:
- Certificate Revocation List (CRL): a signed, immutable ledger that lists the serial number of all revoked certificates. CRLs are signed by trusted infrastructure and served from cloud storage for high performance and availability.
- Online Certificate Status Protocol (OCSP): a standard API for requesting the revocation status of a particular certificate. The Certificate Manager OCSP responder is a shared-nothing service that uses CRLs as a data source.
Provisioners
Provisioners are used by issuing authorities to authenticate certificate requests. Provisioners make it easy to automate certificate management where possible, and support semi-automated / self-serve workflows where required.
Certificate lifetimes, access control policies, renewal, templates, and many other options are configurable per-provisioner. Since an issuing authority can have multiple provisioners, you implement complex authentication and authorization policies and issue different kinds of certificates from one issuing authority.
Some of the more popular workflows provisioners enable include:
- Passwords: in its simplest form, the
JWK
provisioner can be used to get a certificate using a password. We'll see an example of this below. - One-time tokens: the
JWK
provisioner also supports one-time tokens usingstep ca token
, which can be generated by orchestration or configuration management and passed to a container or host to obtain a certificate. - ACME: the
ACME
provisioner implements the ACME standard created by Let's Encrypt. It can be used to automatically get a certificate for a domain name or IP address. A rich client ecosystem and built-in support in many tools makes ACME easy to integrate. See the ACME documentation to learn more. - Single Sign-on: The
OIDC
provisioner uses OAuth and OpenID Connect (OIDC) to get a certificate using single sign-on via Google, Okta, Azure AD, or any other compatible identity provider. See the OIDC documentation to learn more. There are two common scenarios where this is useful:- Authenticating users (engineers, operators, etc.) who need a certificate for code signing or to authenticate to databases, services, or other infrastructure using mutual TLS.
- Self-serve / semi-automated workflows for administrators to obtain certificates for workloads, devices, and other infrastructure, where automation is not possible.
- Cloud VMs: the
IID
provisioner can be used to get certificates to your VMs running on AWS, GCP, or Azure. - Kubernetes:
- cert-manager is a popular ACME
client for Kubernetes. It creates
certificate secret resources
that can be used by containers and ingresses.
step-issuer
lets you connectcert-manager
to an issuing authority. - autocert lets you use pod annotations to automatically inject certificates into containers using volume mounts.
- The
k8ssa
provisioner lets you get certificates using Kubernetes service accounts.
- cert-manager is a popular ACME
client for Kubernetes. It creates
certificate secret resources
that can be used by containers and ingresses.
- Devices: The
X5C
andSSHPOP
provisioners let you get a certificate using an existing x509 or SSH certificate issued from another authority. This can be used by devices to exchange long-lived birth certificates issued at manufacture time for short-lived workload certificates and for other derived credential workflows where a certificate from a canonical CA is used to automatically obtain certificates from one or more special-purpose CA(s).
Certificate Lifecycle Management
Bootstrapping Trust
Before issuing a certificate, your software and systems must be configured to
trust your authorities. More precisely, you need to configure everything to
trust your root certificate (i.e., the certificate that belongs to your root
CA). You can include your root certificate in base images, distribute it using
configuration management, or use
step ca bootstrap
or step ca root
to
securely download your root certificate from an issuing authority. You can also
download the root certificate for your authorities from the Certificate Manager
dashboard.
step ca bootstrap
configuresstep
to trust your root CA and use a particular issuing authority.step ca root
securely downloads your root certificate.
Once you've downloaded your root certificate, you can add it to your system's default
trust store using
step certificate install
:
$ step ca root root.crt
The root certificate has been saved in root.crt.
$ step certificate install root.crt
Password:
Certificate r.crt has been installed.
X.509v3 Root CA Certificate (ECDSA P-256) [Serial: 1829...9147]
Subject: Root CA
Issuer: Root CA
Valid from: 2020-09-18T20:55:03Z
to: 2030-09-16T20:55:03Z
Or configure software to trust it (see Hello mTLS for more examples):
$ cat nginx.conf
server {
listen 443 ssl;
server_name myserver.internal.net;
ssl_client_certificate /etc/nginx/client_certs/root.crt;
ssl_verify_client optional;
location / {
if ($ssl_client_verify != SUCCESS) {
return 403;
}
}
}
Issuing Certificates
Issuing a certificate using step
is simple. In it's most basic form, the
step ca certificate
command provides simple interactive workflows for people to get a certificate
from an issuing authority. Flags and options make scripting and other
non-interactive workflows easy. We'll use several variations of this command in
the examples below.
$ step ca certificate example.com example.crt example.key
Use the arrow keys to navigate: ↓ ↑ → ←
What provisioner key do you want to use?
▸ mike@example.com (JWK) [kid: -6v0dwnSlz2D5opq-B5prQmyD9DhzdJSHMidiu1TjYg]
sshpop (SSHPOP)
x5c (X5C)
Google (OIDC) [client: 650445034027-jsjdrkiskeq9ke99ud2rqkst82ft8uch.apps.googleusercontent.com]
✔ Provisioner: mike@smallstep.com (JWK) [kid: -6v0dwnSlz2D5opq-B5prQmyD9DhzdJSHMidiu1TjYg]
✔ Please enter the password to decrypt the provisioner key:
✔ CA: https://ca.example.com
✔ Certificate: example.crt
✔ Private Key: example.key
Templates
Templates give you granular control over certificate details. By default, Certificate Manager is tuned to issue short-lived certificates for use with TLS. Templates let you customize every detail of a certificate, down to the OID, to support any use case.
Concretely, a template is a JSON representation of a certificate that's
materialized using Go's text/template
module and sprig functions. They look
like this:
Context from certificate requests and authentication credentials are made available as template variables, so you can adjust certificate details based on who's requesting the certificate.
Inventories
Inventories are lists of hosts, services, people, or any other endpoint. An inventory record lets you associate metadata with an endpoint. Inventories work together with templates. An inventory lookup can be triggered by a certificate request. Metadata is exposed as template variables, where it can be used to customize certificates or authorize certificate requests.
For example, you can use inventories to map the email address from an OIDC identity token to a username, authorize a host to get certificates for its workloads, or map an authenticated device ID from a TPM attestation certificate to a DNS name.
Renewal
Certificates expire. Certificate Manager makes renewing a certificate ahead of expiration easy. Renewals are authenticated using your existing certificate, and produce an identical certificate with a new serial number and extended lifetime.
In its most primitive form, renewal is a simple single-command operation:
step ca renew svc.crt svc.key
More than a dozen command-line flags make
step ca renew
flexible and easy to integrate into almost any operational environment. For a
taste of what's possible, consider:
step ca renew --daemon --exec "nginx -s reload" svc.crt svc.key
The --daemon
flag starts a long-running process that will continuously renew
svc.crt
after 2/3 of its lifetime has elapsed. The --exec "nginx -s reload"
option tells Nginx to reload svc.crt
after each renewal. We even take care of
retries if renewal fail (with exponential backoff and jitter to mitigate denial
of service issues and thundering herds).
step ca renew
is designed to integrate nicely with cron
or systemd
, which
is what we recommend for production deploys. For more details see our
documentation on production-ready automated renewal.
It's also common to trigger renewal from configuration management or from a
container sidecar. Regardless, step
makes renewal simple and secure.
Disabling Renewal
Finally, sometimes you don't want certificates to be renewed. For example, if
you're issuing certificates using single sign-on (the OIDC
provisioner), you
probably want your users to re-authenticate with your identity provider
periodically to get a new certificate instead of renewing. Renewal can be
enabled and disabled per-provisioner to accommodate this (it's disabled by
default for the OIDC
provisioner).
Renewal After Expiry
- This is an Advanced Authority feature
Some use cases demand an option to renew expired certificates. For example, intermittently-connected devices may not be able to reach the CA in time for a renewal. The ability to renew expired certificates can be enabled on a per-provisioner basis. Enable it in the Smallstep dashboard when you create a new provisioner, or on the command line for an existing provisioner. While sometimes necessary, this feature comes with risks and is disabled by default.
Revocation
Certificates should be revoked if they're compromised or no longer needed. Smallstep Certificate Manager supports both passive and active revocation.
- Passive revocation disables renewal for a particular certificate. The certificate will be trusted until it expires.
- Active revocation explicitly invalidates a certificate ahead of expiry. CRL and OCSP are used to distribute revocation status to relying parties. Active revocation is often necessary for incident response. However, caching, availability, and configuration challenges can make active revocation unreliable. For systems that support it, we've found that revoking authorization can be easier to implement and more reliable than active certificate revocation. We've covered this topic in more detail on our blog.
Using Passive Revocation
Passive revocation is on by default and works well in conjunction with short-lived certificates.
To revoke a certificate, run step ca revoke
and pass the certificate's serial number.
$ step ca revoke 31952909121245729293840904958100006722
Once revoked, the certificate is no longer renewable.
Using Active Revocation
Active revocation (CRL and OCSP) is available if you're using an Advanced Authority. It can be enabled when you create the authority. To revoke a certificate, you will need either the certificate and its private key (for mutual TLS), or a JWK provisioner (for JWT authentication). You cannot revoke a certificate using the default administrative OIDC provisioner.
To revoke a certificate using mutual TLS, run:
$ step ca revoke --cert example.crt --key example.key
With a JWK provisioner, to revoke a certificate using JWT authentication, run:
$ SERIAL=31952909121245729293840904958100006722
$ TOKEN=$(step ca token --provisioner jwk-provisioner --revoke $SERIAL)
$ step ca revoke --token $TOKEN $SERIAL
Observability & Alerting
Certificate Manager is designed to support large, rapidly evolving infrastructures. In these environments, it's easy for a small, unnoticed failure to cascade into a large outage.
Certificate Manager can alert you ahead of certificate expiry via email or by calling a webhook. Alerts can be fed into your SIEM or sent to slack so you can remediate ahead of an outage. Issued certificates are catalogued for easy auditability. The Certificate Manager interface allows you to sort and filter, and quickly list and export certificate subjects, issue & expiry dates, issuing authority, provisioner, and other certificate details.
Smallstep API
The Smallstep API can be used to manage your authorities programmatically. The API is currently an alpha release with limited availability. Request access to this API for your account by creating a support ticket for our Customer Engineering team. Be sure to include your team name in your request.
Certificate Manager in Action
Let's look at how these features come together in a few common workflows.
To configure step
to use our authority, run:
step ca bootstrap \
--ca-url https://prod.yourco.ca.smallstep.com
--fingerprint 6d04ff00618125114b97870f835c4a28fce2fec91e4b86946147f5c6a5555ff8
Simple Password-based Issuance
To get a certificate, run step ca certificate
. You'll be prompted to select a
provisioner. In this example, we'll use the JWK
provisioner in a
password-based flow:
$ step ca certificate admin@yourco.com --san=any-name any.crt any.key
✔ Provisioner: pass (JWK) [kid: x6v0dwnSlz2D5opq-B5prQmyD9DhzdJSHMidiu1TjYg]
✔ Please enter the password to decrypt the provisioner key:
✔ CA: https://prod.yourco.ca.smallstep.com
✔ Certificate: any.crt
✔ Private Key: any.key
Once authenticated, the authority will obtain inventory metadata and apply any templates associated with the selected provisioner. That's it. The issued certificate is ready to use.
$ step certificate inspect any.crt -short
X.509v3 TLS Certificate (ECDSA P-256) [Serial: 3329...8201]
Subject: any-name
admin@yourco.com
Issuer: Prod Intermediate CA
Provisioner: pass [ID: TvGU...0zXw]
Valid from: 2021-07-07T22:59:05Z
to: 2021-07-08T23:00:05Z
Renewing this certificate is equally straightforward:
$ step ca renew any.crt any.key --force
Your certificate has been saved in any.crt.
If you inspect the newly issued certificate, you'll find it has an extended lifetime.
One-time Tokens
The JWK
provisioner is very flexible. Under the hood, it's based on
short-lived one-time tokens that are signed by a private key. The issuing
authority escrows an encrypted copy of the private key. In the password flow
above, step
obtained the encrypted private key from the issuing authority and
prompted for the password to decrypt it. It generated a one-time token, then
immediately used it to get a certificate.
Instead of broadly distributing the JWK
provisioner password, we can generate
a token in one place (e.g., CI/CD, configuration management, or orchestration):
$ step ca token alice@yourco.com
Then pass the token along for a host, container, or service to obtain a certificate:
$ step ca certificate alice@yourco.com alice.crt alice.key --token $TOKEN
Single Sign-On
Certificates provide a secure, flexible, scalable mechanism for authenticating
people, too (e.g., for SSH access, API access, or to connect to a BeyondCorp
identity-aware proxy). Most organizations already have an identity provider
(IdP) for authenticating people. The OIDC
provisioner lets you leverage
authentication services from G Suite, Okta, Azure AD, and any other IdP that
supports OAuth OIDC to authenticate a certificate request.
$ step certificate inspect ee.crt -short
X.509v3 TLS Certificate (ECDSA P-256) [Serial: 2676...4589]
Subject: developer@yourco.com
05dc4b5d-769a-4659-92ee-9cf5d3b720b3
https://auth.okta.com#05dc4b5d-769a-4659-92ee-9cf5d3b720b3
Issuer: Prod Intermediate CA
Provisioner: okta [ID: 5310...fc2a]
Valid from: 2021-06-30T22:52:09Z
to: 2021-07-01T14:53:09Z
ACME
The ACME
provisioner implements the
ACME standard, created and used
by Let's Encrypt to secure over 260 million public websites. ACME lets you
automatically get a certificate for a domain name or IP address. ACME's rich
client ecosystem makes it especially easy to integrate.
ACME is a simple challenge-response protocol. A client orders a certificate and the authority responds with a set of challenges for the client to complete to prove control over the identifier(s) in the order. For example, if a client orders a certificate for a DNS identifier, the authority may challenge the client to serve a random number over HTTP from a random path to prove control over the domain name.
You can use any standards-compliant ACME client to get a certificate from your Certificate Manager authorities. Here's an example workflow using Certbot, an open source ACME client maintained by the EFF:
sudo REQUESTS_CA_BUNDLE=$(step path)/certs/root_ca.crt \
certbot certonly -n --standalone -d foo.internal \
--server https://ca.internal/acme/acme/directory
ACME works almost anywhere and is especially useful for issuing certificates to web servers for use with TLS. To issue certificates for internal names that aren't in public DNS the ACME server does need access to local DNS. It's common to use a linked CA or a network-local RA. Reference the Certificate Manager docs for more details.
Next steps
To learn more:
- Get hands-on - Mint your free hosted Authority in less than five minutes.
- Get advice- Talk to Smallstep about your project.