Production considerations when running a certificate authority server

step-ca is built for robust certificate management in distributed systems. As with any entity in your infrastructure, running step-ca effectively in production requires some knowledge of its strengths and limitations. This document addresses the important production considerations that operators should know about when running step-ca as a certificate authority server.

Overview

Good Security Practices

In this section we recommend a few best practices when it comes to running, deploying, and managing your own online certificate authority server and PKI. Security is a moving target. We expect our recommendations to change and evolve as well.

Safeguard Your Root And Intermediate CA Keys

When you initialize a two-tier CA, two private keys are generated: one intermediate private key, and one root private key. It is very important that these private keys be kept secret.

The intermediate key is used by the CA to sign certificates. The root key is not needed for day-to-day CA operation and should be stored offline. The keys can be generated on an air-gapped device or on a Hardware Security Module (HSM).

Here's an example key protection strategy for a high-security production PKI.

In this example, step-ca acts as a subordinate CA to an offline root CA.

See the Cryptographic Protection section of our Configuration Guide to learn more about your options for using HSMs or cloud KMS with step-ca.

Use Strong Passwords and Store Them Well

When you initialize your PKI, the root and intermediate private keys will be encrypted with the same password.

Use a password manager to generate random passwords, or let step ca init generate a strong password for you.

After initializing your CA, we recommend that you immediately change the password for the intermediate CA private key:

You'll use this new intermediate key password to start step-ca.

Once you've changed the intermediate private key password, you should never have to use the root private key password again. So, then what should you do with it?

Bury it in a cave high in the mountains.

Or, store it in a password manager or secrets manager. There are many to choose from and the choice will depend on the risk & security profile of your organization.

In addition to using a password manager to store all passwords (private key, provisioner password, etc.) we recommend using a threshold cryptography algorithm like Shamir's Secret Sharing to divide the root private key password across a handful of trusted parties.

Avoid Storing Passwords in Environment Variables

systemd discourages using the environment for secrets because it doesn't consider it secure and exposes a unit's environment over dbus. From systemd.exec(5):

Note that environment variables are not suitable for passing secrets (such as passwords, key material, …) to service processes. Environment variables set for a unit are exposed to unprivileged clients via D-Bus IPC, and generally not understood as being data that requires protection. Moreover, environment variables are propagated down the process tree, including across security boundaries (such as setuid/setgid executables), and hence might leak to processes that should not have access to the secret data.

For some isolated environments, we could see an argument for the convenience of an environment variable. Even then, there can be subtle issues. For example, anyone with access to the Docker daemon can view all of the environment variables of running Docker containers, using docker inspect.

For posterity, however, if you've secured your environment and rely on it for secrets, there is a way to pass a password into step-ca from an environment variable in Bash:

This method is known as Bash Process Subsitution, and on most systems the password will not appear in ps output. However, this approach is not recommended simply because it's so difficult to ensure security with environment variables.

Delete Your Default Provisioner

When you initialize your PKI, a default JWK provisioner will be created and it's private key will be encrypted using the same password used to encrypt the root CA private key. Before deploying step-ca you should remove this provisioner and add new provisioners that are encrypted with secure, random passwords. See the section on managing provisioners.

Use Short-Lived Certificates

We recommend the following:

Certificates from step-ca expire in 24 hours by default. We made it easy for you to automate the renewal of your certificates using the step command. Carpe diem!

You can configure certificate lifetimes in the ca.json file.

Enable Active Revocation On Your Intermediate CA

The value of a two-tiered PKI is that you can add your root CA certificate to the certificate trust store on all of your nodes, and store your root private key completely offline. A leaf certificate signed by the CA always comes in a bundle that contains the intermediate CA certificate alongside the leaf certificate. With this bundle, any client that trusts your root CA can verify the complete chain of trust.

Now, what if one day your intermediate CA key is compromised? You could issue a new intermediate using your root CA key, but your old intermediate has a 10 year validity period! So, you're stuck having to rotate your root CA too, and that's a much bigger project because you have to distribute the new root certificate to everyone and ensure the old one is no longer trusted. To avoid this scenario, you can use active revocation on your intermediate CA certificate, making it possible to immediately revoke a compromised intermediate.

While step-ca doesn't directly support active revocation mechanisms like Certificate Revocation Lists (CRLs) or the Online Certificate Status Portocol (OCSP), you can independently manage your own CRL if you like.

Create an intermediate CA with a CRL distribution endpoint

Let's make it possible to revoke your intermediate CA down the road if necessary. This setup is more complex than the default step-ca PKI, but it offers an insurance policy for a compromised intermediate CA.

Revoke A Certificate

To revoke a certificate, add it to the index.txt file before regenerating the CRL file. The format for this CRL database file is:

Source: OpenSSL PKI Tutorial

Use Templates With Care

Because certificate templates add a lot of flexibility to step-ca, they can be a source of subtle vulnerabilities in your PKI. If you use custom certificate templates, be sure they are tightly restricted for your use case. Use extreme caution when referencing user-supplied data marked as Insecure in templates.

Create a Service User to Run step-ca

Make sure that the configuration folders, private keys, and password file used by the CA are only accessible by this user. If you're running step-ca on port 443, you'll need the step-ca binary to be able to bind to that port. See Running step-ca as a Daemon for details.

Running step-ca as a Daemon

Note: This section requires a Linux OS running systemd version 245 or greater.

High Availability

A few things to consider / implement when running multiple instances of step-ca:

Load balancing or proxying step-ca traffic

If you need to place a load balancer or reverse proxy downstream from the CA, we recommend using layer 4 (TCP) load balancing or proxying (aka "TLS passthrough").

We also recommend enabling remote provisioner management so that your provisioner configuration can be shared by all step-ca instances.

Layer 7 proxying is not recommended, becase the step toolchain is built around TLS:

Further Reading

Automate X.509 Certificate Lifecycle Management

By default, step-ca issues short-lived certificates that expire after 24 hours. Short-lived certificates are excellent security hygiene because they offer regular key rotation and passive revocation

Short-lived certificates create a problem though: For any long-lived workloads, you will need to renew your certificates each day before they expire. This section will show three approaches to renew certs with step.

Creating short-lived certificates

First, let's initialize a new PKI and start step-ca. We'll write a password out to password.txt so we don't have to enter it repeatedly.

Now let's generate a single-use bootstrap token and use it to obtain a certificate:

In a production environment, you might use something like Kubernetes or Chef to generate this token and give it to a host or client that needs a certificate.

Now we can generate a keypair locally, and use our bootstrap token to obtain a certificate for foo.local from step-ca:

By default, step-ca issues certificates valid for 24 hours. This is suitably short for many scenarios. If it's not right for you, you can adjust the defaultTLSCertDuration per provisioner or pass the --not-after flag to the step ca certificate to adjust the lifetime of an individual certificate. Very short lifetimes (eg. five minutes) are better from a security perspective, but this can be difficult in practice.

With short-lived certificates, your services and hosts will need to renew their certificates regularly, by extending their lifetimes before they expire.

You can do this manually with the following command:

Note the change in the validity period relative to the original certificate above.

Automated renewal

What good are short-lived certificates if we can't renew them automatically?

Here are three options for setting up automated renewal of certificates using step ca renew:

Renewal using systemd timers

This approach runs a periodic systemd timer for each certificate you want to keep current. The timer will run a one-shot systemd service every few minutes. The one-shot service checks the certificate and renews it if more than ⅔ of its lifetime has elapsed. Upon renewal, the service can try to reload or restart a service using the certificate files, if it exists. Custom post-renewal commands can be configured as well.

We will leverage systemd service templates to simplify configuration of certificate renewal for many target services.

In /etc/systemd/system, we'll start by creating template files cert-renewer@.service, and cert-renewer@.timer.

Service templates accept a single argument after the @, called the service unit argument. For example, for cert-renewer@postgresql.service, the service unit argument is postgresql. In the template, %i represents the service unit argument.

Let's review the service template first, then we'll look at the timer template.

Create a service unit template file:

Add the following configuration:

(This file is maintained on GitHub)

With this template file in place, we can now ask systemd to start any cert-renewer@*.service.

For example, if you have a systemd service called postgresql.service, with certificate and key files located in /etc/step/certs/postgresql.crt and /etc/step/certs/postgresql.key, you can manually run systemctl start cert-renewer@postgresql.service and systemd will immediately check that certificate's readiness for renewal, and potentially renew it. If the certificate is successfully renewed, the postgresql service will be reloaded or restarted.

Customizing a service unit

You'll often need to customize your service unit for a given service.

For example, you may need to:

Instead of modifying the service template, we'll use service template overrides here. The overrides live in the drop-in configuration directory for the service being custommized.

Here's an example override for a Lighttpd service that uses Docker Compose and is not managed by systemd:

/etc/systemd/system/cert-renewer@lighttpd-docker.service.d/override.conf:

Here's a more complex example that calls the Grafana Data source HTTP API to refresh a client certificate stored in Grafana's configuration database.

/etc/systemd/system/cert-renewer@grafana-loki-datasource.service.d/override.conf:

When the certificate is successfully renewed:

Enabling systemd renewal timers

The final piece of the puzzle is the renewal timer. Timers and services go hand in hand in systemd: A cert-renewer@postgresql.timer will always trigger a corresponding cert-renewer@postgresql.service.

Therefore, instead of enabling the cert-renewer@*.service service units directly, we'll enable timer units that will periodically trigger each service unit.

Create a timer unit template file:

Add the following configuration to the file:

(This file is maintained on GitHub)

Timers using this template will run every 5-10 minutes, with a randomized delay on each timer. The randomized delay helps with thundering herd problems that can occur when many virtual machines that were provisioned together try to renew their certificates at the same time.

With this template in place, let's start timers for our postgresql and grafana-server renewer services:

Your periodic timers are now running and will run on system startup. You can override your timer units too, but you probably won't need to.

The standalone step renewal daemon

Another way to automate renewal is with the step renewal daemon. With this method, step ca renew operates as a daemon that will keep your certificates up-to-date:

When daemonized, step ca renew will attempt a renewal when the certificate's lifetime is approximately two-thirds elapsed. So, for a certificate with a 24 hour lifetime, it will attempt a renewal after about 16 hours.

There is some random jitter built into the daemon's schedule to prevent a large number of renewals from being sent to the CA simultaneously (eg. by a multitude of virtual machines that were provisioned at the same time and have identical certificate expiration dates).

If the CA is unreachable, renewals are retried every minute.

You can trigger renewal anytime by sending a SIGHUP signal to the step ca renew process ID.

You can add step ca renew --daemon as a systemd service that runs on startup and restarts as needed.

Here's an example of setting up everything via systemd:

Be sure the User has write access to the certificate and key you're renewing.

Rescan the systemd unit files:

Enable and start the service:

Notifying Certificate-Dependent Services

Many services that depend on certificates will only read the certificate files on startup. So when you renew a certificate, a server process that depends on it may not detect that it has changed.

It's common for services (eg. nginx) to respond to a SIGHUP signal by reloading configuration files and certificates. To address this, step ca renew can send a SIGHUP to your service after each renewal. Here's an example for nginx:

cron-based renewal

With cron-based renewal, you can have step ca renew run at a regular cadence (eg. every five minutes) and renew a certificate only when it's approaching its expiration date.

You can request that certificate renewal only occur if the certificate is approaching its expiry using the --expires-in <duration> flag. The <duration> is a time interval like 4h or 30m. Renewal will only occur if the expiry is within <duration> of the current time. For example:

With --expires-in, we add a random jitter to <duration> (between 0 and <duration>/20). This helps with thundering herd problems if many virtual machines that were provisioned together try to renew their certificates at the same time.

Here's an example of setting up renewal via cron on a Debian-based system:

Be sure the user (in this example, step) has write access to the certificate and key you're renewing.

Caveats of Automated Renewal

Revoking a Certificate

step ca renew allows a certificate owner to extend the lifetime of a certificate before it expires. Unfortunately, it also lets an attacker with the private key do the same thing. To prevent this, you need to explicitly tell step-ca to revoke a retired certificate. See the certificate revocation section for details.

Renewal after expiry: For intermittently-connected devices

You can configure a provisioner to allow certificates to be renewed after they expire. This is not the default behavior, but it's useful if you have devices with intermittent network connectivity. The device will be able to renew its certificate the next time it has network access, provided the certificate has not been revoked.

To configure renewal after expiry for a provisioner, pass the --allow-renewal-after-expiry flag to the step ca provisioner management commands. See managing provisioners.

X.509 Certificate Revocation

The hardware, software and policies for managing and distributing public keys and certificates is called Public Key Infrastructure (PKI).

The beauty of certificate-based PKI is that once certificates are issued, the infrastructure itself can be completely decentralized. You can independently validate any certificate you receive without communicating with any central authority.

Certificate-based is PKI inherently fault-tolerant, and trivial to scale.

With simplicity comes an inherent trade-off: Once a certificate is issued, the certificate authority (CA) can't un-issue it. It's valid until it expires. Certificates do eventually expire, but until they do, a bad actor can use a compromised private key to impersonate the certificate owner.

This is one reason step-ca is designed to issue short-lived certificates. When you revoke a certificate in step-ca, the CA will block the certificate's future renewal. This is called passive revocation. Passive revocation is a good option for internal PKI, because it avoids the complexity of relying on centralized third parties to check the real-time revocation status of a certificate.

A certificate that has been passively revoked will still be valid for the remainder of it's validity period.

Contrast this with active revocation techniques used on the public internet. Web TLS certificates are valid for up to a year. Web browsers have to check CRLs (Certificate Revocation Lists) or use OCSP (Online Certificate Status Protocol) to verify that every certificate they receive has not been revoked. Active revocation requires clients to take an active role in certificate validation for the benefit of real-time certificate status.

This section will walk through a few examples for revoking X.509 certificates using step-ca.

1. Create a certificate for localhost

Let's create a certificate that we'll revoke in a minute.

2. Renew the certificate

Before revoking the certificate, note that it can be renewed with step ca renew.

3. Revoke the certificate

There's two methods for revoking a certificate:

Revoke the certificate using its serial number

Using the serial number method, you have to authenticate your request (in this case, with the CA's provisioner password):

Revoke the certificate using the certificate and private key

In this case, the certificate and key authenticate the request, so you don't need a password:

4. Check that the certificate has been revoked

You'll get an HTTP 401 error when you try to renew it.

Using a revocation token

You can also revoke a certificate in two steps by first creating a revocation token and then exchanging that token in a revocation request.

You can also revoke certificates in offline mode:

Further Reading

Sane Cryptographic Defaults

The step ecosystem uses sane defaults so that you don't have to be a security engineer to use our step-ca safely. Our defaults align with best current practices in the industry for using cryptographic primitives and higher order abstractions, like JWTs.

This section describes our defaults and explains the rationale behind them. Our selections and guidance will change and evolve over time as security and cryptography are constantly changing in response to real world pressures.

Tokens

We use JWTs (JSON Web Tokens) to prove authenticity and identity within the step ecosystem. When configured well, JWTs are a great way to sign and encode data. It's easy to use JWTs insecurely, though, so you must be deliberate about how you validate and verify them (see RFC7519).

step-ca produces JWTs that:

If you're using step-ca JWTs in your code, be sure to verify and validate every standard attribute of the JWT. step crypto jwt verify can validate any JWT for you, and it follows the spec to the letter.

Key Types and Ciphers

Supported Key Types: ECDSA, EdDSA, and RSA
Default Key Type: ECDSA
Default Curve Bits: P-256

We chose ECDSA keys because they offer better security and performance than RSA keys. At 256 bits, ECDSA keys provide 128 bits of security, and they are supported by most modern clients.

More notes on the choice of key type:

Default PEM Cipher: AES128
Supported PEM Key Sizes: 128, 192, and 256 bits

We've chosen the AES encryption algorithm for writing private keys to disk because it was the official choice of the Advanced Encryption Standard contest.

All supported key sizes are considered to be unbreakable for the foreseeable future. We chose 128 bits as our default because the performance is better as compared to the greater key sizes, and because 128 bits are sufficient for most security needs.

X.509 Certificates

Root CA Certificate

The Root CA certificate is generated once, when you run step ca init.

Validity (10 year window)

A 10 year window is advisable until software and tools can be written for rotating the root certificate.

Basic Constraints

Key Usage

Key Usage describes how the certificate can be used.

Intermediate CA Certificate

The Intermediate CA certificate is generated once, when you run step ca init. It is signed by the Root CA certificate.

The Path Length of the intermediate certificate is 0. Otherwise it uses the same defaults as the root certificate.

A Path Length of zero indicates that there can be no additional intermediary certificates in the path between the intermediate CA certificate and end-entity certificates.

Leaf (End Entity) Certificate

These are the certificates issued by the step-ca server.

Validity (24 hour window)

The default is a 24hr window. This value is somewhat arbitrary. However, our goal is to have seamless end-entity certificate rotation. Rotating certificates frequently is a good security measure because it gives attackers very little time to form an attack and limits the usefulness of any single private key in the system.

We will continue to work towards decreasing this window because we believe it significantly reduces the probability and effectiveness of any attack.

Key Usage

Key Usage describes how the certificate can be used.

Extended Key Usage

TLS Defaults

These are the defaults used for communication between step and step-ca.

Min TLS Version: TLS 1.2
Max TLS Version: TLS 1.2

The PCI Security Standards Council required all payment processors and merchants to move to TLS 1.2 and above by June 30, 2018. By setting TLS 1.2 as the default for all TLS protocol negotiation, we encourage our users to adopt the same security conventions.

Renegotiation: Never

TLS renegotiation significantly complicates the state machine and has been the source of numerous, subtle security issues. Therefore, by default we disable it.

Default TLS Cipher Suites

The default 'ciphersuites' are a list of two cipher combinations. For communication between services running step there is no need for cipher suite negotiation. The server can specify a single cipher suite which the client is already known to support.

Reasons for selecting TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305:

The http2 spec requires the TLS_ECDHE_(RSA|ECDSA)_WITH_AES_128_GCM_SHA256 ciphersuite be accepted by the server, therefore it makes our list of default ciphersuites.

Approved TLS Cipher Suites

Above is a list of step-approved cipher suites. Not all communication can be resolved with step TLS functionality. For those connections, the list of server supported cipher suites must have more options in case older clients do not support our favored cipher suite.

Reasons for selecting these cipher suites can be found in the following ssllabs article.