How to use step-ca with Hardware Security Modules (HSMs)

Carl Tashian

Follow Smallstep

We're excited to add support for Hardware Security Modules (HSMs), via PKCS #11, to step-ca version 0.15.8.

An HSM is a specialized device that is designed to generate, store, and use private keys securely. The private keys on an HSM cannot be exported from the device. One can only run signing operations on the device. This is an excellent way to protect private keys for a Certificate Authority, whose primary function is to sign Certificate Signing Requests. In the early 2000s, HSMs were sometimes marketed as "eCommerce Accelerators" because companies used them primarily to speed up cryptographic operations for SSL connections. Today, they are more often used to meet tight security and compliance requirements.

PKCS (Public-Key Cryptography Standards) #11 is the most common platform-independent API used by HSMs. There's even a software-based "HSM," SoftHSMv2, which offers a PKCS #11 interface for testing purposes, without the hardware.

With PKCS #11 support, step-ca can be configured to use almost any HSM. HSMs range from the $80 USB Nitrokey HSM 2 all the way up to the $39,000 Entrust nShield Connect XC High rack-mounted HSM.

AWS CloudHSM is an interesting offering because it offers real HSM hardware by the hour and—you guessed it—it's in the cloud. CloudHSM offers HSM clusters, with multi-AZ support as needed. It offers private, single-tenant HSMs with FIPS 140-2 Level 3 security, and costs about $1,000/month/HSM.

In this post, I'll show you how to set up step-ca using AWS CloudHSM. Our Configuration Guide details how to use step-ca's HSM support with any PKCS #11 device.

Tutorial: step-ca + AWS CloudHSM

AWS CloudHSM is an HSM that you can access from your VPC using mutual TLS. It comes with an optional client library that makes it compatible with step-ca via the PKCS #11 API. In this example, we will store the CA certificates for step-ca as files on an EC2 VM, but we'll delegate signing operations to the CloudHSM cluster.

0. Before you begin

You will need:

• An active CloudHSM cluster. To set one up from scratch, follow all of the instructions in the Getting Started section of the AWS CloudHSM User Guide. There's a lot of steps here and it will take at least 30 minutes to do this initial setup.

  When you're done, you will have:

  • activated an HSM cluster with at least one HSM in it
  • mapped an EC2 instance to the cloudhsm cluster's security group. I used an instance running Ubuntu 18.04.
  • installed and configured the cloudhsm-client package
  • created and installed (into /opt/cloudhsm/etc) your self-signed cluster CA certificate
  • downloaded and signed your cluster CSR using your customer CA
  • initialized the cluster using the CSR you made (aws cloudhsmv2 initialize-cluster)
  • changed the default HSM cluster admin password, using the cloudhsm_mgmt_util tool

• The CloudHSM PKCS #11 Library. On your EC2 instance, install the cloudhsm-client-pkcs11 by following the instructions in Installing the PKCS #11 Library.

1. Create a certificate user (CU) to use with step-ca

On your EC2 instance, run the HSM management utility, log in to the HSM, and create a "CU" type user for use with step-ca:

$ /opt/cloudhsm/bin/cloudhsm_mgmt_util /opt/cloudhsm/etc/cloudhsm_mgmt_util.cfg
Ignoring E2E enable flag in the configuration file
Connecting to the server(s), it may take time
depending on the server(s) load, please wait...
Connecting to server '172.31.27.83': hostname '172.31.27.83', port 2225...
Connected to server '172.31.27.83': hostname '172.31.27.83', port 2225.
E2E enabled on server 0(172.31.27.83)
aws-cloudhsm> loginHSM CO admin <admin_password>
aws-cloudhsm> createUser CU step_ca RiFJrg93Tn_EXAMPLE
aws-cloudhsm> quit

Note your password! We'll need it later to configure step-ca.

2. Build step-ca with HSM support

Now let's build step-ca. HSM support requires a CGO version of step-ca. Unfortunately, step-ca doesn't distribute CGO binaries, so we'll need to build it from scratch to enable HSM support.

On the VM, run the following as root to build step-ca:

apt update
apt upgrade -y
apt install -y make gcc ack libpcsclite-dev pkg-config
GO_VERSION=1.16
curl -LO https://golang.org/dl/go$GO_VERSION.linux-amd64.tar.gz
tar -C /usr/local -xzf go$GO_VERSION.linux-amd64.tar.gz
cat<<EOF >> /etc/profile
export PATH=\$PATH:/usr/local/go/bin
EOF

Now sign out and back in, so the configuration takes effect.

As a non-root user, let's build the CA:

curl -LO https://github.com/smallstep/certificates/releases/download/v0.15.8/step-certificates_0.15.8.tar.gz
mkdir certificates
pushd certificates
tar xzf ../step-certificates_0.15.8.tar.gz
make bootstrap
make build GOFLAGS=""
popd

At this point you'll have all of the binaries for step-ca in the certificates/bin directory.

3. Create your Public Key Infrastructure (PKI) keys and certificates

Now we'll create keys on the HSM for our CA, using the step-pkcs11-init utility.

I've added an extra space before setting the HSM_USER and HSM_PASSWORD environment variables here. In Bash, the extra space will exclude these values from our shell history.

$  HSM_USER=step_ca HSM_PASSWORD=RiFJrg93Tn_EXAMPLE
$ certificates/bin/step-pkcs11-init --no-certs --ssh \
                     --kms "pkcs11:module-path=/opt/cloudhsm/lib/libcloudhsm_pkcs11.so;token=cavium?pin-value=$HSM_USER:$HSM_PASSWORD"
Creating PKI ...
✔ Root Key: pkcs11:id=7330;object=root-key
✔ Root Certificate: root_ca.crt
✔ Intermediate Key: pkcs11:id=7331;object=intermediate-key
✔ Intermediate Certificate: intermediate_ca.crt
✔ SSH Host Key: pkcs11:id=7332;object=ssh-host-key
✔ SSH User Key: pkcs11:id=7333;object=ssh-user-key

What just happened? The utility created SSH and X.509 CA private keys on the HSM, and it saved the root and intermediate CA certificates onto disk (CloudHSM doesn't yet support saving the certificates to the HSM). It printed the URIs for the keys we're going to use in the CA. (Save these! You'll need them to configure your CA.)

If you run into trouble here, be sure the cloudhsm-client service is running and the credentials are correct.

4. Configure step-ca

First, let's get step-ca installed:

$ sudo cp certificates/bin/step-ca /usr/bin
$ sudo setcap CAP_NET_BIND_SERVICE=+eip /usr/bin/step-ca

We'll also want to install the step command line tool:

$ wget https://github.com/smallstep/cli/releases/download/v0.15.8/step_0.15.8_amd64.deb
$ sudo dpkg -i step_linux_0.15.8_amd64.deb

Now run step ca init --ssh and go through the steps. We're not going to use the PKI generated here, but we will use the configuration file as our basis.

Once you have your PKI generated in .step, move your certificate files into their homes:

$ mv root_ca.crt intermediate_ca.crt .step/certs
$ rm -r .step/secrets

Now open up .step/config/ca.json in an editor. You'll want the top of the file to reference the HSM intermediate key and SSH keys you just created. For this, you'll need to change the key and ssh key to match the pkcs11: URIs above. These are the values you need to change:

{
        ...
        "key": "pkcs11:id=7331;object=intermediate-key",
        "ssh": {
                "hostKey": "pkcs11:id=7332;object=ssh-host-key",
                "userKey": "pkcs11:id=7333;object=ssh-user-key"
        },
        ...
}

Finally, add a new kms top-level object, with your pkcs11 module URI:

{
        ...
        "kms": {
                "type": "pkcs11",
                "uri": "pkcs11:module-path=/opt/cloudhsm/lib/libcloudhsm_pkcs11.so;token=cavium?pin-value=step_ca:RiFJrg93Tn_EXAMPLE"
        },
        ...
}

The CA should start right up, using the HSM:

$ step-ca .step/config/ca.json
badger 2021/03/04 00:24:20 INFO: All 1 tables opened in 0s
badger 2021/03/04 00:24:20 INFO: Replaying file id: 0 at offset: 1407
badger 2021/03/04 00:24:20 INFO: Replay took: 4.527µs
badger 2021/03/04 00:24:20 DEBUG: Value log discard stats empty
2021/03/04 00:24:20 Serving HTTPS on :443 ...

In another window, get a test certificate for localhost:

$ step ca certificate localhost localhost.key localhost.crt

Tada. You're now getting certificates signed by the CloudHSM cluster.

Alternatives to HSMs for storing private keys in step-ca

• step-ca also supports AWS Key Management Service (KMS) or Google Cloud KMS. These services are HSM-backed. However, they are not single-tenant solutions, so they won't meet everyone's security or compliance requirements. Our configuration guide has instructions for setting up step-ca to use cloud key management services.
• For on-premise installations, the YubiKey PIV application can be used, with any recent YubiKey. See our post, Build a Tiny Certificate Authority for your Homelab.
• Commercial offering: For about $75/vCPU/month, you can run step-ca as a Registration Authority (RA) in front of Google's Certificate Authority Service. See this post for details.

Read more about Cryptographic Protection in step-ca in our Configuration Guide

Other new features in step and step-ca v0.15.8

• HSM support! In case that's not obvious from the rest of this post.
• Retired key management certificate slots (slots 82-95) are now supported by YubiKey PIV.
• We've updated the Windows install instructions for step and we now support Windows installation via Scoop.