SSH certificate login advanced example: Run an SSH CA and connect to hosts using SSH certificates

OpenSSH and SSHD have supported SSH certificate login for almost ten years. In this tutorial, you'll learn how to use SSH certificates (for hosts & users) generated by open source step-ca using the step ssh sub-command.

About this tutorial

Learn how to configure OpenSSH and SSHD for certificate authentication using step-ca
Examples include copy/paste code blocks and pre-generated PKI.
When complete, you will be able to use OpenSSH and SSHD for SSH certificate login with a private SSH certificate authority.
Estimated effort: Reading time ~10 mins, Lab time ~60 to 180 mins.

If you run into any issues please let us know in GitHub Discussions.

Requirements

Open Source - This tutorial assumes you have initialized and started up a step-cainstance using the steps in Getting Started.
Smallstep Certificate Manager - Please contact Smallstep Customer Success if you would like to activate SSH certificates on your Certificate Manager account.
Vagrant

Overview

You can find all of the code in the example below here: https://github.com/smallstep/step-ssh-example

The code in this repo comes with a pre-generated PKI. You will need step v0.13.3+ and Vagrant, plus a provider like VirtualBox installed locally.

Setup VM

You're going to run a CA in your local environment and we'll use SSH to connect to a Vagrant VM (representing a remote host) that has sshd pre-configured to accept SSH user certificates signed by our CA.

With Vagrant installed, run the following commands inside the repo:

$ vagrant up
Bringing machine 'testhost' up with 'virtualbox' provider...
==> testhost: Importing base box 'ubuntu/bionic64'...
    [...]
==> testhost: Preparing network interfaces based on configuration...
    testhost: Adapter 1: nat
    testhost: Adapter 2: hostonly
==> testhost: Forwarding ports...
    testhost: 22 (guest) => 2222 (host) (adapter 1)
==> testhost: Running 'pre-boot' VM customizations...
==> testhost: Booting VM...
==> testhost: Waiting for machine to boot. This may take a few minutes...
    testhost: SSH address: 127.0.0.1:2222
    testhost: SSH username: vagrant
    testhost: SSH auth method: private key
    testhost: VirtualBox Version: 6.0
    [...]
==> testhost: Setting hostname...
==> testhost: Configuring and enabling network interfaces...
==> testhost: Mounting shared folders...
    testhost: /keys => /Users/sourishkrout/dev/src/smallstep/code/src/github.com/smallstep/step-examples/ssh-example/keys
    testhost: /vagrant => /Users/sourishkrout/dev/src/smallstep/code/src/github.com/smallstep/step-examples/ssh-example
==> testhost: Running provisioner: shell...
    testhost: Running: inline script
    testhost: Add following line to your local hosts ~/.ssh/known_hosts file to accept host certs
    testhost: @cert-authority * ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJM+jkIdieQvdPb8DwnfnJudEc9PgVBqLDWHKgvqoIiMXhuIyGstQ9ULOBMdJkqxMjkRTFZp1iFvIk+iU6hwTA=
    testhost: Add a /etc/hosts file entry testhost to resolve to 192.168.0.101
    testhost: Check out README.md to learn how to grab user ssh certs to log into testhost

Configure ssh client to accept host certs

Go ahead and follow the instructions printed by Vagrant. This will enable your local SSH client to accept SSH host certificates (signed by the root SSH host private key). The following command will append the SSH host CA key (root SSH host public key corresponding to the root SSH host private key) to your local known_hosts file:

$ echo "@cert-authority * ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJM+jkIdieQvdPb8DwnfnJudEc9PgVBqLDWHKgvqoIiMXhuIyGstQ9ULOBMdJkqxMjkRTFZp1iFvIk+iU6hwTA=" >> ~/.ssh/known_hosts

You can also find the root SSH host CA key stored at step/certs/ssh_host_key.pub in this repo.

SSH certificates bind names to public keys. This SSH host certificate has the identity testhost which is why the following entry must be added to the local /etc/hosts file on the VM:

$ tail -n 1 /etc/hosts
192.168.0.101   testhost

Configure sshd to accept user certs

Vagrant has already configured sshd on testhost, the VM generated by Vagrant. Please note that for demo purposes the PKI is shared with the VM using a shared directory mount. Below you can see the relevant lines from the testhost VM's sshd_config:

$ tail -n 5 /etc/ssh/sshd_config
#       PermitTTY no
#       ForceCommand cvs server
TrustedUserCAKeys /keys/ssh_user_key.pub
HostKey /keys/ssh_host_ecdsa_key
HostCertificate /keys/ssh_host_ecdsa_key-cert.pub

TrustUserCAKeys: The root SSH user public key used to verify SSH user certificates.
HostKey: The SSH private key specific to this host.
HostCertificate: The SSH public certificate that uniquely identifies this host (signed by the root SSH host private key).

A valid user certificate is required to log into the testhost VM. Using step, you can authenticate with your SSH-enabled CA and fetch a new SSH certificate.

In one terminal window, run the following command to startup your CA (password is password):

$ export STEPPATH=pwd/step
$ step-ca step/config/ca.json
Please enter the password to decrypt step/secrets/intermediate_ca_key: password
Please enter the password to decrypt step/secrets/ssh_host_key: password
Please enter the password to decrypt step/secrets/ssh_user_key: password
2019/09/11 22:59:01 Serving HTTPS on :443 ...

In another terminal window run:

$ export STEPPATH=pwd/step
$ step ssh certificate testuser testuser_ecdsa --ca-url https://localhost --root step/certs/root_ca.crt
✔ Provisioner: admin (JWK) [kid: ux6AhkfzgclpI65xJeGHzNqHCmdCl0-nWO8YqF1mcn0]
✔ Please enter the password to decrypt the provisioner key: password
✔ CA: https://localhost
Please enter the password to encrypt the private key: your-own-password
✔ Private Key: testuser_ecdsa
✔ Public Key: testuser_ecdsa.pub
✔ Certificate: testuser_ecdsa-cert.pub
✔ SSH Agent: yes

step-ca enforces authentication for all certificate requests and uses the concept of provisioners to carry out this enforcement. Provisioners are configured in step/config/ca.json. Authenticating as one of the sanctioned provisioners indicates to step-ca that you have the right to provision new SSH certificates.

In the above invocation of step ssh certificate, you have authenticated your request using a JWK provisioner which requires a password to decrypt a private key. There are a handful of supported provisioners, each with their own authentication methods. The OIDC provisioner is particularly interesting for SSH user certificates because it enables Single Sign-On SSH.

step ssh certificate adds the new SSH user certificate to your local ssh agent. The default lifetime of an SSH certificate from step-ca is 16 hours. The lifetime can be configured using command line options.

Run step ssh certificate -h in step for documentation and examples).

$ ssh-add -l
256 SHA256:xt5VeMEG8uf+SBlddauylJHv9+Bl0E6H+46AV94+Its testuser (ECDSA-CERT)

$ ssh testuser@testhost
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)

[...]

As you can see the testhost VM will welcome you with a matching testuser@testhost prompt.

Learn how to use OAuth OIDC providers like Google Workspace or Instance Identity Documents to bootstrap SSH host and user certificates in the step reference.

Generate ssh host certificates

This example repo includes a pre-generated SSH host certificate and key. To replace it, or generate SSH certificates for other hosts, run the following command:

step ssh certificate --host --principal testhost --principal testhost.internal testhost ssh_host_ecdsa_key

Where --principal identifies the hostname(s), ideally FQDNs, for the machine. For a single principal you can short cut the command to:

step ssh certificate --host testhost ssh_host_ecdsa_key

Generate your own PKI for step-ca

We recommend using your own PKI for usage outside of this example. You can initialize your step-ca with both X509 and SSH certificate authorities using the following command:

$ export STEPPATH=/tmp/mystep
$ step ca init --ssh
✔ What would you like to name your new PKI? (e.g. Smallstep): Smallstep
✔ What DNS names or IP addresses would you like to add to your new CA? (e.g. ca.smallstep.com[,1.1.1.1,etc.]): localhost
✔ What address will your new CA listen at? (e.g. :443): :443
✔ What would you like to name the first provisioner for your new CA? (e.g. you@smallstep.com): admin
✔ What do you want your password to be? [leave empty and we will generate one]:

Generating root certificate...
all done!

Generating intermediate certificate...

Generating user and host SSH certificate signing keys...
all done!

✔ Root certificate: /tmp/mystep/certs/root_ca.crt
✔ Root private key: /tmp/mystep/secrets/root_ca_key
✔ Root fingerprint: d601c93a6256080e42cf02087fdc737f1429226ada6c040bac6494332e01527e
✔ Intermediate certificate: /tmp/mystep/certs/intermediate_ca.crt
✔ Intermediate private key: /tmp/mystep/secrets/intermediate_ca_key
✔ SSH user root certificate: /tmp/mystep/certs/ssh_user_key.pub
✔ SSH user root private key: /tmp/mystep/secrets/ssh_user_key
✔ SSH host root certificate: /tmp/mystep/certs/ssh_host_key.pub
✔ SSH host root private key: /tmp/mystep/secrets/ssh_host_key
✔ Default configuration: /tmp/mystep/config/defaults.json
✔ Certificate Authority configuration: /tmp/mystep/config/ca.json\n
Your PKI is ready to go. To generate certificates for individual services see 'step help ca'.

Now you can launch your instance of step-ca with your own PKI like so:

$ step-ca $(step path)/config/ca.json
Please enter the password to decrypt /tmp/mystep/secrets/intermediate_ca_key:
Please enter the password to decrypt /tmp/mystep/secrets/ssh_host_key:
Please enter the password to decrypt /tmp/mystep/secrets/ssh_user_key:
2019/09/11 23:34:13 Serving HTTPS on :443 ...

Please note that after you regenerate ssh_host_key.pub and ssh_user_key.pub, you will have to reconfigure ssh and sshd for clients and hosts to accept the new CA keys. Check out this host bootstrapping script for configuration examples.