In this post we’ll design a break glass procedure for reaching SSH hosts in an emergency, using security keys that you can store offline. This is just one approach, but you can adapt it to your circumstances. We will store an offline SSH Certificate Authority on a hardware security key, and have our hosts trust that CA. This will work on pretty much any OpenSSH setup, including our single sign-on SSH.
Why would you want this? Only as an option of last resort. A backdoor into your servers when, for whatever reason, nothing else works.
Why use certificates instead of public/private keys for emergency access?
- Passive revocation. Certificates expire; public keys don’t. You can mint an SSH certificate valid for 1 minute, or even 5 seconds. Once it expires, the certificate will become unusable for new connections. This is perfect for occasional emergency access.
- You’ll be able to create a certificate for any account on your hosts and send short-lived certificates to colleagues as needed.
You will need
- Hardware security keys that support resident keys. Resident keys are crypto keys that are stored completely on the security key—sometimes protected by an alphanumeric PIN. The public part of a resident key can be exported from the security key when needed, along with a private key handle. The Yubikey 5 series keys support resident keys. Preferably, you’ll dedicate these keys to emergency host access only. For this post I’ll just use one key, but you should have an extra for backup.
- You’ll need a safe place to store these keys.
- OpenSSH 8.2 or higher on your local machine and on the servers you want emergency access to. Ubuntu 20.04 ships with OpenSSH 8.2.
- (optional) The
stepCLI to inspect certificates — install it with
brew install stepon MacOS (Linux install instructions are here)
Create a certificate authority that will reside on the security key
Insert the key and run:
$ ssh-keygen -t ecdsa-sk -f sk-user-ca -O resident -C [security key ID]
For the comment (
-C), I supplied
email@example.com remind me which security key this CA refers to (on Yubikeys, the serial number is printed on the key itself).
In addition to adding a key to the Yubikey, this will generate two files locally:
sk-user-ca, the key handle which references the private key stored on the security key,
sk-user-ca.pub, which will be the public key for your CA.
But don’t worry, there’s another really private key stored on the Yubikey, that cannot be extracted.
Get your hosts to trust the CA
rooton your hosts, add the following to your SSHD config (
/etc/ssh/sshd_config) if you don’t already have it:
Then, append the contents of your CA public key (
/etc/ssh/ca.pubon the host.
Restart your SSHD server:
# systemctl sshd restart
Now we can try accessing the host. But we’re going to need a certificate first!
Make a key pair that will be associated with the certificate:
$ ssh-keygen -t ecdsa -f emergency
Certificates and key pairs
It’s tempting to think of an SSH certificate as a replacement for a public/private key pair. But a certificate alone is not enough to authenticate a user. Every SSH certificate also has a private key associated with it. That’s why we need to generate this
emergencykey pair before we issue ourselves a certificate. What matters is that a signed certificate is presented to the server, pointing to a key pair for which you have the private key. We don’t actually need
emergency.pubto run any of these commands, but we get one from
So, public key exchange is still alive and well, even with certificates. Certificates just eliminate the need for the server to store public keys.
Now create the certificate itself; it will allow logins within a 10 minute window that started 5 minutes ago (to account for clock skew), for the user
ubuntu. You can change these values as needed.
$ ssh-keygen -s sk-user-ca -I test-key -n ubuntu -V -5m:+5m emergency
You’ll be asked to touch your security key to sign the certificate.
You can add additional usernames separated by commas, eg.
You have a certificate! Now it needs the right permission:
$ chmod 600 emergency-cert.pub
You can inspect the certificate you just made, with:
$ step ssh inspect emergency-cert.pub
Here’s what mine looks like:
emergency-cert.pub Type: firstname.lastname@example.org user certificate Public key: ECDSA-CERT SHA256:EJSfzfQv1UK44/LOKhBbuh5oRMqxXGBSr+UAzA7cork Signing CA: SK-ECDSA SHA256:kLJ7xfTTPQN0G/IF2cq5TB3EitaV4k3XczcBZcLPQ0E Key ID: "test-key" Serial: 0 Valid: from 2020-06-24T16:53:03 to 2020-06-24T17:03:03 Principals: ubuntu Critical Options: (none) Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc
Here, the public key is the
emergenceykey you created, and the signing CA is
You should be ready to ssh:
$ ssh -i emergency ubuntu@my-hostname ubuntu@my-hostname:~$
That’s it! 🥳 You can now create SSH certificates for any user on a host that trusts your emergency Certificate Authority.
You can now delete
emergency*. You can keep
sk-user-ca*around, but you don’t need to because it also resides on the security key. You may also want to remove the original PEM public key from your hosts (eg. in
ubuntuuser), if you have been using that for emergency access.
Emergency access runbook
The next time you need emergency access, here’s how to set yourself up.
Insert the security key and run:
$ ssh-add -K
This will bring the CA’s public key and key handle into the SSH agent.
Now export the public key to make a certificate:
$ ssh-add -L | tail -1 > sk-user-ca.pub
Create a certificate for any user that will last an hour—or less, if you want:
$ ssh-keygen -t ecdsa -f emergency $ ssh-keygen -Us sk-user-ca.pub -I test-key -n [username] -V -5m:+60m emergency $ chmod 600 emergency-cert.pub
Now, you’re ready to SSH:
$ ssh -i emergency username@host
If your existing
.ssh/config file causes any trouble when connecting, you can run ssh with
-F none to bypass it.
If you need to send a certificate to a colleague, Magic Wormhole is a simple and secure option. The only files they’ll need are
What I love about this approach is that it’s hardware backed. You can put the security keys in a safe, and they aren’t going anywhere until you need them.