SSH Emergency Access
Carl Tashian
In this post we'll design a break glass procedure for reaching SSH hosts in an emergency, using security keys that you can store offline. This is just one approach, but you can adapt it to your circumstances. We will store an offline SSH Certificate Authority on a hardware security key, and have our hosts trust that CA. This will work on pretty much any OpenSSH setup, including our single sign-on SSH.
Why would you want this? Only as an option of last resort. A backdoor into your servers when, for whatever reason, nothing else works.
Why use certificates instead of public/private keys for emergency access?
- Passive revocation. Certificates expire; public keys don't. You can mint an SSH certificate valid for 1 minute, or even 5 seconds. Once it expires, the certificate will become unusable for new connections. This is perfect for occasional emergency access.
- You'll be able to create a certificate for any account on your hosts and send short-lived certificates to colleagues as needed.
You will need
- Hardware security keys that support resident keys. Resident keys are crypto keys that are stored completely on the security key—sometimes protected by an alphanumeric PIN. The public part of a resident key can be exported from the security key when needed, along with a private key handle. The Yubikey 5 series keys support resident keys. Preferably, you'll dedicate these keys to emergency host access only. For this post I'll just use one key, but you should have an extra for backup.
- You'll need a safe place to store these keys.
- OpenSSH 8.2 or higher on your local machine and on the servers you want emergency access to. Ubuntu 20.04 ships with OpenSSH 8.2.
- (optional) The
step
CLI to inspect certificates — install it withbrew install step
on MacOS (Linux install instructions are here)
Instructions
-
Create a certificate authority that will reside on the security key
Insert the key and run:
$ ssh-keygen -t ecdsa-sk -f sk-user-ca -O resident -C [security key ID]
For the comment (
-C
), I suppliedyubikey-9-512-742@smallstep.com
to remind me which security key this CA refers to (on Yubikeys, the serial number is printed on the key itself).In addition to adding a key to the Yubikey, this will generate two files locally:
sk-user-ca
, the key handle which references the private key stored on the security key,sk-user-ca.pub
, which will be the public key for your CA.
But don't worry, there's another really private key stored on the Yubikey, that cannot be extracted.
-
Get your hosts to trust the CA
As
root
on your hosts, add the following to your SSHD config (/etc/ssh/sshd_config
) if you don't already have it:TrustedUserCAKeys /etc/ssh/ca.pub
Then, append the contents of your CA public key (
sk-user-ca.pub
) to/etc/ssh/ca.pub
on the host.Restart your SSHD server:
# systemctl sshd restart
Now we can try accessing the host. But we're going to need a certificate first!
Make a key pair that will be associated with the certificate:
$ ssh-keygen -t ecdsa -f emergency
Certificates and key pairs
It's tempting to think of an SSH certificate as a replacement for a public/private key pair. But a certificate alone is not enough to authenticate a user. Every SSH certificate also has a private key associated with it. That's why we need to generate this
emergency
key pair before we issue ourselves a certificate. What matters is that a signed certificate is presented to the server, pointing to a key pair for which you have the private key. We don't actually needemergency.pub
to run any of these commands, but we get one fromssh-keygen
anyway.So, public key exchange is still alive and well, even with certificates. Certificates just eliminate the need for the server to store public keys. Now create the certificate itself; it will allow logins within a 10 minute window that started 5 minutes ago (to account for clock skew), for the user
ubuntu
. You can change these values as needed.$ ssh-keygen -s sk-user-ca -I test-key -n ubuntu -V -5m:+5m emergency
You'll be asked to touch your security key to sign the certificate.
You can add additional usernames separated by commas, eg.
-n ubuntu,carl,ec2-user
You have a certificate! Now it needs the right permission:
$ chmod 600 emergency-cert.pub
You can inspect the certificate you just made, with:
$ step ssh inspect emergency-cert.pub
Here's what mine looks like:
emergency-cert.pub Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate Public key: ECDSA-CERT SHA256:EJSfzfQv1UK44/LOKhBbuh5oRMqxXGBSr+UAzA7cork Signing CA: SK-ECDSA SHA256:kLJ7xfTTPQN0G/IF2cq5TB3EitaV4k3XczcBZcLPQ0E Key ID: "test-key" Serial: 0 Valid: from 2020-06-24T16:53:03 to 2020-06-24T17:03:03 Principals: ubuntu Critical Options: (none) Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc
Here, the public key is the
emergencey
key you created, and the signing CA issk-user-ca
.You should be ready to ssh:
$ ssh -i emergency ubuntu@my-hostname ubuntu@my-hostname:~$
That's it! 🥳 You can now create SSH certificates for any user on a host that trusts your emergency Certificate Authority.
-
Finishing up
You can now delete
emergency*
. You can keepsk-user-ca*
around, but you don't need to because it also resides on the security key. You may also want to remove the original PEM public key from your hosts (eg. in~/.ssh/authorized_keys
for theubuntu
user), if you have been using that for emergency access.
Emergency access runbook
The next time you need emergency access, here's how to set yourself up.
Insert the security key and run:
$ ssh-add -K
This will bring the CA's public key and key handle into the SSH agent.
Now export the public key to make a certificate:
$ ssh-add -L | tail -1 > sk-user-ca.pub
Create a certificate for any user that will last an hour—or less, if you want:
$ ssh-keygen -t ecdsa -f emergency $ ssh-keygen -Us sk-user-ca.pub -I test-key -n [username] -V -5m:+60m emergency $ chmod 600 emergency-cert.pub
Now, you're ready to SSH:
$ ssh -i emergency username@host
If your existing .ssh/config
file causes any trouble when connecting, you can run ssh with -F none
to bypass it.
If you need to send a certificate to a colleague, Magic Wormhole is a simple and secure option. The only files they'll need are emergency
and emergency-cert.pub
.
What I love about this approach is that it's hardware backed. You can put the security keys in a safe, and they aren't going anywhere until you need them.
Subscribe to updates
Unsubscribe anytime, see Privacy Policy
Carl Tashian (Website, LinkedIn) is an engineer, writer, exec coach, and startup all-rounder. He's currently an Offroad Engineer at Smallstep. He co-founded and built the engineering team at Trove, and he wrote the code that opens your Zipcar. He lives in San Francisco with his wife Siobhan and he loves to play the modular synthesizer 🎛️🎚️