SSH Agent Explained
The SSH agent is a central part of OpenSSH. In this post, I'll explain what the agent is, how to use it, and how it works to keep your keys safe. I'll also describe agent forwarding and how it works. I'll help you reduce your risk when using agent forwarding, and I'll share an alternative to agent forwarding that you can use when accessing your internal hosts through bastions.
What is the SSH agent?
ssh-agent is a key manager for SSH. It holds your keys and certificates in memory, unencrypted, and ready for use by
ssh. It saves you from typing a passphrase every time you connect to a server. It runs in the background on your system, separately from
ssh, and it usually starts up the first time you run
ssh after a reboot.
The SSH agent keeps private keys safe because of what it doesn't do:
- It doesn't write any key material to disk.
- It doesn't allow your private keys to be exported.
Private keys stored in the agent can only be used for one purpose: signing a message.
But if the agent can only sign messages, how does SSH encrypt and decrypt traffic?
When first learning about public and private SSH keys, it's natural to assume that SSH uses these key pairs to encrypt and decrypt traffic. That's what I thought. But it's not the case. An SSH key pair is only used for authentication during the initial handshake.
For example, here's how a user's key is verified during the SSH handshake, from the server's perspective:
- The client presents a public key to the server.
- The server generates and sends a brief, random message, asking the client to sign it using the private key.
- The client asks the SSH agent to sign the message and forwards the result back to the server.
- The server checks the signature using the client's public key.
- The server now has proof that the client is in possession of their private key.
Later in the handshake process, a set of new, ephemeral and symmetric keys are generated and used to encrypt the SSH session traffic. These keys may not even last the entire session; a "rekey" event happens at regular intervals.
The agent protocol
SSH uses a Unix domain socket to talk to the agent via the SSH agent protocol. Most people use the
ssh-agent that comes with OpenSSH, but there's a variety of open-source alternatives.
The agent protocol is so simple that one could write a basic SSH agent in a day or two. It only has a few primary operations:
- Add a regular key pair (public and decrypted private keys)
- Add a constrained key pair (public and decrypted private keys)
- Add a key (regular or constrained) from a smart card (public key only)
- Remove a key
- List keys stored in the agent
- Sign a message with a key stored in the agent
- Lock or unlock the entire agent with a passphrase
🤔 What's a constrained key? It's usually a key that either has a limited lifetime or one that demands explicit user confirmation when it is used. The
ssh-addcommand is your gateway to the SSH agent. It performs all of these operations except for signing. When you run
ssh-addwithout any parameters, it will scan your home directory for some standard keys and add them to your agent. By default, it looks for:
Once you add keys to the keychain, they will be used automatically by
ssh-agent and the macOS Keychain
ssh-agent that ships with macOS can store the passphrase for keys in the macOS Keychain, which makes it even easier to re-add keys to the agent after a reboot. Depending on your Keychain settings, you still may need to unlock the keychain after a reboot. To store key passphrases in the Keychain, run
ssh-add -K [key filename]. Passphrases are usually stored in the "Local Items" keychain.
ssh-agent will use these stored passphrases automatically as needed.
What is agent forwarding?
SSH's agent forwarding feature allows your local SSH agent to reach through an existing SSH connection and transparently authenticate on a more distant server. For example, say you SSH into an EC2 instance, and you want to clone a private GitHub repository from there. Without agent forwarding, you'd have to store a copy of your GitHub private key on the EC2 host. With agent forwarding, the SSH client on EC2 can use the keys on your local computer to authenticate to GitHub.
How agent forwarding works
First, a little background. SSH connections can have multiple channels. Here's a common example: an interactive connection to a bastion host (jump box) runs on one channel. When agent forwarding is enabled for a connection (usually using
ssh -A), a second channel is opened up in the background to forward any agent requests back to your local machine.
ssh's perspective, there is no difference between a remote and a local
ssh-agent. SSH always looks at the
$SSH_AUTH_SOCK environment variable to find the Unix domain socket for the agent. When you connect to a remote host with agent forwarding enabled, SSHD will create a remote Unix domain socket linked to the agent forwarding channel, and export an
$SSH_AUTH_SOCK pointing to it.
Agent forwarding comes with a risk
When you forward
ssh-agent's Unix domain socket to a remote host, it creates a security risk: anyone with root access on the remote host can discreetly access your local SSH agent through the socket. They can use your keys to impersonate you on other machines on the network.
Here's an example of how that might look:
How to reduce your risk when agent forwarding
Here are a few ways to make agent forwarding safer:
Don't turn on
Many guides on agent forwarding will suggest turning on
ForwardAgentusing the following configuration:
Host example.com ForwardAgent yes
We suggest not doing that. Instead, only use agent forwarding in circumstances where you need it.
ssh -Aturns on agent forwarding for a single session.
Lock your ssh agent when you use agent forwarding.
ssh-add -xlocks the agent with a password, and
ssh-add -Xunlocks it. When you're connected to a remote host with agent forwarding, no one will be able to snake their way into your agent without the password.
Or use an alternative SSH agent that prompts you when it's being used. Sekey and Secretive are open source projects that use Touch ID on macOS to store keys in the MacBook Pro's security enclave.
Or don't use agent forwarding at all. If you're trying to access internal hosts through a bastion,
ProxyJumpis a much safer alternative for this use case. (see below)
ProxyJump: a safer alternative
When you want to go through a bastion host (jumpbox), you really don't need agent forwarding. A better approach is to use the
Instead of forwarding the agent through a separate channel,
ProxyJump forwards the standard input and output of your local SSH client through the bastion and on to the remote host. Here's how that works:
ssh -J bastion.example.com cloud.computer.internalto connect to
cloud.computer.internalis a hostname that can be looked up using DNS lookup on
- Your SSH client uses keys from your agent to connect to
- Once connected, SSHD on the bastion connects to
cloud.computer.internaland hands that connection off to your local SSH client.
- Your local SSH client runs through the handshake again, this time with
You can think of it as SSHing within an SSH session; except the
ssh program never runs on the bastion. Instead,
sshd connects to
cloud.computer.internal and gives control of that connection (standard in and out) back to your local SSH, which then performs a second handshake.
Let's say my bastion host is
bastion.example.com. I could set up my
~/.ssh/config file like this:
Host bastion.example.com User carl Host *.computer.internal ProxyJump bastion.example.com User carl
Then I just run
ssh cloud.computer.internal to connect to an internal destination through the bastion—without agent forwarding.
ProxyJump doesn't work...
Older versions of SSH and SSHD (prior to 7.2, released in 2016) don't support
ProxyJump. But you can do an equivalent operation using
ProxyCommand and netcat. Here's an example:
ssh -o ProxyCommand="ssh bastion.example.com nc %h %p" cloud.computer.internal
The magic here is that SSH itself is the proxy you're using for SSH. The
nc %h %p part simply opens up a raw socket connection to
cloud.computer.internal on port 22. The standard I/O of the parent ssh command is piped right into the
ProxyCommand so that the parent ssh can authenticate to the internal host through the proxy connection.
We love talking about all things SSH. Hit us up on Twitter if you have questions or feedback!
Carl Tashian (LinkedIn, Twitter) has a couple decades of experience as an engineer, writer, and founder. Before Smallstep, he co-founded and built the engineering team at Trove. He also wrote the code that opens your Zipcar. These days he can be found down all the best PKI rabbit holes, chasing information to help you secure your infrastructure.