SSH Tips & Tricks

Carl Tashian

Follow Smallstep

Here are some of our best tips & tricks for using SSH more effectively. This post will cover how to:

• Add a second factor to your SSH login
• Use agent forwarding safely
• Exit from stuck SSH sessions
• Keep a persistent terminal open
• Share a remote terminal session with a friend (without Zoom!)

Add a second factor to your SSH

Here's five different ways to add a second factor to your SSH connections:

1. Upgrade your OpenSSH and use a security key. In February 2020 OpenSSH added support for FIDO U2F (Universal Second Factor) security keys. This is a great new feature, but there's a caveat: Because this upgrade introduces new key types for security keys, you'll only be able to use a security key if both client and server have been upgraded to OpenSSH 8.2+. You can check your client version with ssh -V and a remote sshd version with nc [servername] 22

   Two new key types were added—ecdsa-skand ed25519-sk (along with corresponding certificate types). To generate your key files, insert your security key and run:

   $ ssh-keygen -t ecdsa-sk -f ~/.ssh/id_ecdsa_sk
   

   This creates public and private keys tied to your U2F device. A private key on the U2F device is used to decrypt the on-disk private "key handle" when the security key is activated.

   You can also supply a passphrase for your keys, as a second factor.

   OpenSSH also supports a different style of generating -sk-type keys: resident keys. With the resident key approach, the key handle is stored on the U2F device. This way, you'll always have it on the security key when you need it. Create resident keys with:

   $ ssh-keygen -t ecdsa-sk -O resident -f ~/.ssh/id_ecdsa_sk
   

   Then, to bring the key handle back into memory on a new machine, insert the security key and run:

   $ ssh-add -K
   

   You'll still need to activate the security key when connecting to a host.

2. Use PIV+PKCS11 and a Yubikey. If you want to connect to machines running older versions of SSHD, you can still use a security key—with a different approach. Yubico has a guide for using U2F+SSH with PIV/PKCS11. This is not the same thing as FIDO U2F. It works, but it's arcane to use.

3. Use the custom yubikey-agent ssh-agent. Filippo Valsorda wrote an SSH agent for Yubikeys. It's brand new and has minimal features.

4. Use the Secure Enclave (on a Mac). Sekey and Secretive are open source SSH agents that store private keys in the Mac's secure enclave and make the signing function available via Touch ID.

5. Use Single Sign On SSH. I wrote a tutorial to help you set this up. One advantage of single sign on SSH is, you can leverage the security policy of your identity provider—including multi-factor authentication (MFA) support.

Use agent forwarding safely

Agent forwarding in SSH allows a remote host to access your local machine's SSH agent. When you SSH with agent forwarding turned on (often using ssh -A), the connection will have two channels: Your interactive session, and a channel for agent forwarding. The Unix domain socket created by your local SSH agent gets connected to the remote host through this channel. This is risky because a user with root access on the remote machine can access your local SSH agent and potentially impersonate you on the network. With the standard SSH agent that ships with OpenSSH, you wouldn't even know this was happening. If you use a U2F key (or Sekey), you will effectively thwart any attempt to use your SSH agent.

Even with this precaution, it's still a good idea to use agent forwarding sparingly. Don't use it for all sessions—use it when you know you'll need it for a specific SSH session.

Exit stuck sessions

SSH sessions can often hang due to network interruptions, a program that gets out of control, or one of those terminal escape sequences that lock keyboard input. 🤦‍♂️

Here's how to get out of a stuck session:

1. Exit automatically on network interruptions. In your .ssh/config, add:

   ServerAliveInterval 5
   ServerAliveCountMax 1
   

   ssh will check the connection by sending an echo to the remote host every ServerAliveInterval seconds. If more than ServerAliveCountMax echos are sent without a response, ssh will timeout and exit.

2. Break out of the session. ssh includes the escape character ~ by default. The command ~. closes an open connection and brings you back to the terminal. (You can only enter escape sequences on a new line.) ~? lists all of the commands you can use during a session. On international keyboards, you may need to press the ~ key twice to send the ~ character.

Why do stuck sessions happen? Computers didn't move around very much when the internet was invented. When you’re on a laptop and you move between IPv4 WiFi networks, your IP address changes. Because SSH relies on TCP connections, and TCP connections depend on endpoints with stable IP addresses, anytime you change networks your SSH connections will become orphaned and effectively lost. When your IP address changes, it takes a while for your network stack to discover that a connection has been orphaned. With TCP connections, we don’t want one side to close down the connection too quickly when there's a network issue. So TCP will keep retrying to send data for a while before it gives up. Meanwhile in your terminal, the session appears stuck. IPv6 adds some new mobility features that make it possible for a device to retain a home address while moving between networks. Someday, maybe this won't be as much of an issue.

How to keep a persistent terminal open on the remote host

Here's two different approaches to keeping a session going when you're either moving between networks or wanting to disconnect for a while:

1. Use Mosh or Eternal Terminal

   If you really need a connection that does not go down even when you move between networks, use Mosh—the mobile shell. Mosh is a secure shell that uses SSH for the handshake, then switches to its own encrypted channel for the session. This separate channel is very resilient. It can handle a dropped internet connection, changes to your laptop's IP address, major network lag, and more, thanks to the magic of UDP and the synchronization protocol that Mosh uses.

   To use Mosh, you'll need to install it on both your client and server and open up ports 60000-61000 for inbound UDP traffic to your remote host. Then just mosh user@server to connect.

   Mosh operates at the level of screens and keystrokes, and this gives it a lot of benefits over SSH, which ferries the binary stream of standard input and output between the client and server. If we only have to synchronize screens and keystrokes, then an interrupted connection becomes much quicker to resume later. SSH would have to buffer and send everything that happened, but Mosh only needs to buffer keystrokes and synchronize the latest frame of the terminal window with the client.

2. Use tmux

   If you'd like to "come and go as you please" and keep the same terminal session going on a remote host, use the terminal multiplexer tmux. I love tmux and I use it all the time. If your SSH connection dies, just reconnect and type tmux attach to return to your tmux session. It also has some great features, like in-terminal tabs and panes that are similar to the tabs in macOS Terminal, and terminals that can be shared with others.

   Some folks like tricking out their tmux with Byobu, a package which provides a lot of usability enhancements and key bindings for tmux. Byobu ships with Ubuntu, and is easy to install on a Mac with Homebrew.

Share a remote terminal session with a friend

When debugging a complex issue on your servers, you might like to share an SSH session with someone who is not in the same room. tmux is perfect for this for terminal sharing! Here are the steps:

1. Make sure tmux is installed on your bastion, or on whatever server you want to use.
2. Both of you will need to SSH to the machine using the same account.
3. One person should run tmux to start the tmux session
4. The other person should run tmux attach
5. Voila! You're sharing a terminal.

If you want more sophisticated multi-user tmux sharing, there's tmate, which is a fork of tmux that makes shared terminal sessions a lot easier.

What are your favorite SSH tips & tricks?

Hit us up on Twitter!—we'd love to expand this post with your best suggestions for using SSH.