Smallstep SSH

Host Quickstart

Before you begin

  • You'll need a account on the smallstep platform. Need one? Register here
  • You'll need the enrollment token you received upon signup.
  • We support the following host platforms:
    • Ubuntu 18.04 LTS
    • CentOS 7 and 8
    • Debian 10
  • Running this quickstart will modify config files related to systemd, PAM, NSS, and SSHD:
    • We add step-ssh-renew and step-ssh-metadata services to systemd:
      • step-ssh-renew rotates the SSH host certificate every eight hours.
      • step-ssh-metadata syncs user ACLs with your CA every 3 seconds.
    • And we modify the following files:
      • /etc/pam.d/sshd
      • /etc/pam.d/sudo
      • /etc/pam.d/su
      • /etc/ssh/sshd_config
      • /etc/nsswitch.conf
    • We suggest backing these up before you begin setup on a particular base machine type, so that you can quickly revert changes if needed.

Step-by-Step Instructions

Step 0. Understanding Host Tags

This step only applies to multi-user environments.

Host Tags (key-value pairs) are the pillar our access control model. Rather than mapping people or groups directly to hosts, you'll map tag combinations to your hosts and to your user groups. First you'll put your hosts into logical groups using tags, eg. role:web or env:staging. Then, you'll grant user groups access to all hosts with a specific tag combination. Finally, you'll choose which user group tag combinations will allow sudo privileges on any matching hosts.

Let's look at an example:

  • The developers group will have access to myserver #1 only.
  • The data group will have access to myserver #2 and myserver #3.
  • The ops group will have sudo access to myserver #2 and myserver #3.

Of course, hosts and groups can have as many tag combinations as you like. Take a minute to think about how you'd like to use Host Tags in your environment.

Step 1. Run all steps as root

Now let's set up your host. You'll run the entire Host Configuration setup as the root user:

sudo su

Step 2. Install step CLI tool

curl -L -o step https://files.smallstep.com/step-linux-0.14.6
install -m 0755 -t /usr/bin step

Step 3. Install the step-ssh utilities

This step will install modules and services.

  • Install on Ubuntu 18.04

      curl -LO https://files.smallstep.com/step-ssh_0.18.8-1_amd64.deb
      dpkg -i step-ssh_0.18.8-1_amd64.deb
    

    Comment out the broken pam_systemd.so in common-session, which causes a delay at login (this is a known Ubuntu/Debian bug):

      sed -e '/pam_systemd.so/s/^/#/g' -i /etc/pam.d/common-session
    
  • Install on CentOS 7

      curl -LO "https://files.smallstep.com/step-ssh-0.18.8-1.el7.x86_64.rpm"
      # yum -y install step-ssh-0.18.8-1.el7.x86_64.rpm
    
  • Install on CentOS 8

      # curl -LO "https://files.smallstep.com/step-ssh-0.18.8-1.el8.x86_64.rpm"
      yum -y install step-ssh-0.18.8-1.el8.x86_64.rpm
    

Step 4. Configure step to connect to your CA

step ca bootstrap --team="[your smallstep team ID]"

Step 5. Get an SSH host certificate

Remember the enrollment token you got when you signed up? You'll need it now. If you downloaded it, the file is called enrollment_token.

_👇 This leading space will (usually) keep the token out of your shell's history.

  export enrollment_token="[your enrollment token]"
 export hostname="[your hostname]"

The hostname is your host's canonical hostname or IP. This will be the name clients use to SSH to this host.

For hosts with a single hostname

Run the following to issue a certificate for your host:

step ssh certificate $hostname /etc/ssh/ssh_host_ecdsa_key.pub \
    --host --sign --provisioner "Service Account" --token $enrollment_token

If a host has multiple hostnames…

Note: When a host has multiple hostnames, your users will only be able to ssh to the canonical $hostname, as shown by the step ssh hosts command.

If you need multiple hostnames in your host certificate (e.g., public and private hostnames, or a hostname and an IP address), you can pass each of them to step ssh certificate via the --principal flag:

step ssh certificate $hostname /etc/ssh/ssh_host_ecdsa_key.pub \
    --host --sign --provisioner "Service Account" --token $enrollment_token \
    --principal $hostname --principal 10.0.0.42

When multiple hostnames are needed, the canonical $hostname must be passed twice: Once to establish the certificate's Key ID, and again explicity as its first Principal.

Step 6. Configure SSHD to use certificate authentication

step ssh config --host --set Certificate=ssh_host_ecdsa_key-cert.pub \
                         --set Key=ssh_host_ecdsa_key

This command will add a few lines of configuration to the end of your /etc/ssh/sshd_config to enable certificate authentication. These lines are annotated with a comment that says # autogenerated by step @ <timestamp> so you can identify them later if you need to modify or revert these changes.

Step 7. Activate PAM/NSS Modules & HUP SSHD

step-ssh activate "$hostname"

The step-ssh activate command will leverage a short-lived identity certificate to authenticate itself to the host inventory.

Step 8. Register the host and add tags(s)

This command will leverage the host identity certificate to authenticate itself to the host inventory.

step-ssh-ctl register --hostname "$hostname"

Registering a host with host tags

For access control in multi-user environments, host tags can be assigned via the --tag flag.

step-ssh-ctl register --tag <key=value> --tag <role=web> --hostname "$hostname"

It is possible to rerun step-ssh-ctl register multiple times, to rename the host, replace its tags, or change the bastion settings. Note: This command replaces all existing tags and bastion settings for a host.

Registering a bastion host (jump boxes)

If the host you're registering is a bastion, add the --is-bastion flag:

step-ssh-ctl register --hostname "$hostname" --is-bastion

Note: Your bastion host will need the nc command installed. Our bastion host support uses nc (along with the ProxyCommand directive) because it's widely compatible with older SSHD servers.

Registering a host behind a bastion

If the host you're registering is behind a bastion, add the --bastion flag:

step-ssh-ctl register --hostname "$hostname" --bastion "[bastion hostname]"

Step 9. Test your installation

Before you sign out of your sudo session, test your installation by logging in and running sudo in a separate session.

This step is especially important if you have made any non-standard changes to your PAM or NSS stacks.

Now sign in at https://smallstep.com/app/[Team ID]

You should see your host listed under the “Hosts” tab.

Need to remove a host?

On the host, run:

step-ssh-ctl unregister

This will remove the host from the host inventory on the CA, using the host identity certificate.

Optional Hardening

  • Now that you're using certificates, you may wish to explicitly disallow the use of authorized_keys files on the host.

    You can do this by setting AuthorizedKeysFile none in sshd_config. You may want to allow authorized_keys for an emergency access account, however. A configuration for this might look like:

     Match User *,!ubuntu
          AuthorizedKeysFile none
    

Troubleshooting Tips

  • Having trouble? You should be able to revert any changes by running step-ssh deactivate.
  • Suspect host or user certificates are not working? In other words, your ssh client fails to log in or shows “trust on first use” warning? Try this:
    • Be sure you can connect to our test server, with ssh ssh-test.smallstep.com.

    • On your client run step ssh list and find your user certificate marked (ECDSA-CERT), e.g. 256 SHA256:Bb2TcimUYj8Nc5w4FhpZ/gmeNIIvLIzphTx35NzaRoA alice@smallstep.com (ECDSA-CERT), which you can inspect with step ssh list --raw alice@smallstep.com | step ssh inspect. Be sure the current time is neither before or after the period specified in Valid: .

    • On the destination host run step ssh inspect /etc/ssh/ssh_host_ecdsa_key-cert.pub. Make sure the current time is within the period of Valid:.

    • The destination host's sshd_config should show included the following lines for sshd (make sure the service has reloaded its config) to leverage certificate-based authentication:

      $ tail -n 7 /etc/ssh/sshd_config
      #       ForceCommand cvs server
      
      # autogenerated by step @ 2020-04-02T21:16:05Z
      TrustedUserCAKeys /etc/ssh/ca.pub
      HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub
      HostKey /etc/ssh/ssh_host_ecdsa_key
      # end
      
    • You can run sshd -t (as root) to test you SSHD configuration. No output means the file is OK; you'll get an error if any of the referenced files (eg. host keys) are not accessible to sshd.

  • Suspect your ssh client might have a problem? Use verbose logging ssh -v <hostname>. Look out for following key lines/events below in your log. If you don't see errors the absence of these events might indicate your config is invalid / not being applied.
    • Your team's config is applied for this specific host:

      debug1: Reading configuration data /etc/ssh/ssh_config
      debug1: /etc/ssh/ssh_config line 48: Applying options for *
      debug1: /etc/ssh/ssh_config line 52: Applying options for *
      debug1: Executing proxy command: exec step ssh proxycommand --provisioner "okta" alice ec2-52-200-74-193.compute-1.amazonaws.com 22
      
    • The host certificate passed authentication on the client:

      debug1: Server host certificate: ecdsa-sha2-nistp256-cert-v01@openssh.com SHA256:46gC0CEzXWN4acTHGQldL6H+QlbhB4+KPZjkoRToI/w, serial 8551898981883739717 ID "ec2-52-200-74-193.compute-1.amazonaws.com" CA ecdsa-sha2-nistp256 SHA256:sqfZG6AOPUvcheFUIZDX+DEesnyfNZQ5JwqpcxUzY+0 valid from 2020-04-14T04:45:10 to 2020-05-14T04:46:10
      debug1: Host 'ec2-52-200-74-193.compute-1.amazonaws.com' is known and matches the ECDSA-CERT host certificate.
      debug1: Found CA key in /Users/alice/.step/ssh/known_hosts:1
      
    • The username being used for authentication:

      debug1: Authenticating to ec2-52-200-74-193.compute-1.amazonaws.com:22 as 'alice'
      
    • The ssh client offers the user's certificate to the destination host:

      debug1: Server accepts key: alice@smallstep.com ECDSA-CERT SHA256:VIa1uWhBTjjtpW3IBkUG/aFGfqlUhjkXNQVk6Hc1lXc agent
      debug1: sign_and_send_pubkey: no separate private key for certificate "alice@smallstep.com"