Smallstep SSH

Host Quickstart

Before you begin

  • You'll need a account on the smallstep platform. Need one? Register here
  • We support the following host platforms:
    • Ubuntu 18.04 LTS
    • CentOS 7 and 8
    • Debian 10
  • Running this quickstart will modify config files related to systemd, PAM, NSS, and SSHD.
  • Specifically, we modify the following files:
    • /etc/pam.d/sshd
    • /etc/pam.d/sudo
    • /etc/pam.d/su
    • /etc/ssh/sshd_config
    • /etc/nsswitch.conf
  • We suggest backing these up before you begin setup on a particular base machine type, so that you can quickly revert changes if needed.

Features

The following registration features are supported:

  • New Host Registration
    • Configure Linux hosts to be managed by Smallstep SSH
  • Host Tags Support
    • Use tags to identify hosts and apply access control rules

Step-by-Step Instructions

Step 0: Understanding Host Tags

Host Tags are used to create logical groups of hosts to simplify access control. Rather than granting groups of users access host-by-host, you can grant access to a tag combination. All hosts with that tag combination are automatically added as part of the host grant.

Let's look at an example:

  • A host tag combination of db: dev gives you access to myserver #1.
  • A host tag combination of db: prod gives you access to myserver #2 and myserver #3.

Step 1. sudo su

Run the entire Host Configuration setup as the root user.

# sudo su

Step 2. Install step

# curl -L -o step https://files.smallstep.com/step-linux-0.14.4
# sudo install -m 0755 -t /usr/bin step

Step 3. Install step-ssh utilities

  • Install on Ubuntu 18.04

      # curl -LO https://files.smallstep.com/step-ssh_0.18.8-1_amd64.deb
      # dpkg -i step-ssh_0.18.8-1_amd64.deb
    

    Comment out the broken pam_systemd.so in common-session which is causing a delay at login (this is a known Ubuntu/Debian bug):

      # sed -e '/pam_systemd.so/s/^/#/g' -i /etc/pam.d/common-session
    

    step-ssh_0.18.8_amd64.deb installs modules and services.

  • Install on CentOS 7

      # curl -LO "https://files.smallstep.com/step-ssh-0.18.8-1.el7.x86_64.rpm"
      # yum -y install step-ssh-0.18.8-1.el7.x86_64.rpm
    

    step-ssh-0.18.8-1.el7.x86_64.rpm installs modules and services.

  • Install on CentOS 8

      # curl -LO "https://files.smallstep.com/step-ssh-0.18.8-1.el8.x86_64.rpm"
      # yum -y install step-ssh-0.18.8-1.el8.x86_64.rpm
    

    step-ssh-0.18.8-1.el8.x86_64.rpm installs modules and services.

Step 4. Configure step to connect to your CA

# step ca bootstrap --team="[your smallstep team name]"

Step 5. Get an SSH host certificate

For this step, you will need the contents of the enrollment token you received upon signup. If you downloaded it, the file will be called enrollment_token.

# export enrollment_token="[enrollment-token]"
# export hostname="[your-hostname]"
# step ssh certificate $hostname /etc/ssh/ssh_host_ecdsa_key.pub \
    --host --sign --provisioner "Service Account" --token $enrollment_token

The hostname used here must be the name that SSH clients use to connect to this host. If multiple names are used to SSH to a machine (e.g., a hostname and an IP address) you can pass each of them to step ssh certificate via the --principal flag, like:

# step ssh certificate $hostname /etc/ssh/ssh_host_ecdsa_key.pub \
    --host --sign --provisioner "Service Account" --token $enrollment_token \
    --principal $hostname --principal 10.0.0.42

Notice that hostname needs to be passed as an additional --principal explicitly when multiple principals are passed.

Step 6. Configure OpenSSH to use SSH certificate authentication

# step ssh config --host --set Certificate=ssh_host_ecdsa_key-cert.pub \
                         --set Key=ssh_host_ecdsa_key

This command will add a few lines of configuration to the end of your /etc/ssh/sshd_config to enable certificate authentication. These lines are annotated with a comment that says # autogenerated by step @ <timestamp> so you can identify them later if you need to modify or revert these changes.

Step 7. Activate PAM/NSS Modules & HUP SSHD

The commands in step 6 must be run as sudo.

# step-ssh activate "$hostname"

The step-ssh activate command will leverage a short-lived identity certificate to authenticate itself to the host inventory.

Step 8. Register the host and add tags(s)

# step-ssh-ctl register --tag <key=value> --tag <role=web> --hostname "$hostname"

Hosts can be added to multiple tags via the --tag flag. If the host is a bastion, add the --is-bastion flag. If the host is behind a bastion, add the --bastion <bastion> flag, where <bastion> is the hostname of the bastion (the <bastion> must already be running and registered as a bastion for this to work). This command will also leverage the host identity certificate to authenticate itself to the host inventory. It is possible to rerun this command multiple times for, e.g., renaming the host or replacing its tags. Please note a rerun of the register command will replace existing tags & bastion information.

Step 9. Sign in to the smallstep UI

Sign in at https://smallstep.com/app/[TEAM NAME]

You should see the host(s) you just added, under “Hosts".

Need to remove a host?

On the host, run:

# step-ssh-ctl unregister

This will remove the host from the host inventory on the CA, using the host identity certificate.

Troubleshooting Tips

  • Smallstep uses systemd, PAM, and NSS. If you've made non-standard changes to your PAM or NSS stacks, we recommend that you run step-ssh activate as root and do not log out of your root shell until you've tested your installation (e.g., by logging in and running sudo in a separate terminal).
  • Having trouble? You should be able to revert any changes by running step-ssh deactivate.
  • Suspect host or user certificates are not working? In other words, your ssh client fails to log in or shows “trust on first use” warning? Try this.
    • On your client run step ssh list and find your user certificate marked (ECDSA-CERT), e.g. 256 SHA256:Bb2TcimUYj8Nc5w4FhpZ/gmeNIIvLIzphTx35NzaRoA johndoe@smallstep.com (ECDSA-CERT), which you can inspect with step ssh list --raw johndoe@smallstep.com | step ssh inspect. Be sure the current time is neither before or after the period specified in Valid: .

    • On the destination host run step ssh inspect /etc/ssh/ssh_host_ecdsa_key-cert.pub. Make sure the current time is within the period of Valid:.

    • The destination host's sshd_config should show included the following lines for sshd (make sure the service reloaded its config) to leverage certificate-based authentication:

      $ tail -n 7 /etc/ssh/sshd_config
      #       ForceCommand cvs server
      
      # autogenerated by step @ 2020-04-02T21:16:05Z
      TrustedUserCAKeys /etc/ssh/ca.pub
      HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub
      HostKey /etc/ssh/ssh_host_ecdsa_key
      # end
      
  • Suspect your ssh client might have a problem? Use verbose logging ssh -v <hostname>. Look out for following key lines/events below in your log. If you don't see errors the absence of these events might indicate your config is invalid / not being applied.
    • Your team's config is applied for this specific host:

      debug1: Reading configuration data /etc/ssh/ssh_config
      debug1: /etc/ssh/ssh_config line 48: Applying options for *
      debug1: /etc/ssh/ssh_config line 52: Applying options for *
      debug1: Executing proxy command: exec step ssh proxycommand --provisioner "okta" johndoe ec2-52-200-74-193.compute-1.amazonaws.com 22
      
    • The host certificate passed authentication on the client:

      debug1: Server host certificate: ecdsa-sha2-nistp256-cert-v01@openssh.com SHA256:46gC0CEzXWN4acTHGQldL6H+QlbhB4+KPZjkoRToI/w, serial 8551898981883739717 ID "ec2-52-200-74-193.compute-1.amazonaws.com" CA ecdsa-sha2-nistp256 SHA256:sqfZG6AOPUvcheFUIZDX+DEesnyfNZQ5JwqpcxUzY+0 valid from 2020-04-14T04:45:10 to 2020-05-14T04:46:10
      debug1: Host 'ec2-52-200-74-193.compute-1.amazonaws.com' is known and matches the ECDSA-CERT host certificate.
      debug1: Found CA key in /Users/johndoe/.step/ssh/known_hosts:1
      
    • The username being used for authentication:

      debug1: Authenticating to ec2-52-200-74-193.compute-1.amazonaws.com:22 as 'johndoe'
      
    • The ssh client offers the user's certificate to the destination host:

      debug1: Server accepts key: johndoe@smallstep.com ECDSA-CERT SHA256:VIa1uWhBTjjtpW3IBkUG/aFGfqlUhjkXNQVk6Hc1lXc agent
      debug1: sign_and_send_pubkey: no separate private key for certificate "johndoe@smallstep.com"