The magic of systemd-creds
Carl Tashian
TL;DR The systemd suite's new systemd-creds feature fills an important security gap: It allows you to protect service credentials using a TPM, rather than storing them on disk in plaintext.
This is perfect for storing a service's private keys, TLS certificates, account credentials, or passwords!
But there are several subtleties around how to set it up and use it securely.
In this tutorial, we'll use systemd-creds to secure the CA password for step-ca Certificate Authority server on a Google Cloud VM.
Background
Like the Secure Enclave on Apple devices, Trusted Platform Module (TPM) chips protect private keys at the hardware level. And the TPM 2.0 interface is now pretty much ubiquitous in server hardware and cloud VMs. Google Cloud rolled out virtual TPMs as part of its Shielded VM features in 2018, and since 2020 vTPMs have been enabled by default. In May, Amazon rolled out its NitroTPM support for EC2 instances. TPMs are everywhere!
In Linux, the systemd suite has added a lot of TPM2 support over the past year or so. Services managed by systemd can now leverage a TPM for protecting credentials, bind them to your specific device and software configuration. You can encrypt certificates, passwords, PEM keys, or any other sensitive material needed to start up a server.
So, let's use this feature to encrypt a CA password for the step-ca online Certificate Authority daemon.
Prerequisites
For this project you will need:
-
A Linux machine (or virtual environment) with TPM 2.0 support:
$ sudo dmesg | grep TPM [ 0.018942] ACPI: TPM2 0x00000000BEB7B000 000034 (v04 GOOGLE 00000001 GOOG 00000001) [ 0.018968] ACPI: Reserving TPM2 table memory at [mem 0xbeb7b000-0xbeb7b033] [ 3.676084] systemd[1]: systemd 251.2-1-arch running in system mode (+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified) [ 6.842149] tpm_tis MSFT0101:00: 2.0 TPM (device-id 0x9009, rev-id 0)
-
For this tutorial, I'll be using a Google Cloud Platform account and project. To follow along, you'll need the
gcloudcommand on your local computer (brew install google-cloud-sdk) that you've authenticated to your Google Cloud Platform account. -
You'll need systemd 250+, preferably 251. If you're running systemd 251, you can confirm your system is ready to use systemd credentials by running
systemd-creds has-tpm2. It should return a simpleyes. -
For production, use an environment with UEFI Secure Boot enabled:
$ sudo dmesg | grep Secure [ 0.000000] secureboot: Secure boot enabled [ 0.000000] Kernel is locked down from EFI Secure Boot mode; see man kernel_lockdown.7 [ 0.023818] secureboot: Secure boot enabled
(For testing, it's OK if secure boot isn't enabled.)
-
For production, use an encrypted
/var. By defaultsystemd-credssplits its master private key across both the TPM and a file on disk, and requires both keys to decrypt a service credential.(Why not just use the TPM alone? You can. But in many cases an attacker could install the same bootloader and OS kernel on another volume on your machine, and gain access to the TPM key.)
Diving In
So that anyone can try this out, we opted to use Google Cloud for this example.
Unfortunately I wasn't able to find a cloud VM image that met all of these criteria (as of June 2022, systemd 251 is still very new), so I opted for Arch Linux without UEFI Secure Boot support. I wouldn't use this in production, but for the purpose of this tutorial it's okay; the instructions are the same.
1. Create an Arch Linux VM.
Let's make an e2-medium VM that has vTPM support enabled:
2. Connect, update the system, and restart
After it restarts, reconnect. You should see that TPM 2.0 is enabled and detected, and that you're running systemd 251 or higher:
That's great. But, is systemd ready to use its TPM support? Run:
sudo systemd-creds has-tpm2
Output:
Without Secure Boot, we have only "partial" TPM2 support here. We'll proceed with partial support, but in a production environment you'd want to confirm secure boot and full TPM2 support.
3. Configure systemd-creds
We need to set up the host credentials for systemd-creds in order to use it. Run:
Output:
In this step, systemd-creds creates a key that's split between the TPM and the filesystem.
This file will indeed be encrypted at rest by GCP, but Linux can't see that.
3. Install dependencies.
The tpm2-abrmd service is needed.
This service is the TPM2 access broker.
It allows different applications to share the TPM device.
4. Install the step and step-ca binaries.
5. Initialize your CA
To initialize your CA, run the following.
Change CA_NAME, CA_HOSTNAMES (the DNS names your CA will answer on), and EMAIL to match your local setup.
export STEPPATH=/etc/step-ca < /dev/urandom tr -dc A-Za-z0-9 | head -c40 > password.txt CA_NAME="Smallstep" CA_HOSTNAMES="127.0.0.1,localhost" EMAIL="carl@smallstep.com" sudo -E step ca init --ssh --name "$CA_NAME" \ --dns "$CA_HOSTNAMES" \ --address ":443" --provisioner "$EMAIL" \ --password-file "password.txt" sudo useradd --system --home /etc/step-ca --shell /bin/false step sudo chown -R step:step /etc/step-ca
6. Add a systemd service for your CA.
We've created a service unit for step-ca, which you can download:
sudo curl -sL https://raw.githubusercontent.com/smallstep/certificates/master/systemd/step-ca.service \ -o /etc/systemd/system/step-ca.service
7. Encrypt your CA credentials.
In this step we're going to make a drop-in override for the step-ca.service.
The override unit will contain our encrypted credential.
sudo mkdir /etc/systemd/system/step-ca.service.d cat<<EOF | sudo tee /etc/systemd/system/step-ca.service.d/overrides.conf [Unit] ConditionFileNotEmpty= [Service] ExecStart= ExecStart=/usr/bin/step-ca config/ca.json --password-file \${CREDENTIALS_DIRECTORY}/step-ca EOF sudo systemd-creds encrypt --name=step-ca -p password.txt - | sudo tee -a /etc/systemd/system/step-ca.service.d/overrides.conf sudo systemctl daemon-reload
8. Start step-ca.service
The service will start up and read its credentials from the TPM.
$ sudo systemctl start step-ca $ sudo systemctl status step-ca ● step-ca.service - step-ca service Loaded: loaded (/etc/systemd/system/step-ca.service; disabled; vendor preset: disabled) Drop-In: /etc/systemd/system/step-ca.service.d └─overrides.conf Active: active (running) since Wed 2022-06-15 23:33:30 UTC; 2s ago Docs: https://smallstep.com/docs/step-ca https://smallstep.com/docs/step-ca/certificate-authority-server-production Main PID: 2176 (step-ca) Tasks: 7 (limit: 4694) Memory: 19.2M CPU: 133ms CGroup: /system.slice/step-ca.service └─2176 /usr/bin/step-ca config/ca.json --password-file /run/credentials/step-ca.service/step-ca Jun 15 23:33:31 arch-test step-ca[2176]: badger 2022/06/15 23:33:31 INFO: All 0 tables opened in 0s Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 Starting Smallstep CA/0.20.0 (linux/amd64) Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 Documentation: https://u.step.sm/docs/ca Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 Community Discord: https://u.step.sm/discord Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 Config file: config/ca.json Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 The primary server URL is https://127.0.0.1:443 Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 Root certificates are available at https://127.0.0.1:443/roots.pem Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 Additional configured hostnames: localhost Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 X.509 Root Fingerprint: c8719fad9b7639dbc0b78fdc92def5b4b7c83725ac4a84d2fbdc271c991d8f Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 Serving HTTPS on :443 ...
9. Finishing up
Finally, make an offline copy of the contents of password.txt.
Once you have, you can delete the plaintext file:
A side note on PCRs
Platform Configuration Registers (PCRs) are system integrity registers, consisting of 24 protected hashes stored in a TPM.
PCRs are zeroed out when the system boots, and only operation that can be performed on a PCR is "extend." Extending is a one-way function that appends a passed-in hash to the PCR's current hash, hashes the result, and replaces the PCR with it.
PCRs record system properties into well-known PCRs, mostly during boot time. Values such as the hardware configuration, hashes of the machine's trusted firmware, and the UEFI code signing certificate database (things like the Machine Owner Key other key credentials needed for proof of a Secure Boot).
PCRs merit a deeper discussion beyond the scope of this tutorial.
But, for the purpose of systemd-creds, the encryption key is bound to PCR 7 by default.
You may want to bind it to PCRs 7 and 14, using --tpm2-pcrs=7+14 when running systemd-creds setup.
By binding your key to TPM PCRs 7 and 14, you are trusting the kernel binaries provided by your OS distro. An attacker could install the same OS on another volume and potentially unseal your systemd-creds TPM-stored key. This is why the master credential key for systemd-creds is split across the TPM and disk by default.
For a list of well-known PCR values in Linux, see the shim bootloader README.tpm file
and the docs for systemd-cryptenroll
Also check out Lennart Poettering's blog post authenticated boot and disk encryption for his take on the brittleness of PCRs and why you should choose carefully.
If you really want to lock things down, you could create your own code signing CA and add it to the certificate database in the Linux shim bootloader, and then sign your own kernel images. But that's a lot of work.
Other important caveats
-
Because our systemd credential is encrypted, it's okay to store it in a world-readable systemd unit file. For production, you can store encrypted credentials in the credentials directory, outside of unit files and with tighter permissions.
-
When the service is running, the credentials live on disk so they can be read by the service. The systemd service should use mount namespacing so that the credentials directory is only visible to that service. Mount namespacing is the same mechanism containers use to keep their filesystems isolated from the host environment. Use
PrivateMounts=trueto enable mount namespacing for a service.
Sources:
Carl Tashian (Website, LinkedIn) is an engineer, writer, exec coach, and startup all-rounder. He's currently an Offroad Engineer at Smallstep. He co-founded and built the engineering team at Trove, and he wrote the code that opens your Zipcar. He lives in San Francisco with his wife Siobhan and he loves to play the modular synthesizer 🎛️🎚️
