How to Handle Secrets on the Command Line
Carl Tashian
The command line really wasn't designed for secrets. So, keeping secrets secret on the command line requires some extra care and effort. The other day in my homelab I was configuring a TLS client certificate for a Grafana datasource. The intention was to write something I could run on a timer whenever the certificate is renewed. The command needed to:
- Build some JSON with the renewed certificate and private key injected into it
- PUT it to Grafana's API to update the datasource configuration.
I thought I was being very clever when I wrote this lil' Bash pipeline:
BEARER_TOKEN=MhY3b3i3gFpa9otnLQVznJYoWLxpGJUod3iDJwCKRFUVtuALGJooBJuCUf7w9HJfbu; jq -n --arg ca_cert "$(< $STEPPATH/certs/root_ca.crt)" --arg client_cert "$(< $CERT_LOCATION)" --arg client_key "$(< $KEY_LOCATION)" -f ./datasource.jq | curl -s -X PUT -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d "$(< /dev/stdin)" --cacert $STEPPATH/certs/root_ca.crt https://grafana:3000/api/datasources/2
This uses jq
to populate a JSON template for an API request (datasource.jq
) with variables passed in with --arg
, and then PUTs to the API with curl
. Nice, huh?
It didn't take long to discover that this pipeline leaks secrets all over the place:
- The private key credential file is leaked:
"$(< $KEY_LOCATION)"
- The bearer token env variable is leaked:
-H "Authorization: Bearer $BEARER_TOKEN"
- And the secret piped data from
STDIN
is leaked:-d "$(< /dev/stdin)"
All of these values, including the precious contents of the private key file, can be seen via ps
when these commands are running. ps
finds them via /proc/<pid>/cmdline
, which is globally readable for any process ID.
To make atonement, I'm writing this post. We'll look at three methods for handling secrets on the command line: Using piped data, credential files, and environment variables. We'll look at some of the risks of these approaches, and how to use each of them as safely as possible.
But first, let's look at a sanitized version of the above pipeline:
jq -n --rawfile ca_cert $STEPPATH/certs/root_ca.crt --rawfile client_cert $CERT_LOCATION --rawfile client_key $KEY_LOCATION -f ./datasource.jq | curl -s -X PUT -H @api_headers -d @- --cacert $STEPPATH/certs/root_ca.crt https://grafana:3000/api/datasources/2
Notably:
- The
--rawfile
flag injq
moves the credentials closer to where they are used by delegating the responsibility for reading the certificate data from the files tojq
instead ofbash
. - Once
jq
pushes the secret JSON into the pipe,curl
uses the-d @-
flag to pull the secret data directly from the pipe and use it as the HTTPS request body. - And
-H @api_headers
will read the static bearer token from a file (api_headers
).
Now, no secrets will appear in ps
—only filenames.
Piped Secrets
As the sanitized example shows, a pipeline is generally an excellent way to pass secrets around, if the program you're using will accept a secret via STDIN
. Because a pipe only has two ends, right? Imagine yourself whispering a secret into one end of a pipe, and a friend putting their ear up to the other. It's just like that.
Usually. The $(< /dev/stdin)
leak shown above uses a neat Bash substitution to make an otherwise secure pipe insecure. For example, if you run:
$ echo "secret-data" | curl -d "$(</dev/stdin)" https://example.com:3000
Then the output of ps
for curl
will show:
curl -d secret-data https://example.com:3000
Other than that, there's not too much to worry about with pipes.
Credentials Files
What's not to love about a file? It's got an owner. It has permissions and access control. Give each secret a file! Any program that accepts secrets should be able to accept them by passing a filename or by redirecting a file into STDIN
. You can also use files to pass secrets into Docker containers with mounted volumes.
A few notes about storing and retrieving file secrets:
- You'd better get the permissions right
- Avoid leaking the secret in the command string eg. with
"$(< secret_file.txt)"
- Be sure your disk is encrypted at rest, eg. with LUKS
- You may want to encrypt the contents of the file — but, then you need to figure out how to handle the encryption key.
Environment Variables
Using environment variables for secrets is very convenient. And we don't recommend it because it's so easy to leak things:
- Some operating systems still make every process's environment variables world readable. (But, in all the Linuxes I've seen,
/proc/<pid>/environ
is not world-readable.) - In Docker, anyone with access to the Docker daemon can use
docker inspect
to see all of the environment variables for any running container. - In systemd, environment variables in unit files are available to users via the dbus interface (see the recent introduction of
LoadCredential=
for an alternative that uses credential files) - Exported environment variables will get passed to every new process, and then who knows what will happen to them. They might get dumped to
STDOUT
or logged to a debug logfile. - Local (unexported) shell variables are also easy to leak into
ps
output:$ BEARER_TOKEN=MhY3b3i3gFpa9otnLQVznJYoWLxpGJUod3iDJwCKRFUVtuALGJooBJuCUf7w9HJfbu; curl -s -X PUT -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" ...
- Variables can easily end up in shell history. In many shells, adding an extra space before a command will exclude it from shell history.
- In Bash, the
HISTCONTROL
variable must be set toignorespace
. - In the Fish shell, run
fish --private
to start a private mode Fish session.
- In Bash, the
- Secrets can also slip into your editor's search history. For example, in
vim
there's a$HOME/.viminfo
file that contains search history. Runvim -i NONE
to disable it for a session.
If a command takes a password filename, you can use a password environment variable if you're careful about it. For example, say we have a $STEP_CA_PASSWORD
environment variable, and we run the following in Bash:
$ STEP_CA_PASSWORD=amazingpw; step-ca --password-file <(echo -n "$STEP_CA_PASSWORD") $(step path)/config/ca.json
The /proc/<pid>/cmdline
for this process will contain something like:
step-ca --password-file /dev/fd/11 /home/carl/.step/config/ca.json
Using this <()
syntax of process substitution, Bash will create a file using the output of echo -n "$STEP_CA_PASSWORD"
and supply that file's name to --password-file
.
Great, right? Except...wouldn't the /proc/<pid>/cmdline
for the inner echo
contain:
echo -n my-secret-ca-password
Thankfully, in Bash, the answer is no. We are saved by the fact that echo
is a builtin, and a process will never be created. So, if you are able to otherwise secure your environment variables, this approach is safe. The downside is that it appears unsafe, because $STEP_CA_PASSWORD
is still getting substituted into something that certainly looks like it's a command.
What About A Secrets Manager?
Secrets managers can be great because they can make it easier to get secrets closer to where they are used. For example, a Docker container can call out to a secrets manager for its secrets. But, a secrets manager is an extra dependency. Often you need to run a secrets manager server and hit an API. And even with a secrets manager, you may still need Bash to shuttle the secret into your target application. For this post I'm focused on more lightweight solutions.
Speaking of lightweight solutions,
there is an keyring facility in the Linux kernel.
The Linux keyring offers several scopes for storing keys safely in memory that will never be swapped to disk.
A process or even a single thread can have its own keyring,
or you can have a keyring that is inherited across all processes in a user's session.
To manage the keyrings and keys,
use the keyctl
command or keyctl
system calls.
Directly in the command
In case it isn't already abundantly clear, this is very unsafe. There is no way for the caller of a command to choose to hide the command line from being world readable. So, any CLI command worth its salt should not accept passwords directly.
You could maybe pass a password directly in a command if you were running inside a secure container with sandboxing around everything. But, why take the risk?
Now, even for this one there's a caveat. Have you ever run mysql
this way?
$ mysql --user carl --password amazingpw db.smallstep.com
Or curl
this way?
$ curl -u carl:password https://example.com:3000
These commands accept passwords against their own better judgement, for convenience. But, immediately upon startup, they will overwrite argv
with a blank value, effectively hiding the secret. If you run ps
during the curl
command shown here, you'll see:
curl -u https://example.com:3000
Now technically, if a system is overloaded enough, it could be possible to grab the secret from /proc/<pid>/cmdline
before curl
has a chance to overwrite it. But, this approach is better than nothing at all. And these passwords can easily end up in audit logs or shell history. So, better to avoid this approach entirely.
The alternative for curl
is a credential file: A .netrc
file can be used to store credentials for servers you need to connect to.
And for mysql
, you can create option files: a .my.cnf
or an obfuscated .mylogin.cnf
will be read on startup and can contain your passwords.
Subscribe to updates
Unsubscribe anytime, see Privacy Policy
Carl Tashian (Website, LinkedIn) is an engineer, writer, exec coach, and startup all-rounder. He's currently an Offroad Engineer at Smallstep. He co-founded and built the engineering team at Trove, and he wrote the code that opens your Zipcar. He lives in San Francisco with his wife Siobhan and he loves to play the modular synthesizer 🎛️🎚️