Written December 15, 2021
Zero Trust or BeyondProd approaches require authenticated and encrypted communications everywhere. TLS is the cryptographic protocol that powers encryption for all your technologies. For TLS, you need certificates. This practitioner's tutorial provides instructions for automating Istio TLS certificate renewal and enabling server-side encryption.
To have Istio mesh running TLS on a Kubernetes cluster, you will need to be able to request a certificate issued by a trusted certificate authority (CA). If you already have a private CA and root certificate, you can skip to the automated renewal section below. If you need to create a CA, you can:
Request a copy of your CA root certificate, which will be used to make sure each application can trust certificates presented by other applications.
step ca root ca.crt
Your certificate will be saved in
Smallstep CAs use provisioners to authenticate certificate requests using passwords, one-time tokens, single sign-on, and a variety of other mechanisms.
stepCLI and do not require a local network agent. The instructions below focus on the JWK provisioner, but can be repurposed with small tweaks to operationalize all non-ACME provisioners.
To learn more, see Configuring
The right provisioner depends on your operational environment.
The JWK provisioner is the most general-purpose provisioner. It supports password and one-time token-based authentication. To add a JWK provisioner called
istio to a hosted Certificate Manager authority (if you haven't already), run:
step beta ca provisioner add istio --type JWK --create --x509-default-dur 720h
For instructions on adding provisioners to open source
step-ca, or to learn more about other provisioner types, see Configuring
There are various approaches for configuring TLS certificate renewal in Kubernetes. For this tutorial, we will assume you are using Helm and will manage certificate renewal using the open source cert-manager project. You may repurpose various pieces for your own configuration by following the cert-manager or Istio documentation.
First, install cert-manager in your cluster with Helm:
helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set installCRDs=true
In cert-manager, a custom resource type
each TLS certififcate and specifies how to store it.
Create a new
myserver-certificate.yaml for your Istio server. Make sure you specify the same DNS name configured above, and keep track of the Kubernetes secret name where cert-manager will store your certificate and private key.
apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: myserver spec: secretName: myserver-tls duration: 720h dnsNames: - myserver.example.net issuerRef: name: my-ca kind: StepIssuer group: certmanager.step.sm
secretName, and issuer name to suit your own naming style in your Kubernetes cluster, then apply it with
kubectl apply -f myserver-certificate.yaml
This configuration specifies that cert-manager should issue and renew a TLS certificate with the DNS name
myserver.example.net and store the certificate and private key in a Kubernetes secret named
myserver-tls. The certificate is valid for 720 hours, and cert-manager will automatically renew it before expiration and update the
You'll also notice the above
Certificate resource has an
specifying a cert-manager
StepIssuer resource named
Before cert-manager knows how to use your CA
to issue and renew your certificate, we'll need to create
StepIssuer resource and configure it to point to your private CA.
First, let's grab the
name and key
kid from the JWK provisioner we created earlier.
step ca provisioner list
Next, get a base64-encoded version of your CA root certificate:
step ca root -f >(step base64) 2> /dev/null
my-ca-issuer.yaml and fill in the CA URL and provisioner details
with your own.
caBundle refers to the base64-encoded version
of your root certificate.
apiVersion: certmanager.step.sm/v1beta1 kind: StepIssuer metadata: name: my-ca namespace: default spec: url: https://my-ca-url.my-team.ca.smallstep.com caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJpekNDQVRHZ0F3SUJBZ0lRTytFQWg4eS8wVjlQMFhwSHJWajVOVEFLQmdncWhrak9QUVFEQWpBa01TSXcKSUFZRFZRUURFeGxUZEdWd0lFTmxjblJwWm1sallYUmxjeUJTYjI5MElFTkJNQjRYRFRFNU1EZ3hNekU1TVRVdwpNbG9YRFRJNU1EZ3hNREU1TVRVd01sb3dKREVpTUNBR0ExVUVBeE1aVTNSbGNDQkRaWEowYVdacFkyRjBaWE1nClVtOXZkQ0JEUVRCWk1CTUdCeXFHU000OUFnRUdDQ3FHU000OUF3RUhBMElBQkFNVkw3VzBQbTNvSlVmSTR3WGQKa2xERW5uNVhTbWo4NlgwYW1DQTBnY08xdElUUG1DVzNCcGU0cE9vV1V2WlZlUWRvU2NxN3pua1V0Mi9HMnQxTgo3MWlqUlRCRE1BNEdBMVVkRHdFQi93UUVBd0lCQmpBU0JnTlZIUk1CQWY4RUNEQUdBUUgvQWdFQk1CMEdBMVVkCkRnUVdCQlJ1Y1ByVm5QdlpOMHI0QVU5TGcyL2VCcng3a2pBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCUlJBdGsKNXpMY0doQ2FobVBuVzIwZExpdEMzRVdNaVE0bERwN2FFeitFUEFJaEFJOWZWczVxb0l0bVQ4anA2WktVNVEydQphRFBrOGsyQ25OMjdyRnNZV3VwTAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== provisioner: name: istio kid: N6I99Yuk7iGDMk_eW3QaN2admCsrC9UuDN27dlFXUOs passwordRef: name: my-ca-provisioner-password key: password
You'll notice the
passwordRef configuration above, which declares a secret
where step-issuer can expect to find your JWK provisioner password.
You can name this whatever you'd like.
Create the secret in your cluster, then apply your
kubectl create secret generic my-ca-provisioner-password --from-literal=password=Y4nys7f11 kubectl apply -f my-ca-issuer.yaml
If all went as expected now that both your
Certificate and your
StepIssuer are configured, cert-manager should have reached out to your Smallstep CA, issued (or begun issuing) the certificate, and created the
myserver-tls secret with your certificate and private key.
ValidatingWebhookConfiguration to make use of the certificate and private key
myserver-tls secret is as easy as adding a CA Injector annotation
to your exsting webhook resources. Notice that we can obmit the
caBundle in the
clientConfig because of the new
apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: name: smallstep-mutating-webhooks annotations: cert-manager.io/inject-ca-from: myserver-tls ... webhooks: - name: my-webhook.example.com clientConfig: service: namespace: my-service-namespace name: my-service-name path: /my-path port: 1234 ...
Voila! Your webhooks will pick up this
and automatically serve the renewed certificate each time
myserver-tls secret is updated.
Once Istio TLS is configured, you'll need to make sure that clients know to trust certificates signed by your CA. For certificates signed by a public CA (like Let's Encrypt), most clients already include the CA root certificate in their trust stores for certificate verification. But, for a private CA, you will need to explicitly add your CA's root certificate to your clients' trust stores.
step CLI includes a utility command for this purpose on many systems:
step certificate install ca.crt
Rather than manually running the above for each machine that needs to trust your CA, most teams will use some form of automation to distribute the root certificate. Depending on your needs and your IT or DevOps team's approach, this may be a configuration management tool (like Ansible or Puppet), a Mobile Device Management (MDM) solution, or something else. Some examples:
ca.crtdirectly to the
ca-ceritficatesbundle on linux VMs so running applications trust the API servers they call
ca.crtdirectly into base Docker images for gRPC so gRPC clients can always reference the trusted CA
ca.crtin a Kubernetes
Secretand inject it into an environment variable for access from application code
ca.crtin the trust stores of every employee Macbook so their web browsers trust internal websites
step certificate install ca.crton target machines that want
curlto implicity trust the CA
ca.crtin a Kubernetes
ConfigMapand mount it to pods for reference on the filesystem
Alternatively, many clients support passing the CA root certificate as a flag or argument at runtime.
The Practical Zero Trust project is a collection of living documents detailing TLS configuration across a broad spread of technologies. We'd love to make this document better. Feel free to contribute any improvements directly on GitHub.