Smallstep Certificate Manager | Your Hosted Private CA

Securing Istio and Kubernetes With a Private Certificate Authority

Securing Istio and Kubernetes With a Private Certificate Authority

Prior to joining Smallstep, I worked in the realm of network connectivity, with a heavy focus on service mesh. To directly quote 2019 me, “Service mesh is redefining the way we think about security, reliability, and observability when it comes to service-to-service communication.” So it felt fitting to expand on the security component for my first blog post at a security toolkit company.

This post will cover how to secure an Istio mesh running on a Kubernetes cluster with a private certificate authority (CA). But I do want to give shoutouts to some awesome open-source alternatives. LinkerD offers a unique service mesh offering for Kubernetes that utilizes their homegrown lightning-fast dataplane proxy. It is a Cloud Native Computing Foundation (CNCF) incubating project. On the subject of CNCF projects, Kuma is another project you can explore. Much like Istio, it utilizes Envoy as the sidecar proxy, but with a heavier emphasis on universal deployments.

Why Use a Private Certificate Authority

Before we dive into how to achieve this, let’s take a second to talk about why you would want to use a private certificate authority for your Istio workloads. For most it comes down to some combination of:

  • You don’t completely trust your root private key in a kubernetes secret and want a cloud Key Management System (KMS) or Hardware Security Module (HSM) support.
  • You want TLS inside, outside, and between Istio and kubernetes clusters.
  • You manage more than containers and want automated certs for humans, machines, and devices.

There are cases where a private CA is not required. Most service meshes ship with an embedded CA. In Istio, the embedded CA is called Citadel. Citadel provides strong identities to workloads with X.509 certificates. But will all your workloads and services be in Istio? Will it all be in Kubernetes to begin with? I am willing to bet that for the majority of folks, no. Certificates extend beyond Istio workloads to Kubernetes, machine identity, cloud VMs, and across legacy environments. For diverse workloads and distributed machines that all need certificates, using step-ca for managing said certificates is simple. You can use OIDC to connect your identity provider and issue single sign-on certificates to humans. We also support ACME for automating certificates to servers, VMs, and internal websites. By connecting your Istio implementation to the broader tech stack with step-ca, you unlock some powerful extended use cases.

A Hardware Security Module (HSM) is a specialized device that is designed to generate, store, and use private keys securely. For folks that want to store private keys in something other than a kubernetes secret, step-ca supports cloud KMSs and PKCS 11, the specification for HSMs. The private keys on a HSM cannot be exported from the device. One can only run signing operations on the device. This is an excellent way to protect private keys for a Certificate Authority, whose primary function is to sign Certificate Signing Requests. In the early 2000s, HSMs were sometimes marketed as “eCommerce Accelerators” because companies used them primarily to speed up cryptographic operations for SSL connections. Today, they are more often used to meet tight security and compliance requirements. To learn how to use step-ca with HSM, please follow this blog post.

Another added benefit of step-ca is templates. Templates enable a organization to automate certificate details and bring identity standards across the organization. Try one of the several built-in templates for everyday operations, or users can use Golang's text/template syntax to create new templates. Template are JSON documents that can be configured to support certificates for TLS, SSH, code signing, email signing, or any other required certificate format.

And working alongside cert-manager, step-ca delivers enterprise security to Istio workloads. Cert-manager is a Kubernetes certificate management controller. This important add-on helps issue and renew certificates by monitoring Kubernetes secrets. But once we’re outside Kubernetes, step-ca continues to deliver the same enterprise security to your remaining workloads.

Enough talk, let us jump to the terminal and get our hands dirty.


To follow this blog post, please prepare the following things ready on your personal machine:

Kubernetes Cluster - I tested this blog post on GKE and minikube. But if you want to run on OpenShift, follow these instructions to prepare an OpenShift cluster for Istio. Step-CLI - Command-line tool to configure, operate, and automate the smallstep toolchain and open standard identity technologies. istioctl - Command-line tool that allows service operators to debug and diagnose their Istio service mesh deployments. Helm - Package manager for Kubernetes to help use install cert-manager and step-issuer.


Jetstack's cert-manager is a Kubernetes certificate management controller. This important add-on helps issue and renew certificates.


Using Helm, run the following commands to install cert-manager:

$ helm repo add jetstack
"jetstack" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "jetstack" chart repository
Update Complete. ⎈Happy Helming!
$ helm install \
 cert-manager jetstack/cert-manager \
 --namespace cert-manager \
 --create-namespace \
 --version v1.3.1 \
 --set installCRDs=true
NAME: cert-manager
LAST DEPLOYED: Mon May 10 08:21:13 2021
NAMESPACE: cert-manager
STATUS: deployed
cert-manager has been deployed successfully!
In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).
More information on the different types of issuers and how to configure them
can be found in our documentation:
For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`

Check all the pods within the cert-manager namespace.

$ kubectl get pods -n cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7998c69865-vzgg7              1/1     Running   0          5m40s
cert-manager-cainjector-7b744d56fb-c68f4   1/1     Running   0          5m40s
cert-manager-webhook-7d6d4c78bc-4wvns      1/1     Running   0          5m40s

Step Toolkit

cert-manager supports various different external CAs. Step Issuer uses step-certificate as the Certificate Authority in charge of signing the CertificateRequest resources. So before we install our issuer, let's set up a step-certificate CA into our cluster. And if you have any questions about the toolkit or this blog post, visit the Smallstep Discord channel to find our maintainers and community members.

Step Certificate Installation

Using Helm, install step-certificates first:

$ helm repo add smallstep
"smallstep" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "smallstep" chart repository
Update Complete. ⎈Happy Helming!
$ helm install \
 step-certificates smallstep/step-certificates \
 --namespace istio-system \
NAME: step-certificates
LAST DEPLOYED: Mon May 10 08:45:13 2021
NAMESPACE: istio-system
STATUS: deployed
Thanks for installing Step CA.
1. Get the PKI and Provisioner secrets running these commands:
  kubectl get -n istio-system -o jsonpath='{.data.password}' secret/step-certificates-ca-password | base64 --decode
  kubectl get -n istio-system -o jsonpath='{.data.password}' secret/step-certificates-provisioner-password | base64 --decode
2. Get the CA URL and the root certificate fingerprint running this command:
  kubectl -n istio-system logs job.batch/step-certificates
3. Delete the configuration job running this command:
  kubectl -n istio-system delete job.batch/step-certificates

With step-ca installed, we need to get the base64 version of our root certificate and kid. Note that your values will be different from the ones listed on this blog post To get the base64 version of the root certificate:

$ kubectl get -o jsonpath="{.data['root_ca\.crt']}" configmaps/step-certificates-certs -n istio-system | tr -d '\n' | base64

To get the provisioner kid:

$ kubectl get -o jsonpath="{.data['ca\.json']}" configmaps/step-certificates-config -n istio-system | jq .authority.provisioners
   "type": "JWK",
   "name": "admin",
   "key": {
     "use": "sig",
     "kty": "EC",
     "kid": "E4BpJlpL8QayAvkbKJRITAvaKZgJr8w7GsajKOxB4aU",
     "crv": "P-256",
     "alg": "ES256",
     "x": "UBx4JLlU0dZyJ5yqbCVQzS1SnaLV2Ga8U9u8Ib91A-w",
     "y": "MobU8KE6c8qQn8Z2MZ28yiIYn9XfDWJv83wgdFJW0E4"
   "encryptedKey": "eyJhbGciOiJQQkVTMi1IUzI1NitBMTI4S1ciLCJjdHkiOiJqd2sranNvbiIsImVuYyI6IkEyNTZHQ00iLCJwMmMiOjEwMDAwMCwicDJzIjoiWjA3WFc2cXU2SGF1QmhzX3Q5ZlBTQSJ9.kwcZH8r0jpKdN3O_QopnStG5zeMOkWkj7P3oueN5ia6uziv2XnoSaQ.Q0B7eo3i-HgLO9zK.DTxZ_22iRLwiVZOZCMPwL974jz4qJEV2LxJc21BAsCqWLdxZ7SJd8YlrBUHNCS0ZjR9LwgHPOtE4Fn23III0PTqJO0yDmFpLrUDkeDyOVFln56es8MDlb9xyPhQGDVYJ2yFqIAz_gQqMZz8L99k2QjANF2aPC45gfy7jZDN-PYR3-u_i94BBh6ckzfrPeE5qko54CfCPftoFuBw86PRlRVDEW_e31l9sVuq-PFrUF5nb4s-YQDR2BuyyXlT5eFC79TWxmkV3zZdnbsL86Cm7zlEW_tp4uLT_m4N1BONSnOIHznrXQMmMzBdjShG7QLXomZyT4fGPj8EWjDfDieM.eojVrYdpchvYag1EYcX5Yw"

To save these values, create a issuer-config.yaml file and populate it like so:

kind: StepIssuer
 name: step-issuer
 namespace: istio-system
 # The CA URL.
 url: https://step-certificates.istio-system.svc.cluster.local
 # The base64 encoded version of the CA root certificate in PEM format.
 # The provisioner name, kid, and a reference to the provisioner password secret.
   name: admin
   kid: E4BpJlpL8QayAvkbKJRITAvaKZgJr8w7GsajKOxB4aU
     name: step-certificates-provisioner-password
     key: password

Remember to replace the caBundle with the base64 version of the root certificate and the kid with the provisioner kid listed above.

Step Issuer Installation

We need to create an Issuer or ClusterIssuer resource before we can utilize the cert-manager we just deployed. These resources are necessary as they represent signing authorities and detail how certificate requests from Istio workload will be honored. Using Helm again, install the issuer that cert-manager relies on.

$ helm install \
 step-issuer smallstep/step-issuer \
 --namespace istio-system
NAME: step-issuer
LAST DEPLOYED: Mon May 10 09:27:13 2021
NAMESPACE: istio-system
STATUS: deployed
🍻 Happy signing.

Check the istio-system namespace to verify all step components are up and running:

$ kubectl get -n istio-system all
NAME                              READY   STATUS      RESTARTS   AGE
pod/step-certificates-0           1/1     Running     0          18m
pod/step-certificates-l7rhk       0/1     Completed   0          18m
pod/step-issuer-f6ffb88f6-h2bzx   2/2     Running     0          119s
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/step-certificates   ClusterIP   <none>        443/TCP    18m
service/step-issuer         ClusterIP    <none>        8443/TCP   119s
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/step-issuer   1/1     1            1           119s
NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/step-issuer-f6ffb88f6   1         1         1       119s
NAME                                 READY   AGE
statefulset.apps/step-certificates   1/1     18m
NAME                          COMPLETIONS   DURATION   AGE
job.batch/step-certificates   1/1           19s        18m

Issuer Configuration

Apply the previous issuer-config.yaml file we created.

$ kubectl apply -f issuer-config.yaml created

Moments later you should be able to see the status property in the resource:

$ kubectl get step-issuer -n istio-system -o yaml
kind: StepIssuer
 - lastTransitionTime: "2021-05-11T12:05:45Z"
   message: StepIssuer verified and ready to sign certificates
   reason: Verified
   status: "True"
   type: Ready

CertificateRequest Creation

Step Issuer has a controller watching for CertificateRequest resources. To create this CertificateRequest we first need a CSR. We can use step to create one, we will use the password my-password to encrypt the private key:

$ step certificate create --csr internal.csr internal.key
Please enter the password to encrypt the private key:
Your certificate signing request has been saved in internal.csr.
Your private key has been saved in internal.key.
$ cat internal.csr
$ cat internal.key
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-256-CBC,2cf0f7b4029ed9398fd28bee85520ae6

Encode the new CSR using base64:

$ cat internal.csr | tr -d '\n' | base64

Lastly, create a certificate-request.yaml and replace the spec.request value with the base64 value of your new CSR:

kind: CertificateRequest
 name: internal-smallstep-com
 namespace: istio-system
 # The base64 encoded version of the certificate request in PEM format.
 # The duration of the certificate
 duration: 24h
 # If the certificate will be a CA or not.
 # Step certificates won't accept a certificate request if this value is true,
 # you can also omit this.
 isCA: false
 # A reference to the issuer in charge of signing the CSR.
   kind: StepIssuer
   name: step-issuer

Apply it using kubectl:

$ kubectl apply -f certificate-request.yaml created

And moments later the bundled signed certificate with the intermediate as well as the root certificate will be available in the resource:

$ kubectl get internal-smallstep-com -n istio-system -o yaml
kind: CertificateRequest
 - lastTransitionTime: "2021-05-13T12:55:11Z"
   message: Certificate request has been approved by
   status: "True"
   type: Approved

Istio CSR

Connecting cert-manager to Istio without agents and addons is a pain. Trust me, I tried. But after struggling for a week, I stumbled upon cert-manager's Istio-CSR. This agent allows for Istio workload and control plane components to be secured using cert-manager with minimal work.


First, create a istio-csr-values.yaml file to change some of the Helm values before we deploy. Certificates facilitating mTLS, inter and intra cluster, will be signed, delivered, and renewed using the specs we define below.

 # -- Namespace to create CertificateRequests from incoming gRPC CSRs.
 namespace: istio-system
 # -- Issuer group name set on created CertificateRequests from incoming gRPC CSRs.
 # -- Issuer kind set on created CertificateRequests from incoming gRPC CSRs.
 kind: StepIssuer
 # -- Issuer name set on created CertificateRequests from incoming gRPC CSRs.
 name: step-issuer
 # -- Maximum validity duration that can be requested for a certificate.
 # istio-csr will request a duration of the smaller of this value, and that of
 # the incoming gRPC CSR.
 maxDuration: 24h
 # -- Don't delete created CertificateRequests once they have been signed.
 preserveCertificateRequests: false

Next, use Helm to install istio-csr and include the value file we just created.

$ helm install \
cert-manager-istio-csr jetstack/cert-manager-istio-csr \
-f istio-csr-values.yaml \
--namespace cert-manager
NAME: istio-csr
LAST DEPLOYED: Mon May 10 11:45:13 2021
NAMESPACE: cert-manager
STATUS: deployed



Deploy the Istio operator. You will need to have istioctl installed to do so:

$ istioctl operator init
Installing operator controller in namespace: istio-operator using image:
Operator controller will watch namespaces: istio-system
✔ Istio operator installed
✔ Installation complete

Create a file called istio-operator-config.yaml with these following values:

kind: IstioOperator
 name: istio
 namespace: istio-system
 profile: "demo"
     # Change certificate provider to cert-manager istio agent for istio agent
     caAddress: cert-manager-istio-csr.cert-manager.svc:443
         # Disable istiod CA Sever functionality
       - name: ENABLE_CA_SERVER
         value: "false"
       - apiVersion: apps/v1
         kind: Deployment
         name: istiod
           # Mount istiod serving and webhook certificate from Secret mount
         - path: spec.template.spec.containers.[name:discovery].args[7]
           value: "--tlsCertFile=/etc/cert-manager/tls/tls.crt"
         - path: spec.template.spec.containers.[name:discovery].args[8]
           value: "--tlsKeyFile=/etc/cert-manager/tls/tls.key"
         - path: spec.template.spec.containers.[name:discovery].args[9]
           value: "--caCertFile=/etc/cert-manager/ca/root-cert.pem"
         - path: spec.template.spec.containers.[name:discovery].volumeMounts[6]
             name: cert-manager
             mountPath: "/etc/cert-manager/tls"
             readOnly: true
         - path: spec.template.spec.containers.[name:discovery].volumeMounts[7]
             name: ca-root-cert
             mountPath: "/etc/cert-manager/ca"
             readOnly: true
         - path: spec.template.spec.volumes[6]
             name: cert-manager
               secretName: istiod-tls
         - path: spec.template.spec.volumes[7]
             name: ca-root-cert
               defaultMode: 420
               name: istio-ca-root-cert

Apply the operator configurations. The controller deployed by the init command above will detect the IstioOperator resource and then install the Istio components we want.

$ kubectl apply -f istio-operator-config.yaml created

Bookinfo Application

The last step is to deploy an application in our service mesh. The Bookinfo application is a Istio built demo application composed of four separate microservices. For simplicity sake, we'll stick with the demo application they've built to highlight what we've accomplished.


The default Istio installation uses automatic sidecar injection. Label the default namespace with istio-injection=enabled prior to deploying the application there:

$ kubectl label namespace default istio-injection=enabled
namespace/default labeled

Next, deploy the Bookinfo application using kubectl:

$ kubectl apply -f
service/details created
serviceaccount/bookinfo-details created
deployment.apps/details-v1 created
service/ratings created
serviceaccount/bookinfo-ratings created
deployment.apps/ratings-v1 created
service/reviews created
serviceaccount/bookinfo-reviews created
deployment.apps/reviews-v1 created
deployment.apps/reviews-v2 created
deployment.apps/reviews-v3 created
service/productpage created
serviceaccount/bookinfo-productpage created
deployment.apps/productpage-v1 created

Confirm the application is running:

$ kubectl get -n default all
NAME                                  READY   STATUS    RESTARTS   AGE
pod/details-v1-79f774bdb9-9mc24       2/2     Running   0          9h
pod/productpage-v1-6b746f74dc-nw7mt   2/2     Running   0          9h
pod/ratings-v1-b6994bb9-hhlfg         2/2     Running   0          9h
pod/reviews-v1-545db77b95-mkrnc       2/2     Running   0          9h
pod/reviews-v2-7bf8c9648f-wkhbm       2/2     Running   0          9h
pod/reviews-v3-84779c7bbc-468x2       2/2     Running   0          9h
NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/details       ClusterIP   <none>        9080/TCP   9h
service/kubernetes    ClusterIP     <none>        443/TCP    14h
service/productpage   ClusterIP   <none>        9080/TCP   9h
service/ratings       ClusterIP   <none>        9080/TCP   9h
service/reviews       ClusterIP    <none>        9080/TCP   9h
NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/details-v1       1/1     1            1           9h
deployment.apps/productpage-v1   1/1     1            1           9h
deployment.apps/ratings-v1       1/1     1            1           9h
deployment.apps/reviews-v1       1/1     1            1           9h
deployment.apps/reviews-v2       1/1     1            1           9h
deployment.apps/reviews-v3       1/1     1            1           9h
NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/details-v1-79f774bdb9       1         1         1       9h
replicaset.apps/productpage-v1-6b746f74dc   1         1         1       9h
replicaset.apps/ratings-v1-b6994bb9         1         1         1       9h
replicaset.apps/reviews-v1-545db77b95       1         1         1       9h
replicaset.apps/reviews-v2-7bf8c9648f       1         1         1       9h
replicaset.apps/reviews-v3-84779c7bbc       1         1         1       9h

The pods all contain two containers. One of the two containers is for the Envoy sidecar proxy from the automatic sidecar injection.

Examine Certificates

Let us see what is happening behind the scene as we apply all these commands. First, you can get the Step Certificates Root CA acting as the Istio CA.

$ kubectl get cm istio-ca-root-cert -o jsonpath="{.data['root-cert\.pem']}" | step certificate inspect -
       Version: 3 (0x2)
       Serial Number: 136737417284863388231467187453976404479 (0x66deab2c6e44407384aa4004728461ff)
   Signature Algorithm: ECDSA-SHA256
       Issuer: CN=Step Certificates Root CA
           Not Before: May 17 23:09:24 2021 UTC
           Not After : May 15 23:09:24 2031 UTC
       Subject: CN=Step Certificates Root CA

And if we dig a little deeper, we can examine the certificates making mTLS possible in our Istio cluster. Here is the certificate in the review-v1 microservice:

$ istioctl proxy-config secret reviews-v1-545db77b95-mkrnc -n default -o json | \
jq '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | \
sed 's/"//g' | base64 --decode | openssl x509 -noout -text
       Version: 3 (0x2)
       Serial Number:
   Signature Algorithm: ecdsa-with-SHA256
       Issuer: CN=Step Certificates Intermediate CA
           Not Before: May 17 23:45:26 2021 GMT
           Not After : May 18 23:46:26 2021 GMT
       Subject: CN=spiffe://cluster.local/ns/default/sa/bookinfo-reviews


That's all for my first blog post at Smallstep! Automated external certificate management for Istio environments only took 30 minutes. Using step-ca and cert-manager, we secured istio with a private certificate authority. step-ca delivers flexibility and unifies workloads across service mesh, kubernetes, and legacy platforms. With automations like the ACME protocol and enterprise security support for HSMs, smallstep delivers automated certificate management for DevOps. Download step-ca or try our free Certificate Manager SaaS offering and get started today.

Subscribe to updates

Unsubscribe anytime, see Privacy Policy