Securing Istio and Kubernetes With a Private Certificate Authority

Kevin Chen

Follow Smallstep

Securing Istio and Kubernetes With a Private Certificate Authority

Prior to joining Smallstep, I worked in the realm of network connectivity, with a heavy focus on service mesh. To directly quote 2019 me, “Service mesh is redefining the way we think about security, reliability, and observability when it comes to service-to-service communication.” So it felt fitting to expand on the security component for my first blog post at a security toolkit company.

This post will cover how to secure an Istio mesh running on a Kubernetes cluster with a private certificate authority (CA). But I do want to give shoutouts to some awesome open-source alternatives. LinkerD offers a unique service mesh offering for Kubernetes that utilizes their homegrown lightning-fast dataplane proxy. It is a Cloud Native Computing Foundation (CNCF) incubating project. On the subject of CNCF projects, Kuma is another project you can explore. Much like Istio, it utilizes Envoy as the sidecar proxy, but with a heavier emphasis on universal deployments.

Why Use a Private Certificate Authority

Before we dive into how to achieve this, let’s take a second to talk about why you would want to use a private certificate authority for your Istio workloads. For most it comes down to some combination of:

• You don’t completely trust your root private key in a kubernetes secret and want a cloud Key Management System (KMS) or Hardware Security Module (HSM) support.
• You want TLS inside, outside, and between Istio and kubernetes clusters.
• You manage more than containers and want automated certs for humans, machines, and devices.

There are cases where a private CA is not required. Most service meshes ship with an embedded CA. In Istio, the embedded CA is called Citadel. Citadel provides strong identities to workloads with X.509 certificates. But will all your workloads and services be in Istio? Will it all be in Kubernetes to begin with? I am willing to bet that for the majority of folks, no. Certificates extend beyond Istio workloads to Kubernetes, machine identity, cloud VMs, and across legacy environments. For diverse workloads and distributed machines that all need certificates, using step-ca for managing said certificates is simple. You can use OIDC to connect your identity provider and issue single sign-on certificates to humans. We also support ACME for automating certificates to servers, VMs, and internal websites. By connecting your Istio implementation to the broader tech stack with step-ca, you unlock some powerful extended use cases.

A Hardware Security Module (HSM) is a specialized device that is designed to generate, store, and use private keys securely. For folks that want to store private keys in something other than a kubernetes secret, step-ca supports cloud KMSs and PKCS 11, the specification for HSMs. The private keys on a HSM cannot be exported from the device. One can only run signing operations on the device. This is an excellent way to protect private keys for a Certificate Authority, whose primary function is to sign Certificate Signing Requests. In the early 2000s, HSMs were sometimes marketed as “eCommerce Accelerators” because companies used them primarily to speed up cryptographic operations for SSL connections. Today, they are more often used to meet tight security and compliance requirements. To learn how to use step-ca with HSM, please follow this blog post.

Another added benefit of step-ca is templates. Templates enable a organization to automate certificate details and bring identity standards across the organization. Try one of the several built-in templates for everyday operations, or users can use Golang's text/template syntax to create new templates. Template are JSON documents that can be configured to support certificates for TLS, SSH, code signing, email signing, or any other required certificate format.

And working alongside cert-manager, step-ca delivers enterprise security to Istio workloads. Cert-manager is a Kubernetes certificate management controller. This important add-on helps issue and renew certificates by monitoring Kubernetes secrets. But once we’re outside Kubernetes, step-ca continues to deliver the same enterprise security to your remaining workloads.

Enough talk, let us jump to the terminal and get our hands dirty.

Pre-requisites

To follow this blog post, please prepare the following things ready on your personal machine:

Kubernetes Cluster - I tested this blog post on GKE and minikube. But if you want to run on OpenShift, follow these instructions to prepare an OpenShift cluster for Istio. Step-CLI - Command-line tool to configure, operate, and automate the smallstep toolchain and open standard identity technologies. istioctl - Command-line tool that allows service operators to debug and diagnose their Istio service mesh deployments. Helm - Package manager for Kubernetes to help use install cert-manager and step-issuer.

cert-manager

Jetstack's cert-manager is a Kubernetes certificate management controller. This important add-on helps issue and renew certificates.

Installation

Using Helm, run the following commands to install cert-manager:

$ helm repo add jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "jetstack" chart repository
Update Complete. ⎈Happy Helming!⎈

$ helm install \
 cert-manager jetstack/cert-manager \
 --namespace cert-manager \
 --create-namespace \
 --version v1.3.1 \
 --set installCRDs=true
NAME: cert-manager
LAST DEPLOYED: Mon May 10 08:21:13 2021
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager has been deployed successfully!
In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).
More information on the different types of issuers and how to configure them
can be found in our documentation:
https://cert-manager.io/docs/configuration/
For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`
documentation:
https://cert-manager.io/docs/usage/ingress/

Check all the pods within the cert-manager namespace.

$ kubectl get pods -n cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7998c69865-vzgg7              1/1     Running   0          5m40s
cert-manager-cainjector-7b744d56fb-c68f4   1/1     Running   0          5m40s
cert-manager-webhook-7d6d4c78bc-4wvns      1/1     Running   0          5m40s

Step Toolkit

cert-manager supports various different external CAs. Step Issuer uses step-certificate as the Certificate Authority in charge of signing the CertificateRequest resources. So before we install our issuer, let's set up a step-certificate CA into our cluster. And if you have any questions about the toolkit or this blog post, visit the Smallstep Discord channel to find our maintainers and community members.

Step Certificate Installation

Using Helm, install step-certificates first:

$ helm repo add smallstep  https://smallstep.github.io/helm-charts
"smallstep" has been added to your repositories

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "smallstep" chart repository
Update Complete. ⎈Happy Helming!⎈

$ helm install \
 step-certificates smallstep/step-certificates \
 --namespace istio-system \
 --create-namespace
NAME: step-certificates
LAST DEPLOYED: Mon May 10 08:45:13 2021
NAMESPACE: istio-system
STATUS: deployed
REVISION: 1
NOTES:
Thanks for installing Step CA.
1. Get the PKI and Provisioner secrets running these commands:
  kubectl get -n istio-system -o jsonpath='{.data.password}' secret/step-certificates-ca-password | base64 --decode
  kubectl get -n istio-system -o jsonpath='{.data.password}' secret/step-certificates-provisioner-password | base64 --decode
2. Get the CA URL and the root certificate fingerprint running this command:
  kubectl -n istio-system logs job.batch/step-certificates
3. Delete the configuration job running this command:
  kubectl -n istio-system delete job.batch/step-certificates

With step-ca installed, we need to get the base64 version of our root certificate and kid. Note that your values will be different from the ones listed on this blog post To get the base64 version of the root certificate:

$ kubectl get -o jsonpath="{.data['root_ca\.crt']}" configmaps/step-certificates-certs -n istio-system | tr -d '\n' | base64
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJpekNDQVRLZ0F3SUJBZ0lSQUpSUzVPYTR0cHdCTUdwRFF0UnRIcTB3Q2dZSUtvWkl6ajBFQXdJd0pERWkKTUNBR0ExVUVBeE1aVTNSbGNDQkRaWEowYVdacFkyRjBaWE1nVW05dmRDQkRRVEFlRncweU1UQTFNVGN4TVRRdwpORFZhRncwek1UQTFNVFV4TVRRd05EVmFNQ1F4SWpBZ0JnTlZCQU1UR1ZOMFpYQWdRMlZ5ZEdsbWFXTmhkR1Z6CklGSnZiM1FnUTBFd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxaGtqT1BRTUJCd05DQUFSOG9tUUpNbk1CQjcxekpsak0KRWNkYTdiU2tmck9MZjlEamw3WjN1alRjMVdOQTl2ZnBGRFdud0ErbGk5NnlpQVJ6eVFTZTk0d2VVUU55Rnhqegp1NS9BbzBVd1F6QU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFUQWRCZ05WCkhRNEVGZ1FVc1R6WjV6NENjZTJRQ3FZSnhlWDM0bGVVUWJrd0NnWUlLb1pJemowRUF3SURSd0F3UkFJZ1RFc0YKZkl0RThQREVnbGUvMEhWMjZCakl4RDRWOHUzZDZzNmFCbnEvMmw4Q0lEczJvbGpRcVpFMjduNXhNZFNSOS84UworbjVHWU9aanc1UEkxZEFjQ3J1QwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

To get the provisioner kid:

$ kubectl get -o jsonpath="{.data['ca\.json']}" configmaps/step-certificates-config -n istio-system | jq .authority.provisioners
[
 {
   "type": "JWK",
   "name": "admin",
   "key": {
     "use": "sig",
     "kty": "EC",
     "kid": "E4BpJlpL8QayAvkbKJRITAvaKZgJr8w7GsajKOxB4aU",
     "crv": "P-256",
     "alg": "ES256",
     "x": "UBx4JLlU0dZyJ5yqbCVQzS1SnaLV2Ga8U9u8Ib91A-w",
     "y": "MobU8KE6c8qQn8Z2MZ28yiIYn9XfDWJv83wgdFJW0E4"
   },
   "encryptedKey": "eyJhbGciOiJQQkVTMi1IUzI1NitBMTI4S1ciLCJjdHkiOiJqd2sranNvbiIsImVuYyI6IkEyNTZHQ00iLCJwMmMiOjEwMDAwMCwicDJzIjoiWjA3WFc2cXU2SGF1QmhzX3Q5ZlBTQSJ9.kwcZH8r0jpKdN3O_QopnStG5zeMOkWkj7P3oueN5ia6uziv2XnoSaQ.Q0B7eo3i-HgLO9zK.DTxZ_22iRLwiVZOZCMPwL974jz4qJEV2LxJc21BAsCqWLdxZ7SJd8YlrBUHNCS0ZjR9LwgHPOtE4Fn23III0PTqJO0yDmFpLrUDkeDyOVFln56es8MDlb9xyPhQGDVYJ2yFqIAz_gQqMZz8L99k2QjANF2aPC45gfy7jZDN-PYR3-u_i94BBh6ckzfrPeE5qko54CfCPftoFuBw86PRlRVDEW_e31l9sVuq-PFrUF5nb4s-YQDR2BuyyXlT5eFC79TWxmkV3zZdnbsL86Cm7zlEW_tp4uLT_m4N1BONSnOIHznrXQMmMzBdjShG7QLXomZyT4fGPj8EWjDfDieM.eojVrYdpchvYag1EYcX5Yw"
 }
]

To save these values, create a issuer-config.yaml file and populate it like so:

apiVersion: certmanager.step.sm/v1beta1
kind: StepIssuer
metadata:
 name: step-issuer
 namespace: istio-system
spec:
 # The CA URL.
 url: https://step-certificates.istio-system.svc.cluster.local
 # The base64 encoded version of the CA root certificate in PEM format.
 caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJpekNDQVRLZ0F3SUJBZ0lSQUpSUzVPYTR0cHdCTUdwRFF0UnRIcTB3Q2dZSUtvWkl6ajBFQXdJd0pERWkKTUNBR0ExVUVBeE1aVTNSbGNDQkRaWEowYVdacFkyRjBaWE1nVW05dmRDQkRRVEFlRncweU1UQTFNVGN4TVRRdwpORFZhRncwek1UQTFNVFV4TVRRd05EVmFNQ1F4SWpBZ0JnTlZCQU1UR1ZOMFpYQWdRMlZ5ZEdsbWFXTmhkR1Z6CklGSnZiM1FnUTBFd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxaGtqT1BRTUJCd05DQUFSOG9tUUpNbk1CQjcxekpsak0KRWNkYTdiU2tmck9MZjlEamw3WjN1alRjMVdOQTl2ZnBGRFdud0ErbGk5NnlpQVJ6eVFTZTk0d2VVUU55Rnhqegp1NS9BbzBVd1F6QU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFUQWRCZ05WCkhRNEVGZ1FVc1R6WjV6NENjZTJRQ3FZSnhlWDM0bGVVUWJrd0NnWUlLb1pJemowRUF3SURSd0F3UkFJZ1RFc0YKZkl0RThQREVnbGUvMEhWMjZCakl4RDRWOHUzZDZzNmFCbnEvMmw4Q0lEczJvbGpRcVpFMjduNXhNZFNSOS84UworbjVHWU9aanc1UEkxZEFjQ3J1QwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
 # The provisioner name, kid, and a reference to the provisioner password secret.
 provisioner:
   name: admin
   kid: E4BpJlpL8QayAvkbKJRITAvaKZgJr8w7GsajKOxB4aU
   passwordRef:
     name: step-certificates-provisioner-password
     key: password

Remember to replace the caBundle with the base64 version of the root certificate and the kid with the provisioner kid listed above.

Step Issuer Installation

We need to create an Issuer or ClusterIssuer resource before we can utilize the cert-manager we just deployed. These resources are necessary as they represent signing authorities and detail how certificate requests from Istio workload will be honored. Using Helm again, install the issuer that cert-manager relies on.

$ helm install \
 step-issuer smallstep/step-issuer \
 --namespace istio-system
NAME: step-issuer
LAST DEPLOYED: Mon May 10 09:27:13 2021
NAMESPACE: istio-system
STATUS: deployed
REVISION: 1
...
🍻 Happy signing.

Check the istio-system namespace to verify all step components are up and running:

$ kubectl get -n istio-system all
NAME                              READY   STATUS      RESTARTS   AGE
pod/step-certificates-0           1/1     Running     0          18m
pod/step-certificates-l7rhk       0/1     Completed   0          18m
pod/step-issuer-f6ffb88f6-h2bzx   2/2     Running     0          119s
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/step-certificates   ClusterIP   10.108.172.126   <none>        443/TCP    18m
service/step-issuer         ClusterIP   10.104.92.237    <none>        8443/TCP   119s
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/step-issuer   1/1     1            1           119s
NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/step-issuer-f6ffb88f6   1         1         1       119s
NAME                                 READY   AGE
statefulset.apps/step-certificates   1/1     18m
NAME                          COMPLETIONS   DURATION   AGE
job.batch/step-certificates   1/1           19s        18m

Issuer Configuration

Apply the previous issuer-config.yaml file we created.

$ kubectl apply -f issuer-config.yaml
stepissuer.certmanager.step.sm/step-issuer created

Moments later you should be able to see the status property in the resource:

$ kubectl get stepissuers.certmanager.step.sm step-issuer -n istio-system -o yaml
apiVersion: certmanager.step.sm/v1beta1
kind: StepIssuer
...
status:
 conditions:
 - lastTransitionTime: "2021-05-11T12:05:45Z"
   message: StepIssuer verified and ready to sign certificates
   reason: Verified
   status: "True"
   type: Ready

CertificateRequest Creation

Step Issuer has a controller watching for CertificateRequest resources. To create this CertificateRequest we first need a CSR. We can use step to create one, we will use the password my-password to encrypt the private key:

$ step certificate create --csr internal.smallstep.com internal.csr internal.key
Please enter the password to encrypt the private key:
Your certificate signing request has been saved in internal.csr.
Your private key has been saved in internal.key.

$ cat internal.csr
-----BEGIN CERTIFICATE REQUEST-----
MIIBEDCBtwIBADAhMR8wHQYDVQQDExZpbnRlcm5hbC5zbWFsbHN0ZXAuY29tMFkw
EwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEUzfaIYwWAw+EAglS2zxWMBsu0bd6NzF9
Nsa2G7aCUk6vDV1FplRuxo49d7SfdzSPWCoUMYx7OHRBe5Wo2JaOmKA0MDIGCSqG
SIb3DQEJDjElMCMwIQYDVR0RBBowGIIWaW50ZXJuYWwuc21hbGxzdGVwLmNvbTAK
BggqhkjOPQQDAgNIADBFAiEAl6aBHpboyIcdwXMyG2qVkSUaLgWTre/fatA4GJx8
I2cCICnf9arQngpEnxCa0pUWNHFnu7OdNSlM+RK3XlZhcdff
-----END CERTIFICATE REQUEST-----

$ cat internal.key
-----BEGIN EC PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-256-CBC,2cf0f7b4029ed9398fd28bee85520ae6
E3ovadgxvkggvauzUuJGeYObbtQrZ2fE//lFDIPOhKcgqHt4YDSAzfV6vJhYVSNB
Ac0/ncT+A4sRx6qcjjV2xgett99ZDeRHJ9mBdxUBUcAaR3uAqjEJ0vC/LYK4MjPn
yn7cnYeEFaQZKvhohR/QjyaKrNQT5YOAf6JQlb0YlkQ=
-----END EC PRIVATE KEY-----

Encode the new CSR using base64:

$ cat internal.csr | tr -d '\n' | base64
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQkVEQ0J0d0lCQURBaE1SOHdIUVlEVlFRREV4WnBiblJsY201aGJDNXpiV0ZzYkhOMFpYQXVZMjl0TUZrdwpFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRVV6ZmFJWXdXQXcrRUFnbFMyenhXTUJzdTBiZDZOekY5Ck5zYTJHN2FDVWs2dkRWMUZwbFJ1eG80OWQ3U2ZkelNQV0NvVU1ZeDdPSFJCZTVXbzJKYU9tS0EwTURJR0NTcUcKU0liM0RRRUpEakVsTUNNd0lRWURWUjBSQkJvd0dJSVdhVzUwWlhKdVlXd3VjMjFoYkd4emRHVndMbU52YlRBSwpCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUFsNmFCSHBib3lJY2R3WE15RzJxVmtTVWFMZ1dUcmUvZmF0QTRHSng4CkkyY0NJQ25mOWFyUW5ncEVueENhMHBVV05IRm51N09kTlNsTStSSzNYbFpoY2RmZgotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K

Lastly, create a certificate-request.yaml and replace the spec.request value with the base64 value of your new CSR:

apiVersion: cert-manager.io/v1
kind: CertificateRequest
metadata:
 name: internal-smallstep-com
 namespace: istio-system
spec:
 # The base64 encoded version of the certificate request in PEM format.
 request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQkVEQ0J0d0lCQURBaE1SOHdIUVlEVlFRREV4WnBiblJsY201aGJDNXpiV0ZzYkhOMFpYQXVZMjl0TUZrdwpFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRVV6ZmFJWXdXQXcrRUFnbFMyenhXTUJzdTBiZDZOekY5Ck5zYTJHN2FDVWs2dkRWMUZwbFJ1eG80OWQ3U2ZkelNQV0NvVU1ZeDdPSFJCZTVXbzJKYU9tS0EwTURJR0NTcUcKU0liM0RRRUpEakVsTUNNd0lRWURWUjBSQkJvd0dJSVdhVzUwWlhKdVlXd3VjMjFoYkd4emRHVndMbU52YlRBSwpCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUFsNmFCSHBib3lJY2R3WE15RzJxVmtTVWFMZ1dUcmUvZmF0QTRHSng4CkkyY0NJQ25mOWFyUW5ncEVueENhMHBVV05IRm51N09kTlNsTStSSzNYbFpoY2RmZgotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
 # The duration of the certificate
 duration: 24h
 # If the certificate will be a CA or not.
 # Step certificates won't accept a certificate request if this value is true,
 # you can also omit this.
 isCA: false
 # A reference to the issuer in charge of signing the CSR.
 issuerRef:
   group: certmanager.step.sm
   kind: StepIssuer
   name: step-issuer

Apply it using kubectl:

$ kubectl apply -f certificate-request.yaml
certificaterequest.cert-manager.io/internal-smallstep-com created

And moments later the bundled signed certificate with the intermediate as well as the root certificate will be available in the resource:

$ kubectl get certificaterequests.cert-manager.io internal-smallstep-com -n istio-system -o yaml
apiVersion: cert-manager.io/v1
kind: CertificateRequest
...
status:
 conditions:
 - lastTransitionTime: "2021-05-13T12:55:11Z"
   message: Certificate request has been approved by cert-manager.io
   reason: cert-manager.io
   status: "True"
   type: Approved

Istio CSR

Connecting cert-manager to Istio without agents and addons is a pain. Trust me, I tried. But after struggling for a week, I stumbled upon cert-manager's Istio-CSR. This agent allows for Istio workload and control plane components to be secured using cert-manager with minimal work.

Installation

First, create a istio-csr-values.yaml file to change some of the Helm values before we deploy. Certificates facilitating mTLS, inter and intra cluster, will be signed, delivered, and renewed using the specs we define below.

certificate:
 # -- Namespace to create CertificateRequests from incoming gRPC CSRs.
 namespace: istio-system
 # -- Issuer group name set on created CertificateRequests from incoming gRPC CSRs.
 group: certmanager.step.sm
 # -- Issuer kind set on created CertificateRequests from incoming gRPC CSRs.
 kind: StepIssuer
 # -- Issuer name set on created CertificateRequests from incoming gRPC CSRs.
 name: step-issuer
 # -- Maximum validity duration that can be requested for a certificate.
 # istio-csr will request a duration of the smaller of this value, and that of
 # the incoming gRPC CSR.
 maxDuration: 24h
 # -- Don't delete created CertificateRequests once they have been signed.
 preserveCertificateRequests: false

Next, use Helm to install istio-csr and include the value file we just created.

$ helm install \
cert-manager-istio-csr jetstack/cert-manager-istio-csr \
-f istio-csr-values.yaml \
--namespace cert-manager
NAME: istio-csr
LAST DEPLOYED: Mon May 10 11:45:13 2021
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None

Istio

Installation

Deploy the Istio operator. You will need to have istioctl installed to do so:

$ istioctl operator init
Installing operator controller in namespace: istio-operator using image: docker.io/istio/operator:1.9.4
Operator controller will watch namespaces: istio-system
✔ Istio operator installed
✔ Installation complete

Create a file called istio-operator-config.yaml with these following values:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
 name: istio
 namespace: istio-system
spec:
 profile: "demo"
 hub: gcr.io/istio-release
 values:
   global:
     # Change certificate provider to cert-manager istio agent for istio agent
     caAddress: cert-manager-istio-csr.cert-manager.svc:443
 components:
   pilot:
     k8s:
       env:
         # Disable istiod CA Sever functionality
       - name: ENABLE_CA_SERVER
         value: "false"
       overlays:
       - apiVersion: apps/v1
         kind: Deployment
         name: istiod
         patches:
           # Mount istiod serving and webhook certificate from Secret mount
         - path: spec.template.spec.containers.[name:discovery].args[7]
           value: "--tlsCertFile=/etc/cert-manager/tls/tls.crt"
         - path: spec.template.spec.containers.[name:discovery].args[8]
           value: "--tlsKeyFile=/etc/cert-manager/tls/tls.key"
         - path: spec.template.spec.containers.[name:discovery].args[9]
           value: "--caCertFile=/etc/cert-manager/ca/root-cert.pem"
         - path: spec.template.spec.containers.[name:discovery].volumeMounts[6]
           value:
             name: cert-manager
             mountPath: "/etc/cert-manager/tls"
             readOnly: true
         - path: spec.template.spec.containers.[name:discovery].volumeMounts[7]
           value:
             name: ca-root-cert
             mountPath: "/etc/cert-manager/ca"
             readOnly: true
         - path: spec.template.spec.volumes[6]
           value:
             name: cert-manager
             secret:
               secretName: istiod-tls
         - path: spec.template.spec.volumes[7]
           value:
             name: ca-root-cert
             configMap:
               defaultMode: 420
               name: istio-ca-root-cert

Apply the operator configurations. The controller deployed by the init command above will detect the IstioOperator resource and then install the Istio components we want.

$ kubectl apply -f istio-operator-config.yaml
istiooperator.install.istio.io/istio created

Bookinfo Application

The last step is to deploy an application in our service mesh. The Bookinfo application is a Istio built demo application composed of four separate microservices. For simplicity sake, we'll stick with the demo application they've built to highlight what we've accomplished.

Installation

The default Istio installation uses automatic sidecar injection. Label the default namespace with istio-injection=enabled prior to deploying the application there:

$ kubectl label namespace default istio-injection=enabled
namespace/default labeled

Next, deploy the Bookinfo application using kubectl:

$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/bookinfo/platform/kube/bookinfo.yaml
service/details created
serviceaccount/bookinfo-details created
deployment.apps/details-v1 created
service/ratings created
serviceaccount/bookinfo-ratings created
deployment.apps/ratings-v1 created
service/reviews created
serviceaccount/bookinfo-reviews created
deployment.apps/reviews-v1 created
deployment.apps/reviews-v2 created
deployment.apps/reviews-v3 created
service/productpage created
serviceaccount/bookinfo-productpage created
deployment.apps/productpage-v1 created

Confirm the application is running:

$ kubectl get -n default all
NAME                                  READY   STATUS    RESTARTS   AGE
pod/details-v1-79f774bdb9-9mc24       2/2     Running   0          9h
pod/productpage-v1-6b746f74dc-nw7mt   2/2     Running   0          9h
pod/ratings-v1-b6994bb9-hhlfg         2/2     Running   0          9h
pod/reviews-v1-545db77b95-mkrnc       2/2     Running   0          9h
pod/reviews-v2-7bf8c9648f-wkhbm       2/2     Running   0          9h
pod/reviews-v3-84779c7bbc-468x2       2/2     Running   0          9h
NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/details       ClusterIP   10.3.253.254   <none>        9080/TCP   9h
service/kubernetes    ClusterIP   10.3.240.1     <none>        443/TCP    14h
service/productpage   ClusterIP   10.3.245.225   <none>        9080/TCP   9h
service/ratings       ClusterIP   10.3.250.177   <none>        9080/TCP   9h
service/reviews       ClusterIP   10.3.242.91    <none>        9080/TCP   9h
NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/details-v1       1/1     1            1           9h
deployment.apps/productpage-v1   1/1     1            1           9h
deployment.apps/ratings-v1       1/1     1            1           9h
deployment.apps/reviews-v1       1/1     1            1           9h
deployment.apps/reviews-v2       1/1     1            1           9h
deployment.apps/reviews-v3       1/1     1            1           9h
NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/details-v1-79f774bdb9       1         1         1       9h
replicaset.apps/productpage-v1-6b746f74dc   1         1         1       9h
replicaset.apps/ratings-v1-b6994bb9         1         1         1       9h
replicaset.apps/reviews-v1-545db77b95       1         1         1       9h
replicaset.apps/reviews-v2-7bf8c9648f       1         1         1       9h
replicaset.apps/reviews-v3-84779c7bbc       1         1         1       9h

The pods all contain two containers. One of the two containers is for the Envoy sidecar proxy from the automatic sidecar injection.

Examine Certificates

Let us see what is happening behind the scene as we apply all these commands. First, you can get the Step Certificates Root CA acting as the Istio CA.

$ kubectl get cm istio-ca-root-cert -o jsonpath="{.data['root-cert\.pem']}" | step certificate inspect -
Certificate:
   Data:
       Version: 3 (0x2)
       Serial Number: 136737417284863388231467187453976404479 (0x66deab2c6e44407384aa4004728461ff)
   Signature Algorithm: ECDSA-SHA256
       Issuer: CN=Step Certificates Root CA
       Validity
           Not Before: May 17 23:09:24 2021 UTC
           Not After : May 15 23:09:24 2031 UTC
       Subject: CN=Step Certificates Root CA
       ...

And if we dig a little deeper, we can examine the certificates making mTLS possible in our Istio cluster. Here is the certificate in the review-v1 microservice:

$ istioctl proxy-config secret reviews-v1-545db77b95-mkrnc -n default -o json | \
jq '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | \
sed 's/"//g' | base64 --decode | openssl x509 -noout -text
Certificate:
   Data:
       Version: 3 (0x2)
       Serial Number:
           98:24:47:8a:11:f8:18:f4:16:f9:e0:91:79:8b:60:c1
   Signature Algorithm: ecdsa-with-SHA256
       Issuer: CN=Step Certificates Intermediate CA
       Validity
           Not Before: May 17 23:45:26 2021 GMT
           Not After : May 18 23:46:26 2021 GMT
       Subject: CN=spiffe://cluster.local/ns/default/sa/bookinfo-reviews
       ...

Conclusion

That's all for my first blog post at Smallstep! Automated external certificate management for Istio environments only took 30 minutes. Using step-ca and cert-manager, we secured istio with a private certificate authority. step-ca delivers flexibility and unifies workloads across service mesh, kubernetes, and legacy platforms. With automations like the ACME protocol and enterprise security support for HSMs, smallstep delivers automated certificate management for DevOps. Download step-ca or try our free Certificate Manager SaaS offering and get started today.