Securing Istio and Kubernetes With a Private Certificate Authority
Kevin Chen
Securing Istio and Kubernetes With a Private Certificate Authority
Prior to joining Smallstep, I worked in the realm of network connectivity, with a heavy focus on service mesh. To directly quote 2019 me, “Service mesh is redefining the way we think about security, reliability, and observability when it comes to service-to-service communication.” So it felt fitting to expand on the security component for my first blog post at a security toolkit company.
This post will cover how to secure an Istio mesh running on a Kubernetes cluster with a private certificate authority (CA). But I do want to give shoutouts to some awesome open-source alternatives. LinkerD offers a unique service mesh offering for Kubernetes that utilizes their homegrown lightning-fast dataplane proxy. It is a Cloud Native Computing Foundation (CNCF) incubating project. On the subject of CNCF projects, Kuma is another project you can explore. Much like Istio, it utilizes Envoy as the sidecar proxy, but with a heavier emphasis on universal deployments.
Why Use a Private Certificate Authority
Before we dive into how to achieve this, let’s take a second to talk about why you would want to use a private certificate authority for your Istio workloads. For most it comes down to some combination of:
- You don’t completely trust your root private key in a kubernetes secret and want a cloud Key Management System (KMS) or Hardware Security Module (HSM) support.
- You want TLS inside, outside, and between Istio and kubernetes clusters.
- You manage more than containers and want automated certs for humans, machines, and devices.
There are cases where a private CA is not required. Most service meshes ship with an embedded CA. In Istio, the embedded CA is called Citadel. Citadel provides strong identities to workloads with X.509 certificates. But will all your workloads and services be in Istio? Will it all be in Kubernetes to begin with? I am willing to bet that for the majority of folks, no. Certificates extend beyond Istio workloads to Kubernetes, machine identity, cloud VMs, and across legacy environments. For diverse workloads and distributed machines that all need certificates, using step-ca for managing said certificates is simple. You can use OIDC to connect your identity provider and issue single sign-on certificates to humans. We also support ACME for automating certificates to servers, VMs, and internal websites. By connecting your Istio implementation to the broader tech stack with step-ca, you unlock some powerful extended use cases.
A Hardware Security Module (HSM) is a specialized device that is designed to generate, store, and use private keys securely. For folks that want to store private keys in something other than a kubernetes secret, step-ca supports cloud KMSs and PKCS 11, the specification for HSMs. The private keys on a HSM cannot be exported from the device. One can only run signing operations on the device. This is an excellent way to protect private keys for a Certificate Authority, whose primary function is to sign Certificate Signing Requests. In the early 2000s, HSMs were sometimes marketed as “eCommerce Accelerators” because companies used them primarily to speed up cryptographic operations for SSL connections. Today, they are more often used to meet tight security and compliance requirements. To learn how to use step-ca with HSM, please follow this blog post.
Another added benefit of step-ca is templates. Templates enable a organization to automate certificate details and bring identity standards across the organization. Try one of the several built-in templates for everyday operations, or users can use Golang's text/template syntax to create new templates. Template are JSON documents that can be configured to support certificates for TLS, SSH, code signing, email signing, or any other required certificate format.
And working alongside cert-manager, step-ca delivers enterprise security to Istio workloads. Cert-manager is a Kubernetes certificate management controller. This important add-on helps issue and renew certificates by monitoring Kubernetes secrets. But once we’re outside Kubernetes, step-ca continues to deliver the same enterprise security to your remaining workloads.
Enough talk, let us jump to the terminal and get our hands dirty.
Pre-requisites
To follow this blog post, please prepare the following things ready on your personal machine:
Kubernetes Cluster - I tested this blog post on GKE and minikube. But if you want to run on OpenShift, follow these instructions to prepare an OpenShift cluster for Istio. Step-CLI - Command-line tool to configure, operate, and automate the smallstep toolchain and open standard identity technologies. istioctl - Command-line tool that allows service operators to debug and diagnose their Istio service mesh deployments. Helm - Package manager for Kubernetes to help use install cert-manager and step-issuer.
cert-manager
Jetstack's cert-manager is a Kubernetes certificate management controller. This important add-on helps issue and renew certificates.
Installation
Using Helm, run the following commands to install cert-manager:
$ helm repo add jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "jetstack" chart repository
Update Complete. ⎈Happy Helming!⎈
$ helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.3.1 \
--set installCRDs=true
NAME: cert-manager
LAST DEPLOYED: Mon May 10 08:21:13 2021
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager has been deployed successfully!
In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).
More information on the different types of issuers and how to configure them
can be found in our documentation:
https://cert-manager.io/docs/configuration/
For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`
documentation:
https://cert-manager.io/docs/usage/ingress/
Check all the pods within the cert-manager namespace.
$ kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-7998c69865-vzgg7 1/1 Running 0 5m40s
cert-manager-cainjector-7b744d56fb-c68f4 1/1 Running 0 5m40s
cert-manager-webhook-7d6d4c78bc-4wvns 1/1 Running 0 5m40s
Step Toolkit
cert-manager supports various different external CAs. Step Issuer uses step-certificate as the Certificate Authority in charge of signing the CertificateRequest resources. So before we install our issuer, let's set up a step-certificate CA into our cluster. And if you have any questions about the toolkit or this blog post, visit the Smallstep Discord channel to find our maintainers and community members.
Step Certificate Installation
Using Helm, install step-certificates first:
$ helm repo add smallstep https://smallstep.github.io/helm-charts
"smallstep" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "smallstep" chart repository
Update Complete. ⎈Happy Helming!⎈
$ helm install \
step-certificates smallstep/step-certificates \
--namespace istio-system \
--create-namespace
NAME: step-certificates
LAST DEPLOYED: Mon May 10 08:45:13 2021
NAMESPACE: istio-system
STATUS: deployed
REVISION: 1
NOTES:
Thanks for installing Step CA.
1. Get the PKI and Provisioner secrets running these commands:
kubectl get -n istio-system -o jsonpath='{.data.password}' secret/step-certificates-ca-password | base64 --decode
kubectl get -n istio-system -o jsonpath='{.data.password}' secret/step-certificates-provisioner-password | base64 --decode
2. Get the CA URL and the root certificate fingerprint running this command:
kubectl -n istio-system logs job.batch/step-certificates
3. Delete the configuration job running this command:
kubectl -n istio-system delete job.batch/step-certificates
With step-ca installed, we need to get the base64 version of our root certificate and kid. Note that your values will be different from the ones listed on this blog post
To get the base64 version of the root certificate:
$ kubectl get -o jsonpath="{.data['root_ca\.crt']}" configmaps/step-certificates-certs -n istio-system | tr -d '\n' | base64
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJpekNDQVRLZ0F3SUJBZ0lSQUpSUzVPYTR0cHdCTUdwRFF0UnRIcTB3Q2dZSUtvWkl6ajBFQXdJd0pERWkKTUNBR0ExVUVBeE1aVTNSbGNDQkRaWEowYVdacFkyRjBaWE1nVW05dmRDQkRRVEFlRncweU1UQTFNVGN4TVRRdwpORFZhRncwek1UQTFNVFV4TVRRd05EVmFNQ1F4SWpBZ0JnTlZCQU1UR1ZOMFpYQWdRMlZ5ZEdsbWFXTmhkR1Z6CklGSnZiM1FnUTBFd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxaGtqT1BRTUJCd05DQUFSOG9tUUpNbk1CQjcxekpsak0KRWNkYTdiU2tmck9MZjlEamw3WjN1alRjMVdOQTl2ZnBGRFdud0ErbGk5NnlpQVJ6eVFTZTk0d2VVUU55Rnhqegp1NS9BbzBVd1F6QU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFUQWRCZ05WCkhRNEVGZ1FVc1R6WjV6NENjZTJRQ3FZSnhlWDM0bGVVUWJrd0NnWUlLb1pJemowRUF3SURSd0F3UkFJZ1RFc0YKZkl0RThQREVnbGUvMEhWMjZCakl4RDRWOHUzZDZzNmFCbnEvMmw4Q0lEczJvbGpRcVpFMjduNXhNZFNSOS84UworbjVHWU9aanc1UEkxZEFjQ3J1QwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
To get the provisioner kid:
$ kubectl get -o jsonpath="{.data['ca\.json']}" configmaps/step-certificates-config -n istio-system | jq .authority.provisioners
[
{
"type": "JWK",
"name": "admin",
"key": {
"use": "sig",
"kty": "EC",
"kid": "E4BpJlpL8QayAvkbKJRITAvaKZgJr8w7GsajKOxB4aU",
"crv": "P-256",
"alg": "ES256",
"x": "UBx4JLlU0dZyJ5yqbCVQzS1SnaLV2Ga8U9u8Ib91A-w",
"y": "MobU8KE6c8qQn8Z2MZ28yiIYn9XfDWJv83wgdFJW0E4"
},
"encryptedKey": "eyJhbGciOiJQQkVTMi1IUzI1NitBMTI4S1ciLCJjdHkiOiJqd2sranNvbiIsImVuYyI6IkEyNTZHQ00iLCJwMmMiOjEwMDAwMCwicDJzIjoiWjA3WFc2cXU2SGF1QmhzX3Q5ZlBTQSJ9.kwcZH8r0jpKdN3O_QopnStG5zeMOkWkj7P3oueN5ia6uziv2XnoSaQ.Q0B7eo3i-HgLO9zK.DTxZ_22iRLwiVZOZCMPwL974jz4qJEV2LxJc21BAsCqWLdxZ7SJd8YlrBUHNCS0ZjR9LwgHPOtE4Fn23III0PTqJO0yDmFpLrUDkeDyOVFln56es8MDlb9xyPhQGDVYJ2yFqIAz_gQqMZz8L99k2QjANF2aPC45gfy7jZDN-PYR3-u_i94BBh6ckzfrPeE5qko54CfCPftoFuBw86PRlRVDEW_e31l9sVuq-PFrUF5nb4s-YQDR2BuyyXlT5eFC79TWxmkV3zZdnbsL86Cm7zlEW_tp4uLT_m4N1BONSnOIHznrXQMmMzBdjShG7QLXomZyT4fGPj8EWjDfDieM.eojVrYdpchvYag1EYcX5Yw"
}
]
To save these values, create a issuer-config.yaml file and populate it like so:
apiVersion: certmanager.step.sm/v1beta1
kind: StepIssuer
metadata:
name: step-issuer
namespace: istio-system
spec:
# The CA URL.
url: https://step-certificates.istio-system.svc.cluster.local
# The base64 encoded version of the CA root certificate in PEM format.
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJpekNDQVRLZ0F3SUJBZ0lSQUpSUzVPYTR0cHdCTUdwRFF0UnRIcTB3Q2dZSUtvWkl6ajBFQXdJd0pERWkKTUNBR0ExVUVBeE1aVTNSbGNDQkRaWEowYVdacFkyRjBaWE1nVW05dmRDQkRRVEFlRncweU1UQTFNVGN4TVRRdwpORFZhRncwek1UQTFNVFV4TVRRd05EVmFNQ1F4SWpBZ0JnTlZCQU1UR1ZOMFpYQWdRMlZ5ZEdsbWFXTmhkR1Z6CklGSnZiM1FnUTBFd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxaGtqT1BRTUJCd05DQUFSOG9tUUpNbk1CQjcxekpsak0KRWNkYTdiU2tmck9MZjlEamw3WjN1alRjMVdOQTl2ZnBGRFdud0ErbGk5NnlpQVJ6eVFTZTk0d2VVUU55Rnhqegp1NS9BbzBVd1F6QU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFUQWRCZ05WCkhRNEVGZ1FVc1R6WjV6NENjZTJRQ3FZSnhlWDM0bGVVUWJrd0NnWUlLb1pJemowRUF3SURSd0F3UkFJZ1RFc0YKZkl0RThQREVnbGUvMEhWMjZCakl4RDRWOHUzZDZzNmFCbnEvMmw4Q0lEczJvbGpRcVpFMjduNXhNZFNSOS84UworbjVHWU9aanc1UEkxZEFjQ3J1QwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
# The provisioner name, kid, and a reference to the provisioner password secret.
provisioner:
name: admin
kid: E4BpJlpL8QayAvkbKJRITAvaKZgJr8w7GsajKOxB4aU
passwordRef:
name: step-certificates-provisioner-password
key: password
Remember to replace the caBundle with the base64 version of the root certificate and the kid with the provisioner kid listed above.
Step Issuer Installation
We need to create an Issuer or ClusterIssuer resource before we can utilize the cert-manager we just deployed. These resources are necessary as they represent signing authorities and detail how certificate requests from Istio workload will be honored. Using Helm again, install the issuer that cert-manager relies on.
$ helm install \
step-issuer smallstep/step-issuer \
--namespace istio-system
NAME: step-issuer
LAST DEPLOYED: Mon May 10 09:27:13 2021
NAMESPACE: istio-system
STATUS: deployed
REVISION: 1
...
🍻 Happy signing.
Check the istio-system namespace to verify all step components are up and running:
$ kubectl get -n istio-system all
NAME READY STATUS RESTARTS AGE
pod/step-certificates-0 1/1 Running 0 18m
pod/step-certificates-l7rhk 0/1 Completed 0 18m
pod/step-issuer-f6ffb88f6-h2bzx 2/2 Running 0 119s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/step-certificates ClusterIP 10.108.172.126 <none> 443/TCP 18m
service/step-issuer ClusterIP 10.104.92.237 <none> 8443/TCP 119s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/step-issuer 1/1 1 1 119s
NAME DESIRED CURRENT READY AGE
replicaset.apps/step-issuer-f6ffb88f6 1 1 1 119s
NAME READY AGE
statefulset.apps/step-certificates 1/1 18m
NAME COMPLETIONS DURATION AGE
job.batch/step-certificates 1/1 19s 18m
Issuer Configuration
Apply the previous issuer-config.yaml file we created.
$ kubectl apply -f issuer-config.yaml
stepissuer.certmanager.step.sm/step-issuer created
Moments later you should be able to see the status property in the resource:
$ kubectl get stepissuers.certmanager.step.sm step-issuer -n istio-system -o yaml
apiVersion: certmanager.step.sm/v1beta1
kind: StepIssuer
...
status:
conditions:
- lastTransitionTime: "2021-05-11T12:05:45Z"
message: StepIssuer verified and ready to sign certificates
reason: Verified
status: "True"
type: Ready
CertificateRequest Creation
Step Issuer has a controller watching for CertificateRequest resources. To create this CertificateRequest we first need a CSR. We can use step to create one, we will use the password my-password to encrypt the private key:
$ step certificate create --csr internal.smallstep.com internal.csr internal.key
Please enter the password to encrypt the private key:
Your certificate signing request has been saved in internal.csr.
Your private key has been saved in internal.key.
$ cat internal.csr
-----BEGIN CERTIFICATE REQUEST-----
MIIBEDCBtwIBADAhMR8wHQYDVQQDExZpbnRlcm5hbC5zbWFsbHN0ZXAuY29tMFkw
EwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEUzfaIYwWAw+EAglS2zxWMBsu0bd6NzF9
Nsa2G7aCUk6vDV1FplRuxo49d7SfdzSPWCoUMYx7OHRBe5Wo2JaOmKA0MDIGCSqG
SIb3DQEJDjElMCMwIQYDVR0RBBowGIIWaW50ZXJuYWwuc21hbGxzdGVwLmNvbTAK
BggqhkjOPQQDAgNIADBFAiEAl6aBHpboyIcdwXMyG2qVkSUaLgWTre/fatA4GJx8
I2cCICnf9arQngpEnxCa0pUWNHFnu7OdNSlM+RK3XlZhcdff
-----END CERTIFICATE REQUEST-----
$ cat internal.key
-----BEGIN EC PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-256-CBC,2cf0f7b4029ed9398fd28bee85520ae6
E3ovadgxvkggvauzUuJGeYObbtQrZ2fE//lFDIPOhKcgqHt4YDSAzfV6vJhYVSNB
Ac0/ncT+A4sRx6qcjjV2xgett99ZDeRHJ9mBdxUBUcAaR3uAqjEJ0vC/LYK4MjPn
yn7cnYeEFaQZKvhohR/QjyaKrNQT5YOAf6JQlb0YlkQ=
-----END EC PRIVATE KEY-----
Encode the new CSR using base64:
$ cat internal.csr | tr -d '\n' | base64
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQkVEQ0J0d0lCQURBaE1SOHdIUVlEVlFRREV4WnBiblJsY201aGJDNXpiV0ZzYkhOMFpYQXVZMjl0TUZrdwpFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRVV6ZmFJWXdXQXcrRUFnbFMyenhXTUJzdTBiZDZOekY5Ck5zYTJHN2FDVWs2dkRWMUZwbFJ1eG80OWQ3U2ZkelNQV0NvVU1ZeDdPSFJCZTVXbzJKYU9tS0EwTURJR0NTcUcKU0liM0RRRUpEakVsTUNNd0lRWURWUjBSQkJvd0dJSVdhVzUwWlhKdVlXd3VjMjFoYkd4emRHVndMbU52YlRBSwpCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUFsNmFCSHBib3lJY2R3WE15RzJxVmtTVWFMZ1dUcmUvZmF0QTRHSng4CkkyY0NJQ25mOWFyUW5ncEVueENhMHBVV05IRm51N09kTlNsTStSSzNYbFpoY2RmZgotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
Lastly, create a certificate-request.yaml and replace the spec.request value with the base64 value of your new CSR:
apiVersion: cert-manager.io/v1
kind: CertificateRequest
metadata:
name: internal-smallstep-com
namespace: istio-system
spec:
# The base64 encoded version of the certificate request in PEM format.
request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQkVEQ0J0d0lCQURBaE1SOHdIUVlEVlFRREV4WnBiblJsY201aGJDNXpiV0ZzYkhOMFpYQXVZMjl0TUZrdwpFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRVV6ZmFJWXdXQXcrRUFnbFMyenhXTUJzdTBiZDZOekY5Ck5zYTJHN2FDVWs2dkRWMUZwbFJ1eG80OWQ3U2ZkelNQV0NvVU1ZeDdPSFJCZTVXbzJKYU9tS0EwTURJR0NTcUcKU0liM0RRRUpEakVsTUNNd0lRWURWUjBSQkJvd0dJSVdhVzUwWlhKdVlXd3VjMjFoYkd4emRHVndMbU52YlRBSwpCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUFsNmFCSHBib3lJY2R3WE15RzJxVmtTVWFMZ1dUcmUvZmF0QTRHSng4CkkyY0NJQ25mOWFyUW5ncEVueENhMHBVV05IRm51N09kTlNsTStSSzNYbFpoY2RmZgotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
# The duration of the certificate
duration: 24h
# If the certificate will be a CA or not.
# Step certificates won't accept a certificate request if this value is true,
# you can also omit this.
isCA: false
# A reference to the issuer in charge of signing the CSR.
issuerRef:
group: certmanager.step.sm
kind: StepIssuer
name: step-issuer
Apply it using kubectl:
$ kubectl apply -f certificate-request.yaml
certificaterequest.cert-manager.io/internal-smallstep-com created
And moments later the bundled signed certificate with the intermediate as well as the root certificate will be available in the resource:
$ kubectl get certificaterequests.cert-manager.io internal-smallstep-com -n istio-system -o yaml
apiVersion: cert-manager.io/v1
kind: CertificateRequest
...
status:
conditions:
- lastTransitionTime: "2021-05-13T12:55:11Z"
message: Certificate request has been approved by cert-manager.io
reason: cert-manager.io
status: "True"
type: Approved
Istio CSR
Connecting cert-manager to Istio without agents and addons is a pain. Trust me, I tried. But after struggling for a week, I stumbled upon cert-manager's Istio-CSR. This agent allows for Istio workload and control plane components to be secured using cert-manager with minimal work.
Installation
First, create a istio-csr-values.yaml file to change some of the Helm values before we deploy. Certificates facilitating mTLS, inter and intra cluster, will be signed, delivered, and renewed using the specs we define below.
certificate:
# -- Namespace to create CertificateRequests from incoming gRPC CSRs.
namespace: istio-system
# -- Issuer group name set on created CertificateRequests from incoming gRPC CSRs.
group: certmanager.step.sm
# -- Issuer kind set on created CertificateRequests from incoming gRPC CSRs.
kind: StepIssuer
# -- Issuer name set on created CertificateRequests from incoming gRPC CSRs.
name: step-issuer
# -- Maximum validity duration that can be requested for a certificate.
# istio-csr will request a duration of the smaller of this value, and that of
# the incoming gRPC CSR.
maxDuration: 24h
# -- Don't delete created CertificateRequests once they have been signed.
preserveCertificateRequests: false
Next, use Helm to install istio-csr and include the value file we just created.
$ helm install \
cert-manager-istio-csr jetstack/cert-manager-istio-csr \
-f istio-csr-values.yaml \
--namespace cert-manager
NAME: istio-csr
LAST DEPLOYED: Mon May 10 11:45:13 2021
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
Istio
Installation
Deploy the Istio operator. You will need to have istioctl installed to do so:
$ istioctl operator init
Installing operator controller in namespace: istio-operator using image: docker.io/istio/operator:1.9.4
Operator controller will watch namespaces: istio-system
✔ Istio operator installed
✔ Installation complete
Create a file called istio-operator-config.yaml with these following values:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istio
namespace: istio-system
spec:
profile: "demo"
hub: gcr.io/istio-release
values:
global:
# Change certificate provider to cert-manager istio agent for istio agent
caAddress: cert-manager-istio-csr.cert-manager.svc:443
components:
pilot:
k8s:
env:
# Disable istiod CA Sever functionality
- name: ENABLE_CA_SERVER
value: "false"
overlays:
- apiVersion: apps/v1
kind: Deployment
name: istiod
patches:
# Mount istiod serving and webhook certificate from Secret mount
- path: spec.template.spec.containers.[name:discovery].args[7]
value: "--tlsCertFile=/etc/cert-manager/tls/tls.crt"
- path: spec.template.spec.containers.[name:discovery].args[8]
value: "--tlsKeyFile=/etc/cert-manager/tls/tls.key"
- path: spec.template.spec.containers.[name:discovery].args[9]
value: "--caCertFile=/etc/cert-manager/ca/root-cert.pem"
- path: spec.template.spec.containers.[name:discovery].volumeMounts[6]
value:
name: cert-manager
mountPath: "/etc/cert-manager/tls"
readOnly: true
- path: spec.template.spec.containers.[name:discovery].volumeMounts[7]
value:
name: ca-root-cert
mountPath: "/etc/cert-manager/ca"
readOnly: true
- path: spec.template.spec.volumes[6]
value:
name: cert-manager
secret:
secretName: istiod-tls
- path: spec.template.spec.volumes[7]
value:
name: ca-root-cert
configMap:
defaultMode: 420
name: istio-ca-root-cert
Apply the operator configurations. The controller deployed by the init command above will detect the IstioOperator resource and then install the Istio components we want.
$ kubectl apply -f istio-operator-config.yaml
istiooperator.install.istio.io/istio created
Bookinfo Application
The last step is to deploy an application in our service mesh. The Bookinfo application is a Istio built demo application composed of four separate microservices. For simplicity sake, we'll stick with the demo application they've built to highlight what we've accomplished.
Installation
The default Istio installation uses automatic sidecar injection. Label the default namespace with istio-injection=enabled prior to deploying the application there:
$ kubectl label namespace default istio-injection=enabled
namespace/default labeled
Next, deploy the Bookinfo application using kubectl:
$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/bookinfo/platform/kube/bookinfo.yaml
service/details created
serviceaccount/bookinfo-details created
deployment.apps/details-v1 created
service/ratings created
serviceaccount/bookinfo-ratings created
deployment.apps/ratings-v1 created
service/reviews created
serviceaccount/bookinfo-reviews created
deployment.apps/reviews-v1 created
deployment.apps/reviews-v2 created
deployment.apps/reviews-v3 created
service/productpage created
serviceaccount/bookinfo-productpage created
deployment.apps/productpage-v1 created
Confirm the application is running:
$ kubectl get -n default all
NAME READY STATUS RESTARTS AGE
pod/details-v1-79f774bdb9-9mc24 2/2 Running 0 9h
pod/productpage-v1-6b746f74dc-nw7mt 2/2 Running 0 9h
pod/ratings-v1-b6994bb9-hhlfg 2/2 Running 0 9h
pod/reviews-v1-545db77b95-mkrnc 2/2 Running 0 9h
pod/reviews-v2-7bf8c9648f-wkhbm 2/2 Running 0 9h
pod/reviews-v3-84779c7bbc-468x2 2/2 Running 0 9h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/details ClusterIP 10.3.253.254 <none> 9080/TCP 9h
service/kubernetes ClusterIP 10.3.240.1 <none> 443/TCP 14h
service/productpage ClusterIP 10.3.245.225 <none> 9080/TCP 9h
service/ratings ClusterIP 10.3.250.177 <none> 9080/TCP 9h
service/reviews ClusterIP 10.3.242.91 <none> 9080/TCP 9h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/details-v1 1/1 1 1 9h
deployment.apps/productpage-v1 1/1 1 1 9h
deployment.apps/ratings-v1 1/1 1 1 9h
deployment.apps/reviews-v1 1/1 1 1 9h
deployment.apps/reviews-v2 1/1 1 1 9h
deployment.apps/reviews-v3 1/1 1 1 9h
NAME DESIRED CURRENT READY AGE
replicaset.apps/details-v1-79f774bdb9 1 1 1 9h
replicaset.apps/productpage-v1-6b746f74dc 1 1 1 9h
replicaset.apps/ratings-v1-b6994bb9 1 1 1 9h
replicaset.apps/reviews-v1-545db77b95 1 1 1 9h
replicaset.apps/reviews-v2-7bf8c9648f 1 1 1 9h
replicaset.apps/reviews-v3-84779c7bbc 1 1 1 9h
The pods all contain two containers. One of the two containers is for the Envoy sidecar proxy from the automatic sidecar injection.
Examine Certificates
Let us see what is happening behind the scene as we apply all these commands. First, you can get the Step Certificates Root CA acting as the Istio CA.
$ kubectl get cm istio-ca-root-cert -o jsonpath="{.data['root-cert\.pem']}" | step certificate inspect -
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 136737417284863388231467187453976404479 (0x66deab2c6e44407384aa4004728461ff)
Signature Algorithm: ECDSA-SHA256
Issuer: CN=Step Certificates Root CA
Validity
Not Before: May 17 23:09:24 2021 UTC
Not After : May 15 23:09:24 2031 UTC
Subject: CN=Step Certificates Root CA
...
And if we dig a little deeper, we can examine the certificates making mTLS possible in our Istio cluster. Here is the certificate in the review-v1 microservice:
$ istioctl proxy-config secret reviews-v1-545db77b95-mkrnc -n default -o json | \
jq '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | \
sed 's/"//g' | base64 --decode | openssl x509 -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
98:24:47:8a:11:f8:18:f4:16:f9:e0:91:79:8b:60:c1
Signature Algorithm: ecdsa-with-SHA256
Issuer: CN=Step Certificates Intermediate CA
Validity
Not Before: May 17 23:45:26 2021 GMT
Not After : May 18 23:46:26 2021 GMT
Subject: CN=spiffe://cluster.local/ns/default/sa/bookinfo-reviews
...
Conclusion
That's all for my first blog post at Smallstep! Automated external certificate management for Istio environments only took 30 minutes. Using step-ca and cert-manager, we secured istio with a private certificate authority. step-ca delivers flexibility and unifies workloads across service mesh, kubernetes, and legacy platforms. With automations like the ACME protocol and enterprise security support for HSMs, smallstep delivers automated certificate management for DevOps. Download step-ca or try our free Certificate Manager SaaS offering and get started today.
