Go gRPC TLS — Practical Zero Trust

How to get and renew Go gRPC TLS certificates

Written September 14, 2021, last updated September 24, 2021

Zero Trust or BeyondProd approaches require authenticated and encrypted communications everywhere. TLS is the cryptographic protocol that powers encryption for all your technologies. For TLS, you need certificates. This practitioner's tutorial provides instructions for automating Go gRPC TLS certificate renewal and enabling server-side encryption.

Try it

Create a private key and request a certificate

Before you can configure Go gRPC TLS, you will need a certificate issued by a trusted certificate authority (CA). If you already have a certificate, private key, and CA root certificate from your organization's existing CA, you can skip to the Go gRPC TLS configuration section below. If you need to generate a certificate, you can:

Use a free hosted Smallstep Certificate Manager authority

Roll your own with open source step-ca

This tutorial is designed for private networks, not publicly trusted websites or endpoints. If you are looking to automate certificates for public websites, servers, or endpoints, we recommend using Let's Encrypt and the ACME instructions below.

To request a certificate from your CA using the step CLI, bootstrap your CA with step ca bootstrap and run the following command (sub the server name for the actual name / DNS name of your Go gRPC server).

step ca certificate "helloworld.default.svc.cluster.local" server.crt server.key

Your certificate and private key will be saved in server.crt and server.key respectively.

Request a copy of your CA root certificate, which will be used to make sure each application can trust certificates presented by other applications.

step ca root ca.crt

Your certificate will be saved in ca.crt.

Configure Go gRPC to use the certificate

We're going to create a simple gRPC server in Go that loads an X.509 certificate and its key, so that clients can connect using TLS server authentication.

First, let's create our Go module:

mkdir helloworld
cd helloworld
go mod init example.com/helloworld

Now we're going to implement the helloworld.proto Greeting service in a file named main.go. The code loads the X.509 certificate and its key and starts a gRPC server:

package main

import (
    "context"
    "crypto/tls"
    "log"
    "net"

    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials"
    "google.golang.org/grpc/examples/helloworld/helloworld"
    "google.golang.org/grpc/reflection"
)

type helloServer struct {
    helloworld.UnimplementedGreeterServer
}

func (*helloServer) SayHello(ctx context.Context, in *helloworld.HelloRequest) (*helloworld.HelloReply, error) {
    log.Printf("method=SayHello name=%s", in.GetName())
    return &helloworld.HelloReply{Message: "Hello " + in.GetName()}, nil
}

func main() {
    cert, err := tls.LoadX509KeyPair("server.crt", "server.key")
    if err != nil {
        log.Fatal(err)
    }

    opts := []grpc.ServerOption{
        grpc.Creds(credentials.NewServerTLSFromCert(&cert)),
    }

    lis, err := net.Listen("tcp", ":443")
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }

    srv := grpc.NewServer(opts...)
    helloworld.RegisterGreeterServer(srv, &helloServer{})
    reflection.Register(srv)

    log.Printf("server listening at %v", lis.Addr())
    if err := srv.Serve(lis); err != nil {
        log.Fatalf("failed to serve: %v", err)
    }
}

Now let's download the dependencies and compile it:

go mod tidy
go build main.go

And finally, we are going to start the server:

./main

Test Go gRPC TLS configuration

Now let's use grpcurl to test the server. We'll pass the flag -cacert with our root CA certificate file. Without this, the client will not trust the server, and the request will fail:

$ grpcurl -cacert ca.crt \
-d '{"name": "Smallstep"}' myserver.mycompany.net:443 helloworld.Greeter/SayHello
{
  "message": "Hello Smallstep"
}

Going forward we will be using the code in our go-grpc-example repository. Feel free to clone it now:

git clone https://github.com/smallstep/go-grpc-example.git

Operationalize It

Select a provisioner

Smallstep CAs use provisioners to authenticate certificate requests using passwords, one-time tokens, single sign-on, and a variety of other mechanisms.

ACME (RFC8555) is an open standard, used by Let's Encrypt, for authenticating certificate requests. To use ACME on a private network you need to run an ACME server. ACME is harder to setup, but has a large client ecosystem (some software even has built-in support).

Other provisioners use the open source step CLI and do not require a local network agent. The instructions below focus on the JWK provisioner, but can be repurposed with small tweaks to operationalize all non-ACME provisioners.

To learn more, see Configuring step-ca Provisioners.

Show me instructions for...

ACME

Other provisioners

The right provisioner depends on your operational environment.

The JWK provisioner is the most general-purpose provisioner. It supports password and one-time token-based authentication. To add a JWK provisioner called go-grpc to a hosted Certificate Manager authority (if you haven't already), run:

step ca provisioner add go-grpc --type JWK --create --x509-default-dur 720h

For instructions on adding provisioners to open source step-ca, or to learn more about other provisioner types, see Configuring step-ca Provisioners.

Configure Go gRPC TLS Certificate Automation

We've created a systemd-based certificate renewal timer that works with step. Check out our documentation on Renewal using systemd timers for background on how these timers work.

To install the certificate renewal unit files, run:

cd /etc/systemd/system
sudo curl -sL https://files.smallstep.com/cert-renewer@.service \
     -o cert-renewer@.service
sudo curl -sL https://files.smallstep.com/cert-renewer@.timer \
     -o cert-renewer@.timer

The renewal timer will check your certificate files every five minutes and renew them after two-thirds of their lifetime has elapsed.

We will need to create a method that builds a *tls.Certificate on every connection based on the renewed files. We probably should cache the *tls.Certificate, and monitor periodically if the certificate has changed and only return a new certificate if it has changed.

To do that, go.step.sm/crypto includes a method that allows to use a certificate and key files as parameters and automatically reads them again before the certificate expires.

package main

import (
    "context"
    "crypto/tls"
    "flag"
    "log"
    "net"

    "go.step.sm/crypto/tlsutil"

    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials"
    "google.golang.org/grpc/examples/helloworld/helloworld"
    "google.golang.org/grpc/reflection"
)

type helloServer struct {
    helloworld.UnimplementedGreeterServer
}

func (*helloServer) SayHello(ctx context.Context, in *helloworld.HelloRequest) (*helloworld.HelloReply, error) {
    log.Printf("method=SayHello name=%s", in.GetName())
    return &helloworld.HelloReply{Message: "Hello " + in.GetName()}, nil
}

func main() {
    var address, certFile, keyFile string
    flag.StringVar(&address, "address", ":443", "The address to listen to.")
    flag.StringVar(&certFile, "cert", "", "Server certificate file.")
    flag.StringVar(&keyFile, "key", "", "Private key file.")
    flag.Parse()

    if certFile == "" {
        log.Fatalln("flag --cert is required")
    }
    if keyFile == "" {
        log.Fatalln("flag --key is required")
    }

    c, err := tlsutil.NewServerCredentialsFromFile(certFile, keyFile)
    if err != nil {
        log.Fatal("error creating server credentials: %v", err)
    }

    opts := []grpc.ServerOption{
        grpc.Creds(credentials.NewTLS(c.TLSConfig())),
    }

    lis, err := net.Listen("tcp", address)
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }

    srv := grpc.NewServer(opts...)
    helloworld.RegisterGreeterServer(srv, &helloServer{})
    reflection.Register(srv)

    log.Printf("server listening at %v", lis.Addr())
    if err := srv.Serve(lis); err != nil {
        log.Fatalf("failed to serve: %v", err)
    }
}

In our example repository, go-grpc-example, this method is implemented in the server-cert binary. Here is the same code with some small improvements to optionally support mTLS too.

To start the renewal timer, run:

sudo systemctl daemon-reload
sudo systemctl enable --now cert-renewer@go-grpc.timer

You'll see that the timer is active, by checking the output of systemctl list-timers.

Distribute your root certificate to end users and systems

Once Go gRPC TLS is configured, you'll need to make sure that clients know to trust certificates signed by your CA. For certificates signed by a public CA (like Let's Encrypt), most clients already include the CA root certificate in their trust stores for certificate verification. But, for a private CA, you will need to explicitly add your CA's root certificate to your clients' trust stores.

The step CLI includes a utility command for this purpose on many systems:

step certificate install ca.crt

Rather than manually running the above for each machine that needs to trust your CA, most teams will use some form of automation to distribute the root certificate. Depending on your needs and your IT or DevOps team's approach, this may be a configuration management tool (like Ansible or Puppet), a Mobile Device Management (MDM) solution, or something else. Some examples:

Use Ansible to add ca.crt directly to the ca-ceritficates bundle on linux VMs so running applications trust the API servers they call

Bake ca.crt directly into base Docker images for gRPC so gRPC clients can always reference the trusted CA

Store ca.crt in a Kubernetes Secret and inject it into an environment variable for access from application code

Use Jamf to install ca.crt in the trust stores of every employee Macbook so their web browsers trust internal websites

Use Puppet to run step certificate install ca.crt on target machines that want curl to implicity trust the CA

Store ca.crt in a Kubernetes ConfigMap and mount it to pods for reference on the filesystem

Alternatively, many clients support passing the CA root certificate as a flag or argument at runtime.

Research notes

In researching Go gRPC TLS, we did some thorough investigation. Here are our rough notes if you are interested in diving deeper.

Full research notes

Even that step-ca supports to grant certificates using ACME to hostnames like localhost, acme/autocert requires domains with at least a dot (.) in it.

You might have noticed that in the kubernetes example we're not using the built-in ACME binary. It works also in Kuberentes, but you won't be able to scale it to use multiple pods because the ACME flow requires to know a previous state, and our examples and acme/autocert are stateless.

Go gRPC Client

Here is an example of a gRPC client written in Go, as opposed to using grpcurl from the CLI.

package main

import (
    "context"
    "crypto/x509"
    "log"
    "os"
    "time"

    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials"
    "google.golang.org/grpc/examples/helloworld/helloworld"
)

func main() {
    // Address of the server to connect to
    address := "myserver.mycompany.net:443"
    // Path to the root CA
    caCert := "/home/step/certs/root_ca.crt"

    // Read the root CA and create the certificate pool to use
    b, err := os.ReadFile(caCert)
    if err != nil {
        log.Fatalf("error reading %s: %v", caCert, err)
    }
    pool := x509.NewCertPool()
    pool.AppendCertsFromPEM(b)

    // Connect to the remote host using our root CA
    conn, err := grpc.Dial(address,
        grpc.WithTransportCredentials(credentials.NewClientTLSFromCert(pool, "")),
    )
    if err != nil {
        log.Fatalf("did not connect: %v", err)
    }
    defer conn.Close()

    // Create the client and do a request
    client := helloworld.NewGreeterClient(conn)
    ctx, cancel := context.WithTimeout(context.Background(), time.Second)
    defer cancel()
    r, err := client.SayHello(ctx, &helloworld.HelloRequest{
        Name: "Smallstep",
    })
    if err != nil {
        log.Fatalf("could not greet: %v", err)
    }
    log.Printf("Greeting: %s", r.GetMessage())
}

Contribute to this document

The Practical Zero Trust project is a collection of living documents detailing TLS configuration across a broad spread of technologies. We'd love to make this document better. Feel free to contribute any improvements directly on GitHub.

Subscribe to updates

Unsubscribe anytime. See our privacy policy.

Go gRPC TLS — Practical Zero Trust

How to get and renew Go gRPC TLS certificates

Try it

Create a private key and request a certificate

Configure Go gRPC to use the certificate

Test Go gRPC TLS configuration

Operationalize It

Select a provisioner

Show me instructions for...

Configure Go gRPC TLS Certificate Automation

Distribute your root certificate to end users and systems

Research notes

Full research notes

Go gRPC Client

Further reading

Contribute to this document

Subscribe to updates

Learn

Products

Pricing

Documentation

Open Source

Company