KCF Technologies improved OpenVPN security for embedded devices

KCF Technologies replaced passwords with certificates and improved collaboration while better securing OpenVPN endpoints. “One of the things that I saw that we could be doing better was the security around our embedded devices and how they authenticate to the cloud,” said Myron Semack, Chief Infrastructure Architect at KCF Technologies. “Today each base station has a unique secret, effectively a password, and we wanted to upgrade to a more robust certificate-based method.” KCF Technologies is a machine health company with a stated goal is to transform American manufacturing. Data is collected using vibration sensors on machines and passed to a central base station at each site. The base stations securely connect to the KCF Technologies cloud analytics platform. “We work with our customers to analyze the data, to understand what's happening with their machines, and identify issues before they happen.”

“A big project of mine was figuring out a way to improve OpenVPN authentication to base stations in the field,” said J. Hunter Hawke, Cybersecurity Analyst at KCF Technologies. “I started searching, and I ran into large PKI vendors that were very pricey. I kept looking into options and found smallstep on a Reddit comment thread. I went to the website and what I saw looked like something we could quickly make fit our use case. More importantly, it looked like an option we could expand into the long-term plan for KCF technologies as opposed to just something that can solve this one problem.”

Building on open-source step-ca

“I came across a smallstep blog post explaining the certificate chaining and OpenVPN configurations,” continues Hunter. “When I was doing my proof of concept, I copied and pasted files from the blog post over to an EC2 instance, ran it, and up it came. Compared to our other applications, it was a similar amount of effort building the architecture around smallstep.”

High availability was a challenge the KCF team needed to solve. “We created a Fargate cluster running on AWS ECS to host the certificate authority instances. We set up an application load balancer with three instances behind it at the moment. As needed, we can scale them up and down.” Getting to this state required some creative configuration, including a shared EFS file system and a set of lambdas to keep things in sync. “We were running into the problem of how to handle the configuration file, making sure that it always stays up to date on every instance,” said Hunter. “So I wrote a lambda that pulls parameters from a secret store from a second account as one boundary of security.” “We also separated the MySQL database, hosted in RDS, and set up protections around the whole system to make sure that it's highly available.

The KCF team used the smallstep provisioners to help bootstrap certificate issuance across many scenarios. “I wanted to use OIDC provisioner because I thought that was so cool,” said Hunter. “We are using that to issue any developer certificates. There are other in-house applications where OIDC provisioner was a great play that we will look to expand. We are using a JWT provisioner for OpenVPN server certificates. Finally, we are also using the X5C provisioner for authenticating the base stations to the cloud.”

The smallstep solution is now completely automated. “We describe our infrastructure as code using Terraform," Hunter continued. "I have a module defined to stand up a certificate authority and pull the necessary files from the original parent configuration. It then sets up a lambda, a file system, a database, and event notifications. Finally, the module creates an intermediate certificate authority and populates all the necessary configurations. If we need another certificate authority, I can instantiate the model module, hit enter, and spin everything right up."

“Ultimately, it wasn't too much work. It was more or less just fiddling with the smallstep pieces at each step of the way, ensuring that the configuration was consistent and reliable, said Hunter. “I would highly recommend reading all the smallstep documentation, read it end to end, and every word. That would have saved me lots of time.”

More secure, happier customers, and big plans

“Previously, we'd always have to fiddle around with the best method of authenticating,” Hunter continued. “For example, one of my colleagues needed to set up a system that distributes files to base stations. He was initially just going to do it by setting up different passwords. Instead, with smallstep, he used a signed certificate from the certificate authority that all of our base stations automatically already trust. This reduced a bunch of complexity.”

“Smallstep has also improved our collaboration with the Hardware team,” said Hunter. Now that they can use the certificate authority to pull certificates and issue them to devices, they were quite impressed with the product and very happy with all the new fun things they can do to make our hardware more secure. It's easy to take the smallstep toolchain and customize it to what we need.”

Myron also weighed in on some of the benefits of the smallstep toolchain. “Being able to host a certificate authority under the certificates.kcftech.com domain makes it much easier for our customer's I.T. departments to understand and apply firewall rules. It's the KCF Technology domain, as opposed to some random website elsewhere on the Internet, which some larger PKI providers require.”

“The fact that we are going to certificate-based authentication is a Gold Star on our product when compared to many other options out in the industry,” said Myron. “The biggest thing is getting certificates rolled into our products. It's going live with our next generation of hardware, and there is a lot of interest from the Hardware Team in retrofitting this into the existing base stations.”

The KCF team plans to extend the use of the smallstep toolchain. “Now that we have improved the security of pushing data into our cloud, I'll say the next step for this solution is securing remote management of base stations. We can leverage the same certificates to provide a more secure, more traceable, shorter-lived mechanism for our engineers and support staff to check base stations’ status,” said Myron.

The KCF Technologies team is happy with the results. “Smallstep works exactly as intended,” said Hunter. “I haven't run into any problems. I admit I was initially a little worried about handling state among all of them. But I found that there have been no problems whatsoever.” Myron piled on, “I will say you guys have one of the more polished open source solutions. Many open-source things are a random tool by itself that's not useful without a whole bunch of other stuff around it. You guys have built a complete toolchain that makes implementing certificates much easier.“