Integrating Secure Code Signing in the CI/CD Pipeline

In our increasingly digital world, trust is everything.

End users need to trust the software with which they’re interacting, developers and engineers need to be able to create that trust, and security teams need to put the proper controls in place to enable the entire cycle.

Of course, as we all know by now, establishing and maintaining digital trust isn’t always easy.

One of the best ways to establish digital trust is through the use of identities and, more specifically, code signing, which verifies the authenticity of new releases. But how exactly is code signing used in organizations today? What are some of the top challenges teams face in implementing code signing, and what happens when those processes aren’t secure? Perhaps most importantly, how can you ensure secure code signing in your CI/CD pipeline?

We recently answered these questions and more during our webinar Integrating Secure Code Signing in the CI/CD Pipeline. Read on for the highlights of this discussion, or click here to watch the full recording.

First and foremost, let’s talk about establishing digital trust. When it comes to code signing, every organization needs to create a trusted environment where security teams feel comfortable with the controls in place and where developers and engineers can work however and wherever they want without creating unnecessary risk.

Striking that balance, especially for DevOps teams, isn’t easy. Between developers working remotely, workloads running in multiple environments, and DevOps tool chains getting more complex, it’s becoming more and more challenging to know what’s trusted and what’s not. 

This is where identity becomes so important – especially for machines. Quite simply, every machine, every device, and every workload needs an identity, and every one of those identities must be protected and managed.

Code signing certificates are no different: They are the identity behind your software since they’re used to authenticate everything from software builds and artifacts to container images and even firmware and IoT settings. As a result, if that code signing identity is compromised, it undermines trust.

Over the past several years, the role of code signing in establishing digital trust has become more and more critical. End users downloading software from the internet expect that software to be signed. 

That reality combined with the rise of zero-trust environments means that just about every asset a company produces needs to be signed – anything that is deployable (think scripts, infrastructure code, container images, applications, WAR files, EAR files, DLLs, executables, macros, you name it) needs to be signed because malicious activity can happen anywhere.

With that in mind, some of the most common code signing use cases include:

• Protecting infrastructure: Preventing unauthorized downloads and ransomware by allowing only trusted applications to run on your infrastructure.

• Securing devices and firmware: Preventing unauthorized device access by securing firmware and firmware updates with digital signatures and verification.

• Safeguarding software: Ensuring published and distributed software is signed and that keys are protected.

• Trusting containers: Establishing trusted and digitally signed base images for containers and virtual machines deployed in your environment.

• Enabling DevSecOps: Establishing trust in the CI/CD pipeline, from digitally signing source code checked into repositories to signing packages post-build.
• Preventing malicious macros: Preventing unauthorized or malicious macros on internal devices by ensuring that only signed macros are allowed to run.

Given the importance of code signing to establish digital trust and the wide variety of use cases for code signing, organizations must pay close attention to their code signing practices. That’s where the idea of a code-signing maturity model comes into play. That model looks like this:

• Not signing software deliverables
• Vulnerable to unauthorized code access
• No auditability or integrity behind the software

• Signing software deliverables, but with fragmented tools
• Developers handle their own private keys without centralized control
• Signing keys are vulnerable to misuse or compromise

• Use of a centralized platform for code signing policy, workflow, and auditability
• Protecting sensitive signing keys in a secure HSM
• Not integrated with CI/CD processes or workflows and not all use cases are covered

• Signing all containers, artifacts, and software deliverables
• Integrated with native signing tools and workflows
• Full auditability and governance over all signing processes

Unfortunately, many organizations today sit somewhere between Level 0 and Level 1 on this maturity model. The problem isn’t that teams don’t recognize the importance of code signing, but rather that they don’t know how to sign or aren’t up to date on best practices around signing.

For example, we often see keys being passed around USB drives, which creates exposure risks. Or we see different teams using different point solutions, meaning processes are fragmented and don’t translate well across the organization. 

This happens because secure code signing is challenging: You need to make keys accessible to the teams that need them, wherever they are, but make them inaccessible to those who shouldn’t be signing with those keys. It can seem like a Catch-22.

So how exactly do you get to centralization and beyond on the code signing maturity model? There are typically three core challenges that teams need to solve. Fortunately, Keyfactor Signum offers solutions to these challenges through features like key protection, policy controls, event logs, native interfaces, authentication, and attestation. 

The challenge: To start, where are your code signing keys stored? Too often, they’re stored on flash drives, build servers, or workstations – all places vulnerable to third-party, unauthorized access.

The solution: Keyfactor Signum protects these sensitive keys by generating and storing them within an HSM. Notably, Keyfactor prevents these keys from leaving that HSM once generated.

Why it matters: Going forward, this key attestation (or the ability to provide proof to your Public CA that keys are generated on an HSM) will be highly critical, as the CAB Forum will soon be enforcing that Public CAs must have this proof to sign any certificates.

The challenge: Next, who has access to actually sign code, and is there an audit trail of who signed what, when they signed, and what they signed with? It’s not enough to store and generate keys in a secure place, as you also need to control who has access to them.

The solution: Keyfactor Signum establishes this auditability by enforcing rules and policies around who has access to keys, who can use them, and what they can be used for. It also provides a complete audit trail for visibility into everything going on in the environment.

Why it matters: As the volume of use cases that require code signing continues to grow, this level of visibility to verify and validate signing activities will be essential to scaling securely.

The challenge: Finally, developers want something easy to use and won’t change their process. So if you’re storing keys in a secure location and have tight controls over access, how do you still make those keys easily accessible for the teams that need them to sign code?

The solution: Keyfactor Signum makes these protections operationally effective by integrating into native signing tools and dropping into existing build and delivery processes; that way, developers don’t have to change their workflows.

Why it matters: Allowing your developers to continue working the way they already work by introducing native integrations to their existing tools makes it easy for them to get keys as needed (therefore avoiding slowdowns and/or workarounds) while still maintaining security by authenticating through an identity provider.

Knowing what Keyfactor Signum is, is one thing, understanding how exactly it works and what it looks like in practice is another. With that in mind, here’s a look at a typical workflow with Keyfactor Signum:

Once code is committed, wherever it comes from and whatever it is, it needs to be signed. Keyfactor Signum provides integrations to common signing tools (e.g., SignTool, jarsigner, Cosign, etc.) to support that. Because Keyfactor is hosted in the cloud, it allows users to sign from anywhere securely.

Keyfactor Signum also offers different authentication models through a SAML/OAuth agent and an admin console. Through the console, admins have control over and visibility into signing policies and can grant approvals for who can do what, and have access to audit trails to see actual activity.

Altogether, this architecture offers a powerful solution that makes the code signing process easy for developers while giving admins all the control they need to centralize management and ensure keys are appropriately protected.

Finally, how does all of this fit into the CI/CD pipeline? With a solution like Keyfactor Signum, it’s possible to make it so that any binary that goes into your repo is signed, full stop. Here’s what the process looks like:

Once a developer commits code into the CI/CD pipeline, it goes to the source repository and your CI server (whether that’s Jenkins, Azure DevOps, or anything else) runs unit testing. From there, the code gets signed using the Keyfactor Signum client and then it gets stored in the repository.

Going forward, whenever anyone pulls from the repo, that code gets verified. Tools like SignTool, jarsigner, and Cosign all have a verification component built in to check that the code is signed by a verified certificate based on the correct root of trust. This means that even if someone were to create their own signing certificate, it would be outside the designated root of trust, so any code signed with that certificate would fail in the verification stage.

Once code passes the verification stage, integration and load testing can begin, and ultimately, that code can move to production. The key component of all this is that no matter when anyone pulls from the repo, there is always a verification step first before anything else can happen.

Ready for even more on integrating secure code signing into your CI/CD pipeline and how Keyfactor Signum can help? Click here to watch the full webinar, including a live demo of Keyfactor Signum.