Most software and firmware developers recognize the importance of code signing. However, they struggle to implement it effectively and securely. This guide explores common threats and mishaps in code signing operations. We then dive into step-by-step guidance on how to address the needs of your developers and security team — all in one place.
What is Code Signing? The Definitive Roadmap to Secure Code Signing
Practical Code Signing Guidance for InfoSec and DevOps Teams
What is Code Signing?
We live in a world that runs on code. Software permeates virtually every aspect of our lives, from the things we use each day to the critical infrastructure of our society. It’s difficult to imagine a business today that doesn’t depend on software in some way. As the IoT continues to grow, software only becomes more embedded in our physical world.
There was a time when users could trust the software they downloaded, but that is far from true today. As hackers become increasingly adept in the art of spreading malware, even IT professionals find it difficult to know whether the software or the product they’ve purchased is legitimate. This is where code signing comes in.
Code signing is a cryptographic method used by developers to prove that a piece of software is authentic. By digitally signing apps, software, or embedded firmware with a private key, the proof is provided to end-users that the code originates from a trusted and legitimate source and that it hasn’t been tampered with since it was published.
“When you sign a piece of code, you make a statement that the software came from your organization and that you stand behind it.” Ted Shorter | CTO & Co-Founder, Keyfactor
How Code Signing is Used
Enterprise IT teams must ensure that any internal scripts or utilities applied across the business are signed to prevent tampering by internal users or external threats.
In the emerging Internet of Things (IoT), code signing is the most effective way to ensure the integrity of devices from activation through firmware and software updates.
Mobile App Developers
Popular app stores from Microsoft, Google, and Apple require mobile apps to be signed before they can be submitted and published for purchase.
Developers of software are often required to sign code to support installation. Operating systems like Windows and macOS will warn users if software or drivers are not signed.
Why Code Signing Security Matters
Code signing without securing your private keys can expose you to more risk than no code signing at all. Attackers seek to compromise these keys to sign and distribute malicious code to your customers – masked as legitimate software or firmware.
If the private keys linked to your code signing certificates are compromised, it’s game over. Stolen code signing keys are the top prizes for hackers – either sold or used to create signed malware that appears be to published by your developers.
Hackers don’t need your keys to sign malware. If build servers or developer workstations with unhindered access to code signing systems are breached, an attacker can simply submit malware to be signed and distributed without detection.
Developers specialize in code, not security. Code signing keys and certificates can easily be misused or misplaced by developers, making it much easier for would-be attackers to undermine the integrity of your code signing operations.
The Impact of Code Compromise
Software publishers, device manufacturers, and in-house development teams have adopted code signing as a way to protect their intellectual property, their company brand, and their end-users. But the trust and integrity of code signing hinge entirely on the security of your keys.
A single breach in this “chain of trust” can bring your entire business to a halt. Efforts to quickly revoke and re-issue certificates, notify customers, and push out a newly signed update are expensive. This results in revenue loss and the cost of re-establishing trust with your users, partners, and investors.
“Organizations experienced an average of 4 incidents involving the misuse of code signing keys and certificates in the past two years. The average cost for these incidents was $15 million.” – 2019 Ponemon Report, The Impact of Unsecured Digital Identities
Know Your Risks
Recent code signing attacks underscore the importance of managing reputational risk. Whether you consume software or sell it, all business leaders need to invest in the trust associated with their digital brand – and expect the same of their vendors.
Hackers breached a build server with access to the Adobe code signing system. Once inside, the hackers used the server to sign malware with a valid code signing certificate. Adobe responded by revoking the certificate, impacting three of their applications and millions of users.
Bit9 received reports from its customers that malware was discovered in their networks – malware that was digitally signed by Bit9’s own code signing certificate. It was later discovered that hackers infiltrated a virtual machine and stole a certificate to sign and distribute the malware to at least three Bit9 customers.
Hackers don’t always have to steal a valid code signing certificate, especially when manufacturers unknowingly publish it themselves. In 2015, developers at network equipment manufacturer D-Link accidentally published four private code signing keys in open-source firmware. No known malware was signed with these keys, but D-Link received harsh criticism for the incident.
An Advanced Persistent Threat (APT) group based in China, known as Suckfly, stole code signing certificates from at least nine different companies in South Korea. The group then used these certificates over two years to sign hacktools and malware in a targeted campaign against companies in India.
Taiwan-based tech companies D-Link and Changing Information Technology were targeted by another APT group known as BlackTech. At least two code signing certificates were stolen and used to sign Windows malware known as PLEAD, which steals passwords entered into the web browsers of infected machines.
ASUS – a well-known manufacturer of laptops and mobile phones – unknowingly pushed malware to thousands of its customers for at least five months. This sophisticated software supply chain attack became knows as “Operation ShadowHammer.” Hackers compromised two ASUS code signing certificates and pushed out signed malware through the ASUS Live Update Utility, inserting backdoors into at least 1 million devices.
“We saw the updates come down from the Live Update ASUS server. They were trojanized, or malicious updates, and they were signed by ASUS.” – Hackers Hijacked ASUS Software Updates to Install Backdoors on Thousands of Computers
Roadblocks to Success
The good news is that most independent software vendors (ISVs) and device manufacturers recognize the importance of code signing. The biggest challenge is how to implement it in a way that effectively meets the needs of both developers and IT security teams.
Speed vs. Security
Security and PKI teams would prefer to isolate and lock down private keys, but developers need quick access to sign code and push it to production. The biggest problem is how to implement safeguards to prevent misuse of keys and certificates without impeding the productivity of your developers.
Software supply chain attacks are becoming increasingly frequent and sophisticated. Hackers, cybercriminals, and even state-sponsored attacks put the security and integrity of your software at risk. As the cost of code signing certificates on the underground market continues to rise, some attackers have taken a more direct approach.
DevOps & Agile Development
DevOps practices have taken the IT world by storm. Fast and frequent incremental software builds are the name of the game. Any changes to the Software Development Lifecycle (SDLC) can introduce more risk than they aim to prevent. Security must adapt to existing DevOps workflows and signing processes.
Dispersed Dev Teams
Today’s development teams collaborate across globally dispersed locations. When a remote team needs to sign code, the easy solution is to purchase a signing certificate. Once purchased, certificates are often left within reach of hackers and out of the purview of your security team on disparate developer workstations or build servers.
The Roadmap to Secure Code Signing
Enough theory. Let’s dive into the four practical steps your organization can take to overcome these challenges and the right solution to help you get there.
01 | Safeguard Private Keys
Private keys that developers use to sign code are invaluable to hackers. Once compromised, these keys can be used to sign virtually any code and distribute it to thousands of users. These types of attacks have become more and more frequent, as hackers seek to evade malware detection tools.
No industry or enterprise is immune. Trusted vendors in security and hardware such as Bit9 (now Carbon Black) and D-Link have fallen prey to persistent attackers. These attackers successfully weaved through their network to find and compromise code signing keys and used them against their own customers.
The burden to sign code often falls on developers that specialize in writing code, not securing keys. As a result, keys wind up in unsecured network locations such as developer workstations, build servers, and who knows where else. IT security teams are often left unaware of exactly how many code signing certificates they have or where they are stored.
Get a Complete Inventory of your Landscape
Digital certificates and keys used for code signing are high-value assets, yet 71% of organizations don’t know exactly how many they have. Start first by taking an inventory of how many code signing keys you have, where they live, and how they are stored.
Centralize Management of Keys & Certificates
Local code signing creates siloes in security and increases costs. A centralized, server-side solution can simplify administration, improve security, and eliminate the need to have a separate code signing certificate for every developer or build stream.
Store Private Keys in a Certified HSM
Hardware security modules (HSM) are the most effective way to ensure that your private keys remain under your control. Keys can be stored or generated inside the HSM and used to sign code anywhere without ever having to leave the hardware.
Choose a Flexible Solution
Choose a solution that is easy to deploy and scale as demand for code signing grows. Cloud HSM services offer all the benefits of hardware-level security, but without the upfront expense, manual set-up, and ongoing maintenance of dedicated hardware HSMs.
02 | Secure Signing Operations
If a hacker breaches your developer network, they don’t need to steal your keys. By gaining access to a build server or developer workstation with access to code signing infrastructure, hackers can simply submit malware to be signed and distributed without detection.
Better known as “software supply chain attacks,” these threats are even more difficult to detect because they often involve either an insider or an attacker with direct access to code signing. Even the likes of Adobe and ASUS with sophisticated security teams were unable to detect breaches in their code signing infrastructure for months.
Attackers will always find the path of least resistance. Storing private keys in an HSM reduces the risk of key theft, but you can bet that determined threat actors will find another way. In the case of Adobe, private keys were stored securely in an HSM, but hackers instead gained access to a build server and simply requested signatures for malicious code.
It’s absolutely critical to ensure that only the right developers can sign the right code – at the right time. By allowing only authorized users to sign and approve code, you can ensure that even if a hacker does breach your network. This prevents hackers from gaining access to your critical code signing infrastructure.
Define Roles & Separate Duties
Establish separate roles for those authorized to submit code for signing and those authorized to approve signing requests. Dividing these duties will help ensure that only trusted users can sign code, keeping hackers out of signing operations.
Enforce Code Signing Policies & Practices
Ensure that even authorized developers are only granted access to sign code for a defined duration of time, number of signatures, and other parameters to prevent illegitimate signing or internal misuse. Look for solutions that can help you enforce these policies.
Segment Test & Release Signing
If code is signed during the development and testing stages, certificates should be distinct from those used in production signing. Ensure that development keys are not linked to the same root of trust as keys used to sign production code.
Track Signing Activities
Development or release managers should track every use of private keys to sign code. This will help you to ensure that only the right developers are signing the right code, with the right keys.
03 | Integrate with DevOps
It’s no knock against developers to say that security isn’t their top priority. In order to stay ahead of the digital curve, development teams must move fast to write code and push it to production. However, security teams are putting more and more pressure on developers to meet requirements that often clog up the build and release process.
Responsibility to secure code signing keys shouldn’t fall on developers – they simply need access to submit and sign code quickly. It’s about finding the right balance between IT security requirements to lock away private keys and developer’s needs to sign any code, from anywhere, without disruption.
Formalize your Code Signing Process
Document, track, and rigorously follow the steps required to sign code as part of your software development lifecycle (SDLC). Define what checks and verifications (i.e. QA, pen tests, virus scans, static analysis, etc.) must be performed before signing.
Cover Distributed Dev Teams
The ability to enable remote signing is critical with today’s distributed development teams. Choose a solution that enables developers to sign code from anywhere, without the keys ever leaving the secure confines of hardware protection.
Minimize Changes to the SDLC
Take a collaborative approach to ensure that security and development teams achieve mutual goals to protect keys without disrupting the SDLC. It’s important to find solutions that adapt to your processes, not the other way around.
Integrate with Signing Tools & Platforms
Directly integrate code signing processes with existing tools and workflows whenever possible. Code signing should support multiple code file formats, platforms (i.e. Microsoft Authenticode, Java, etc.), and certificate authorities (CA).
04 | Monitor & Audit Compliance
Implementing security is one thing, but keeping developers in compliance with your code signing policies and best practices is another. Code signing security is a continuous process, not a set-and-forget deployment. Certificates inevitably expire, keys and algorithms weaken over time, and threats continue to evolve. Security teams must be able to identify these risks quickly and effectively respond.
Monitor Code Signing Operations
By monitoring code signing requests, authorizations, and signatures in real-time, security teams can more effectively detect anomalous activity and respond within minutes, rather than months.
Log & Audit Key Usage
Keep a comprehensive audit log of who used code signing keys, when, and who authorized the action. Review logs regularly for suspicious activity and ensure that these logs cannot be tampered with.
Include Code Signing Certificates in CLM
All digital certificates in your organization should be governed by certificate lifecycle management (CLM), including Extended Validation and Standard Code Signing Certificates – with provisions for how they are requested, issued, renewed or revoked.
Enterprise Code Signing with Keyfactor
Keyfactor Code Assure is the only platform that gives you complete visibility, control, and protection of code signing operations, without disruption to existing build and release workflows. Code signing certificates and keys are stored centrally in a tamper-resistant HSM. Once inside, the private keys never leave the HSM. Robust APIs enable developers anywhere with quick and controlled access to perform code signing, while security teams retain a full audit trail of code signing activities.