Vulnerability Management

🚀 As explained by Laura

We build our companies, including our own software, on top of other software. That makes sense—nobody has the time or resources to build every component of their business from scratch. That would be a real waste and would stop us from focusing on what matters most. As a result, our software is much like Russian dolls: bigger things containing much smaller things underneath it all.

Figure: Traditional nesting Matryoshka dolls. Credit: Dennis Jarvis (Wikimedia Commons)

There are many benefits using third party components—including libraries and frameworks—in our systems. They allow us to build on others’ expertise, while we focus our development efforts on our project’s unique challenges. Established patterns and standards can emerge that, over time, increase the quality of the entire community’s code, and as consumers of that code, your product benefits too.

​important​ If we accept that all development teams can make mistakes, then we must accept that all software must have flaws. This is important to state upfront. It’s not about your software. It’s about all software. All software is potentially vulnerable.

Vulnerabilities can exist in the code we write, the code we use from third parties, the codes we integrate with, and the tools we use to build and deploy. There’s an active community of researchers involved in the discovery of vulnerabilities as part of their profession. While it’s unlikely that you will interact directly with these groups, it’s a good idea to know they exist.

In this chapter, we look at how the vulnerability discovery process works, and how it affects our use of these third-party libraries and components.

How Vulnerabilities Are Discovered and Tracked

Let’s have a look at the vulnerability discovery process together.

Figure: Vulnerabilities are continuously found, published, and analyzed by the security community.

1. Search: Security researchers identify software of interest and focus on finding vulnerabilities. It’s like being a treasure hunter, every day looking for one little bit of a clue to find your next vulnerability.

2. Discover: A vulnerability is identified and tested. This could take days or weeks or months, depending on the complexity of the technology and the skill of the researcher.

3. Share: When they are ready, the researcher shares the details. They send the details of the flaw and an assessment of its risk to an organization such as the US National Institute of Standards and Technology (NIST), which collates and disseminates details of known vulnerabilities. Other organizations exist around the world serving the same role.

4. Publish: The vulnerability is assessed by NIST and published within the directory. RSS feeds notify subscribers that this vulnerability has been found.

5. Analyze: Vulnerability researchers examine the new flaw, and may expand or adapt on the published vulnerability to create proofs of concept and attack tools, or identify further vulnerabilities.

And so, the cycle continues.

Being Aware of Current Vulnerabilities

There are a range of great sources to use to keep up to date with security vulnerabilities: social media, vendor websites, CVE Details, RSS and news feeds, newsletters, podcasts, and so on. Please remember though, with each of these places, they each have a different motivation for sharing vulnerability information.

Table: Ways to Learn about Vulnerabilities

Case Study: Log4J Vulnerabilities in 2021

For a really clear picture of how this process works and why it’s important to your company, there is no better case study than the Log4J vulnerabilities identified in late 2021.

A standard open-source logging library for the Java language, Log4J is the de facto logging choice for a huge number of applications around the world.

In late 2021, researchers identified a remote code execution vulnerability in the source code for this library. They filed a vulnerability disclosure to both the Apache Software Foundation and NIST, resulting in a worldwide response.

Within hours of the disclosure, people and bots were actively scanning any site on the internet. We saw significant scanning activity that started quickly and ramped up over days.

Due to the relative ease of the exploit and the difficulty in closing all permutations of the attack, there was no choice but to patch the software itself rather than try to fend off attacks at the perimeter.

As more became known about this vulnerability, it became clear that Log4J was embedded into a significant number of applications globally and a significant effort would be needed to keep these applications and their data safe. In the US, this effort was led by the Cybersecurity and Infrastructure Security Agency (CISA) and included the creation of a GitHub repository for the application development community to collaborate in and share recommendations for remediation, as well as confirm affected software and companies. Check out this timeline of events as documented during the first weeks of the issue for a clear understanding of how quickly things progressed from vulnerability to exploitation, and finally to remediation.

​important​ Some key points to take from this case study:

• Software we rely on to do mundane but essential tasks in our applications and software may have vulnerabilities that can be discovered at any time if a sufficiently motivated vulnerability researcher exists.

• These vulnerabilities not only affect the code we write, but also the applications we use to run our businesses—from our office suites to our accounting tools. The world runs on software, and our businesses are exposed to risk from every tool we use.

• Watching security news or other technology information sources is essential for leaders of growing companies and for the engineers and security specialists in the more established companies.

• These risks are not theoretical and, once identified, you need to have a plan of how to respond.

So, let’s spend some time looking at how exactly we can respond to an issue like this vulnerability, and what we can do to fix or adapt if we are affected.

Responding to a Security Vulnerability

If you’ve been notified that a tool or technology you use has a security vulnerability, there are actions you can take and mitigations you can put in place.

Step 1: Research the Vulnerability

Before we act, it’s crucial that we understand the risk this vulnerability poses to our organization. This starts with asking the following questions:

• When was it found?

• What are the details?

• What is the severity of the issue?

• What is the impact of the vulnerability if it is exploited?

• Are there any proof-of-concept tools?

• Have there been reports of compromises using this vulnerability?

Using the news, online communities, and our own research powers, we can answer these questions to understand the scope of the problem. We are not trying to decide if we will take action. We are trying to decide when we take action.

Step 2: Understand Your Exposure

Now that we understand what the impact would be if this vulnerability was exploited, we’re going to try and map out how this technology interacts with and affects our processes, systems, and customers.

• Map the internal impact and exposure. We’re going to map out our business and where this technology touches. This can be a tricky and manual process. Don’t underestimate the complexity needed to get this working.

• Map the external impact and exposure. Finally, we’ll review this technology’s impact on our external elements such as customers, third-party tool providers, and suppliers. Does it interact with our systems or products? Does it relate to our customers or processes related to them?

Step 3: Make a Plan

At this stage, we start to understand our options. These questions will allow us to decide the next steps to take:

• Is there a patch available?

• Has it been tested?

• What else would the patch change?

• Do we have the access and resources to act?

​confusion​ Remember, security is often a footnote in a patch. Companies are not obliged to disclose software vulnerabilities in their products. So please pay close attention. We’re not joking when we say the presence of new emoji in a patch is more likely to get front page news than the security vulnerability that’s hiding three lines below it.

Step 4: Take Action

The next mitigation we need to discuss is patching and updating. In simple terms, this means applying a patch (a change to the code that makes a technology work) using a process to update the version of the technology we are using.

As soon as the patch is available, we’ll update our software. This is the best-case scenario, as sometimes it’s not possible to update your software.

When You Can’t Patch a Vulnerability

There are a few common reasons for not being able to update a technology. In this section we’re going to be pragmatic. We know that you don’t live in a perfect world and you’re making some tough decisions.

• You can’t make the change. It could be that the technology or the vulnerability is in a legacy project, one that’s no longer supported. Or in the worst-case scenario, if you don’t even have the code for that system anymore (it was lost in that disc failure in 2003), it belongs to a third-party contractor or an outsourced company.

• It’s a breaking change. The security change might be small, but it’s included in a large batch of updates that takes you up to a major version of a library. In this case, the amount of work to implement the patch is likely to be significant.

• There’s no patch available. If the vulnerability is an open-source library or framework, it could well be that the community doesn’t have the expertise, resources, knowledge, or even time to get to that vulnerability and patch it. There are still major software systems that have known vulnerabilities in their stacks and have done for some time. Being a big company doesn’t mean you’ll instantly patch in a well-behaved manner.

So, if we can’t patch the vulnerability, what are our options?

Approach: Ignore It

It’s a risky plan, but for completeness, we should mention what to do if you choose to do nothing.

We don’t think you’re going to take this approach to your software security practice, but it feels like doing nothing when we choose to ignore a vulnerability.

This approach tends to apply to legacy and unsupported applications. Many organizations have applications or tools in their environments that are no longer supported, or for whom support is a complex subject.

Sometimes these applications have exceeded their lifetime and are no longer supported efficiently. Your organization may choose not to patch and to try to encourage people to use a new version of a product.

You can see this behavior and action if you follow the release cycles of Microsoft Windows and other operating system products. At some point, they choose to stop providing support and security updates, because otherwise nobody would ever upgrade. If nobody upgrades, they will be stuck supporting old and less innovative products forever.

If you choose to not update the technologies in your environment or the technologies used to build your product, you need to think clearly about the impact this choice will have on the overall vulnerability of your company and its systems, people, and customers.

In most jurisdictions, when company directors choose to intentionally ignore vulnerabilities that could harm the business or its customers, they open themselves up to liability if this vulnerability is later exploited. Check with a lawyer in your country or operating region to understand your directors’ responsibilities when it comes to protecting your organization from cybersecurity threats.

Approach: Change Technologies

We’ve discovered a vulnerability in a library we’re using, for example, the library that’s providing our database connectivity.

What are the advantages of changing it entirely for something new? Maybe it will provide new, updated functionality as well as improved security practices. It could be an opportunity to modernize our technologies and, in limited cases, it might save you money or complexity. However, this comes at a high cost.

None of the advantages are guaranteed. All technologies have the chance of security vulnerabilities and the more modern the stack, or the technology, the less support and community may be available. There may be fewer people with the skills to understand and work with the technology, both internally and within the wider community. You may, as a result, struggle to hire people to fill roles for this bleeding-edge technology.

​danger​ When choosing a new technology, supportability matters. A common risk associated with rapid, unplanned adoption of new technologies or dependencies is around support.

New technologies often don’t have the number of skilled engineers to support them to the same level. This impacts recruiting, retention, and support related to stress in our teams.

​Definition​ Simply put, if you don’t have enough engineers with the right skills, the engineers you have will become burnt out and you may have a very hard time recruiting replacements. We call this problem key person risk. This is the risk from overreliance on a single person or a small group of people to complete critical activities or maintain key systems.

Have a think about your world.

• Are you the only person that supports a tool, process, or system?

• Is there a part of your world that would struggle to function if you took a day off, or won the lotto and decided to move to an exotic island?

If you answered yes to this, then you are a key person risk. Part of securing your organization is to identify all the key person risk in your company and reduce it, enabling everyone on your team to take that dream holiday should the chance arise.

The lesson is to not avoid emerging technologies, but to consciously embrace them, knowing the challenges they may bring.

Approach: Layer Your Defenses

When defending our applications, the part that is exposed to the world is what we call the attack surface.

Figure: Layers of protection.

Deep inside your organizations and your applications lie your data, the precious things in the center of your system. Every time we add software or hardware around our data, we create another layer, a separation from the internal space.

A common approach in security is to layer more controls around our system to protect the inner, more precious workings. This is the digital equivalent of building a very strong wall around the edge of the castle, with a hope of keeping out invaders.

Now there are many reasons that this approach is only part of the picture. But for now, when we’re talking about layering defenses, I’m going to talk about putting something between you and the outside world to stop malicious attacks.

In security, we assume that every step we take to protect something has a chance of failure. By layering our security measures, we hope that when this happens, other controls will prevent the compromise from progressing.

Three Types of Security Controls

Layers also allow us to introduce different types of control, each providing a different set of options for our defense. In our previous diagram, adding another layer on the outside is another challenge for an attacker to get through before they get to our most valuable items in the center.

​Definition​ The first type is preventative controls, which are items that stop attacks from happening.

​Definition​ A good example of a preventative control is a firewall, which is a device that sits between your internal network and the wider internet. It monitors and responds to requests entering and leaving your network to ensure that potentially malicious activity is stopped and only those expected and permitted connections are allowed through.

​Definition​ Detective controls try to spot incidents or malicious activity. They won’t necessarily take action, but they may alert you that a bad thing is happening and you should do something about it.

​Definition​ Finally, responsive controls (or corrective controls) assume that a bad thing has happened and allow us to respond quickly and efficiently to minimize the impact and to maximize our return to operations.

As you develop the security of your organization and its systems, you will gradually introduce each of these types of controls as layers to your defense, creating a comprehensive set of protections suited to your environment and the data you are trying to protect.