This article is a guest submission by Nathan Koester, Staff Security Engineer at Ironclad.

In a typical software product, there are hundreds of dependencies, both direct and indirect. This is an accepted truth, with everyone touting their Software Bill of Materials (SBOM) as effectively maintained. How then, does an organization address management of 3rd party dependencies?

Whether it's java, python, golang, ruby, rust, c, and so on, every single application is built on something else. Because of this, our job in security is to give the best guidance we can to our engineering and development teams on how to proceed, with the normal comment being "do your best".

Security's goal is to minimize risk, platform teams are focused on minimizing downtime, while development teams focus on speed of releasing fixes and new features. With all of the teams in conflict, how can we instill a sense of harmony, where we can address the security risk, while building partnerships with the other teams?

First, let's start with what a dependency is: a critical component in the application that was developed externally. This can mean it was built in house, it's an open source library, or somewhere in the middle. Regardless of where it came from, the challenge is the same: code we don't control is being included, and it's our task to reduce the risk they introduce.

Whether your dependencies, libraries, system packages, etc, are discovered via Software Composition Analysis (SCA), container scanners, or some other method, they provide us visibility into our environment.

Let's take a look at the below image of a simple python script with an external library:

Now when we leverage a tool, such as 'pipdeptree', we can see that utilizing "requests" brought with it 4 additional libraries, none of which we're maintaining. From a developer perspective this is standard practice, while from a security lens, it's additional attack surface that most teams never assess. Our guidance at this point, would be remove unnecessary packages, pin minimum or fixed versions, or find some other library / implementation.

Here's the exact same script as above, but using urllib3, a base package for python requests:

With this simple change, we increase our code complexity while removing a chain of dependencies. If we're going to go the route of "Remove and Re-Write" then we need an extremely strong business case. A few such examples where this may be ideal could be: open source licensing conflicts, deprecated versions, removing excessive libraries to develop software for embedded hardware with finite constraints.

Most of the time, rewriting isn't realistic. Tracing a dependency chain 3-5 layers deep, figuring out which transitive library introduced the problem, and then determining whether upgrading a parent dependency even resolves the issue downstream is its own project. So you're left figuring out where in the chain to patch, and that's where it gets complicated.

But now that we know what dependencies are, and how they can be introduced, let's look at how we address this problem:

• Prioritize the most critical, quick wins, or the common denominator
• Provide really clear remediation steps to make it painless

Before going into the how and why, we should first review "who" cares. Security teams obviously care because of their tools, including SCA, CNAPP, EDR, and so on. Other than security teams, though, we have to remember that platform teams and development teams are the most critical stakeholders. The sooner they are involved in the remediation pipeline, the better. Now normally these teams don't have access to security tools (and for good reason), but we also don't do a good job of coordinating with them. Shouting "Library X is vulnerable and needs patching" helps no one, and burns more bridges than it builds.

Honestly, the security team is the last team to remediate the process, but beyond that there's no clear guidance. It really boils down to "where" the critical components are, and who owns that stage. If it's in the code, such as a gradle include or a python import, then the software engineering team is the most relevant. In order for them to patch this, though, they need clear guidance as to what version addresses the problem, and how critical the path forward is. If you can clear that hurdle, you've won a trusted partner.

Now if the issue exists in system libraries, such as openJDK, stdlib, log4j, and so on, then the platform team is your point of contact. These are the people responsible for maintaining the base infrastructure and focused on uptime. Telling them they have 24 hours to resolve the gap isn't feasible, and just adds more headache, so instead focus on giving as much runway as possible. If there's a critical vulnerability, come to an agreement on the minimal amount of time it takes to address (hotpatches are one thing, but image rebuilds take a while for full testing). If it's 30 days, start the timer, and get out of their way. You don't just sit there, though, you ensure you have as many compensating controls as possible, review your detection tools (SIEM, EDR, CNAPP, etc), and monitor for any hiccup while the platform team works their magic.

Now we know the who, let's discuss the approaches for "how" we can patch.

If you're tasked with managing the mess, and if the "Remove the overhead" approach doesn't make sense, for any number of reasons, our next consideration should be if we can "pin versions". This means that enforce minimum or static versions within our package managers. This could be due to the latest version of a package is considered "alpha" and is unstable, or it could just be that the newest version is a complete re-factor and hasn't been tested in our application yet. This is a key point of patch management, that we ensure a new implementation doesn't change our processing.

On the other hand of version pinning, is that we're not getting security patches or bug fixes as frequently. This means that when a vulnerable package is identified (1st or 3rd party), then we have to go through the chain of dependency updating. To understand the headache that is dependency management, let's look at log4j on Amazon Linux 2 (a well studied issue at this point).

Given a scenario of running a Java application on Amazon Linux 2, what is the first step you should take when attempting to address the vulnerability in log4j? You have your Operating System limitation, the java development kit restriction (what versions exist on Amazon Linux 2), openJDK vs Amazon Corretto, log4j versions, and even if you should just remove log4j entirely.

First step: understand where in your code log4j is being called. This can either be through a direct import, or through a shared logging library. If the code is required for functionality (development environment debug vs production system error handling, for example), then you cannot remove the library.

The next step, then, is to understand if patching it is feasible. If the latest version is 2.14, and doesn't resolve the issue, then you're out of luck with updating, but what about downgrading?

But even once 2.15 dropped, the work wasn't over. Does 2.15 actually fix the issue completely? It didn't. 2.16 was needed for a full fix. Is 2.15 compatible with our JDK version? Does our logging framework support it? This is the part that doesn't get talked about enough: even when a patch exists, figuring out whether it actually works in YOUR environment is its own project.

What about downgrading? The CVE reported 1.X versions were not vulnerable to the latest CVE, but they did still have their own risks. So downgrading is one option, but then you have to either manually rebuild your library depending on log4j, downgrade to an older library and potentially introduce more risk, or just assume the risk, again. But let's say 2.15 gets released, known to address the issue, and your logging library gets updated. However now you have a problem, 2.15 is only present on Amazon Corretto 17, and you're on Amazon Corretto 11. So now you have to update, and go through testing, version pinning new library versions, and do performance testing for stability. All for Amazon to release their own hotpatch for log4j that resolved the vulnerability for other Corretto versions, without having to update anything.

As you can see, all it takes is one package to break, to throw more than a handful of wrenches. As a security practitioner, your job is to understand your organization, evaluate risk, and prioritize remediation and mitigation efforts.

When I was dealing with log4j, we burnt bridges. Not because it was confusing, but because everyone had a different approach. Some members on my team advised rewrites, some advised compensating controls, some advised downgrading. No one was more "right", but the wrong answer is "do nothing".

The reason the bridges were burnt was because we sounded and conveyed panic, because we kept having to push new changes. Yes, Log4j is an extreme scenario, but it shows what happens when a vulnerability is discovered across multiple versions, and you have no effective guidance into how to handle it.

The ideal scenario is that an organization never faces patch management issues, regardless of where dependencies are introduced. Unfortunately, though, reality is rarely ideal. Instead, plan for the reality that your first recommendation won't always be the one that works. Build a process that can adapt when the guidance changes, because it will.

Your goal, when it comes to patching, is preferring a pragmatic approach: pin minimum and maximum versions, require end to end testing for all library changes, and performing routine reviews. If a library is no longer needed, remove it from the package management system. If a major dependency releases a new major version (1.X to 2.X), do a review on the timeline before your version loses support, and provide an upgrade path to migrate. This will require analyzing and testing all new dependency versions, but this can be part of your platform team's process. Let the platform team handle evaluation, stability testing, performance monitoring, and handling all infrastructure changes. You'll be able to understand the change in risk, scope of impact, and you'll already have a process defined should a major upgrade be required for a hot patch or other major security risk.

Here's a practical checklist that's worked for me:

Pin minimum AND maximum versions. Minimum keeps you off known-vulnerable releases. Maximum prevents untested major bumps from sneaking in through automated updates.

Set up automated dependency update PRs with CI gating. Let the automation propose the change, let your pipeline validate it.

Track end-of-support dates for your major dependencies proactively. Don't find out your OS or JDK is EOL during a fire drill. Put it on a calendar.

When a CVE drops, check if the vulnerable function is actually called in your code before panicking. A lot of transitive dependencies include vulnerable code paths that your application never touches.

Build relationships with your platform team before a crisis, not during one. If the first time they hear from security is a P0 escalation, you've already lost.

The hardest truth is security rarely has all of the info. We don't know the reason libraries are included, we don't know what part of the code base exists for what reason, and we don't know why the platform team hasn't upgraded from the last OS version to the most "cutting edge".

So we shouldn't pretend to have all that insight when a vulnerability is discovered. We should provide our assessment and risk tolerance to the appropriate teams, and make an informed decision, together, as to how we should best proceed.

If it were up to me, I'd handle the burden myself. I'd pull the latest develop branch, get the container image, spin up a dev machine in our cloud infra, and I'd dig in. It'd take time to get the CNAPP scan for my setup, and I'm not guaranteed to be an expert on any one component, but I don't need to be. I've worked across a variety of operating systems, hardware, and languages that I feel comfortable investigating the where, the why, and the how. Your DevOps team should already have automations for configuring and deploying builds, so the hard part is out of the way, now you just need to embrace the discomfort. Unfortunately, to be honest, what I want to do and what I can do, they rarely align in security. At best, when working at scale anyway, I make a version change and see what happens in our build pipeline. In fact, I just went through this exact problem trying to patch a vulnerable library, only for half of our E2E tests to fail. I was immediately out of my comfort zone and had to defer to the platform team to get support, only to find out we're mid-migration from yarn to pnpm.

Your build environment normally has a list of packages, such as yarn.lock, requirements.txt, maven files, and so on, so knowing "what" gets included is easy. But the “where” is the hard part. You start using regex, you start using IDEs, and you just look for any file that calls that relevant import/include. This means investigating 3 layers deep, where a template file calls the initial include, but that template file is called in a series of calls. This is where an SCA tool could help, but a legacy SCA solution still doesn't confirm "risk", and definitely doesn't tell the whole story. Understanding if something is exploitable in your environment is a uniquely different question, one that requires a lot more context.

Is the file actually being leveraged, or is it just included? Sometimes developers will start including files and libraries before they're even used, just because "eventually" they may be needed. Technical debt exists everywhere, and you have to feel comfortable navigating it. If it's called, find the git commit that blames who included it, and reach out to them for confirmation. If they agree it's unused, rip it out. If they explain why it's there, ask if we can add guard rails (more sanitization, input validation, try/catch blocks, etc). As you can see, your first goal is to investigate, collect the evidence, and discuss. You're not the SME, and you shouldn't pretend to be. Ask the appropriate developers, get their input, and work with them. This is how you build strong relationships and earn trust in security.

Well, now that we covered the software product, let's look at the infrastructure. This is where things get messy. Are we using NGinx, Apache, Postgres, MySql, Mongodb, or anything else? Well, we're not the experts here, so let's reach out. Is a minor version sufficient to meet our needs? Or is it a full major upgrade? Or worse yet, is it a rip / replace sort of thing? Regardless of the issue, the same fact applies. Let's investigate.

If Apache 2 needs a critical update, but we're sitting on Debian 9, we're in a pickle. Debian Linux is famous for not moving quickly, and there's a reason we're sitting on a stable Debian OS instead of Fedora. Ask your platform team why we're still on Debian 9 instead of Debian 12, and how we can prioritize this fix. Sometimes we need to compile Apache 2's latest version from source: doable, but can be frustrating if it needs other depends on other libraries that aren't natively on Debian 9. So then it's go down the rabbit hole, compile all the dependencies from source, stress test everything, and perform End to End validation tests. If everything looks good, see if you can call it a hotpatch and push to production. Definitely not ideal, but neither is running a no-longer supported Operating System for a production environment. This is why you, as the security lead, need to push for routine maintenance, upgrades, and removing end of life / outdated images and software.

As you can see, hopefully, triaging issues is never the problem, it's understanding the business impact and level of effort of the component in question. This is why security is heavily focused on reactive scenarios, because you'll never have enough information going in, to be fully prepared. The more you can work alongside your teams, understand upcoming risks, such as End of Support software or changes to your devops pipelines, then you can respond quickly and efficiently. Your goal is to never solve the problem alone, but empower your counterparts to act accordingly with all the information you can provide.

The teams that handle patching well aren't the ones with the best tools or the fastest response times. They're the ones that invested in the boring stuff before the fire started: dependency inventories that are actually maintained, platform upgrade cycles that run on a schedule instead of on panic, and relationships between security and engineering that can survive a bad week. None of that is glamorous. But it's what separates a Log4j that burns bridges from a Log4j that's just a hard week.