The ultimate race condition: Securing open source infrastructure

Open source security tools are no longer experimental or only used by hobbyists. They protect some of the largest global enterprises and critical digital infrastructure. This evolution occurred more rapidly than anyone imagined or predicted.

Jason Chan, former VP of Security at Netflix said this about the culture of building and contributing to open source security tools, “Many of the tools we’ve released are aimed at facilitating security in high-velocity and distributed software development organizations. Automation is a big part of our approach, and we seek to keep our members, employees, data, and systems safe and secure while enabling innovation. For our team, scale, speed, and integration with the culture are the keys to enabling the business to move fast.”

As a cost-avoidance business objective, security is expensive and doesn’t generate revenue. It makes sense for companies to collaborate on building best-practice security infrastructures and tooling.

There is an uncanny similarity between physical infrastructure and digital infrastructure on which interconnected systems rely. Physical infrastructure allows companies to produce and deliver goods to consumers with a complex system of roads, rules, fuel, and electricity. Rarely do we pause and wonder who built the road we are traveling on, what complex path a product traversed before landing in our shopping cart, or all the technology that played a part in converting fuel into motion.

Digital infrastructure is similar. It uses packets, protocols, routing, and encryption to secure and ensure that companies can produce and deliver products. Businesses, like users of physical infrastructure, often don’t fully understand what allows them to run their operations successfully. Advancements in technology generally abstract away the complexities they are built on, and modern code design adheres to this practice.

As nice as phrases like “information superhighway” are, the comparison starts to break down at some point. The physical infrastructure we all so desperately rely on was paid for collectively through the shared burden of mandatory taxes. In contrast, open-source projects are maintained by volunteers and donations.

OpenSSL is a great example. Nadia Eghbal explained it best, in the example of encryption as the backbone of cybersecurity – “In 1998, a group of security experts in the UK got together to build a free set of encryption tools for the Internet. By 2014, two-thirds of all Web servers were using OpenSSL, enabling websites to securely pass credit card and other sensitive information over the Internet.”

What most don’t realize is that much of our digital critical infrastructure is also built on top of open-source tech. Kubernetes, another open-source initiative, has single-handedly ushered in a new era of microservice-built architectures. Who, or what is responsible for this new unstoppable force?

This XKCD is perhaps more accurate than we would like to believe.

Fortunately, organizations like the Linux Foundation, Open Collective, Buy Me A Coffee, and GitHub Sponsors are rewarding and incentivizing work on common digital technologies. The latter, for example, has had over $25 million contributed to open-source developers and projects since its inception in just 2019!

The world has accepted this as the future of digital infrastructure. Sharing, rather than building proprietary code, is simply much cheaper, easier, and more efficient. To illustrate, a survey asked organizations about their adoption of open-source software (OSS) solutions. In 2010, 39% of companies ran on open source. By 2015 that number skyrocketed to 78%. Now in 2022, in the midst of a digital transformation built on Apache projects, containerization, and Linux we’ve hit a critical mass of 100% and the survey questions are now looking to understand how rapidly organizations are “increasing use of open source”.

According to the survey organizers, “Out of 2,660 respondents to a recent global survey, 77% increased the use of open source software in their organizations over the last 12 months, and 36.5% indicated that they increased the use significantly.”

Startup technologies can now leverage countless shared libraries to build new and innovative solutions without having to write millions of lines of code to get base functionality and interoperability.

Benefits and drawbacks of relying on OSS

When LastPass, one of the world's most popular password managers, was recently breached, it sent shockwaves around the world. Huge amounts of trust were placed in the developers of the closed source solution. It was blind trust, and many of us were blind-sided. This was followed by similar breach of NortonLifeLock. Many security experts are now recommending adopting open-source password managers, like BitWarden, with code hosted on GitHub where everyone is free to review, audit, and contribute. Are there any downsides to this approach?

One of the biggest benefits of the open source approach is that the code is open for everyone to review. And ironically one of the biggest dangers of open source software is that the code is open for everyone to review.

The benefit and risk comes down to the intentions of the contributor. Are they looking to improve the code and make it more secure, or are they looking for and adding subtle vulnerabilities that might be used for exploitation? Or maybe they simply identify a slight nuance in the code that can be exploited, and they choose to not report in order to leverage it for some nefarious purpose.

The window may be small, as well-intentioned open source code contributors try to be diligent in finding these vulnerabilities before they are weaponized. It is the ultimate “race condition”.

According to a 2022 survey by Synopsis, 78% of the code used open-source libraries, and 81% of those contained at least one known vulnerability. While most of these vulnerabilities are unintentional, there is a non-zero number that are intentionally pushed into code updates as part of subversive adversarial behavior. This digital supply chain threat is real, and needs even more attention as we increasingly introduce software dependencies from untrusted sources into our codebase.

The problem goes deeper still as transitive dependencies (or dependencies of dependencies) create a complex dependency tree, making it more difficult to understand all the libraries involved. Consider the risk you may be putting your own downstream customers in. Having a Software Bill of Materials (SBOM) is now helping to address this challenge.

The Open Source Initiative's mission statement reads, “Open source is a development method for software that harnesses the power of distributed peer review and transparency of process. The promise of open source is better quality, higher reliability, more flexibility, lower cost, and an end to predatory vendor lock-in.”

So has this promise of OSS failed? The global panic caused by Heartbleed and Log4J vulnerability might make you wonder. Yes, the events of these past two years have been a harsh reminder of how much our security depends on the intention of others. This holds true for our digital supply chains and the open source libraries they rely on.

Can open source secure open source?

OSS has eliminated the historical alternative of salesmen in black suits and ties selling proprietary solutions. (Early IBM used to have such a strict dress code!) Businesses today have two primary objectives: innovate fast and innovate securely.

Many commercial vendors with closed-source software still have release cycles of 6-12 months, so even if a vulnerability is identified and fixed, it may take a long time to release the fixed version to the market. Even when the root cause of a breach is a vulnerable open source library, the fault often lies with the closed-source software on which it was built, but not updated when the patch was released. According to one source, over 95% of vulnerable open source components found in 6450 commercial software projects researched had newer versions that fixed the problem.

Open source has proven it’s fast, no doubt, but can it be done securely? Can we harness the power of community-driven innovation to solve the very security problem it has indirectly created? In other words, can open source security solutions secure and protect open-source software itself? Will more companies adopt the collaborative culture of companies like Netflix and use OSS to both build our software and also secure it at the deepest levels?

I’m reminded of the children's song in which an old woman swallows a spider to catch a fly, that she previously had swallowed. I’m not sure if that is the right analogy here. Maybe it’s more akin to a dragon eating its tail? No, not really. What about fighting fire with fire? That might work, but only if you view the fire as a positive creative force and not a destructive one.

Regardless of my poor analogies, the issue is now getting the attention it deserves, and it is quite exciting to watch some of the brightest minds on the planet set out to make the world a more secure place. SecOps teams finally have two viable choices when considering solutions to add to their stack, proprietary or OSS security.

Growth of open source security solutions

In the 2022 State of Open Source Report, respondents were asked about the type of open source software used in their organizations. The results show that OSS permeated every type of technology. Security did make the top ten, but lower on the list at eight, with 22% saying they used OSS in their security stack.

Interestingly though, when asked what the most important considerations are for adopting OSS in their infrastructure, security was top of mind with security and patches topping the list at 27% and stability and robustness coming in second at 26%.

Jim Zemlin, Executive Director of Linux Foundation said, “We believe open source is a public good and across every industry we have a responsibility to come together to improve and support the security of open-source software we all depend on. It is one of the most important things we can do.”

Whether you’re a vendor selling your wares, or an enterprise securing critical assets, you will increasingly find yourself having to choose between the two approaches. Generally speaking, experts have said that proprietary solutions need to be at least ten times better than open source project equivalents or you’d be better off choosing OSS. Again, from the report, the top-five reasons organizations choose open source software is telling.

1. Access to innovations and latest technologies

2. No license cost, overall cost reduction

3. To modernize technology stack

4. Many options for similar technologies

5. Constant releases and patches

One reason for the growing number of security-focused OSS solutions is the continuing “shifting left” occurring within SecOps. As we move to put more emphasis on securing software earlier in the development lifecycle, who better to create tools to accomplish this than the developers themselves? And when those developers are already building on and contributing to open source tools to create their environments, it is natural for them to create and look for open source tools to secure them.

Thus, DevSecOps has been a natural evolution, while traditional perimeter security has not seen the same shift. Security is no longer confined to the darkened room of a SOC, but rather spread across all teams in a new distributed and collaborative way.

The future of security

Open source security is the future. Moving security further left into the CI/CD pipeline is also the future. Democratizing the digital exhaust coming from all the layers of a modern application stack is a natural extension of that, and vital for application security. It is actually possible to unite these data feeds and this telemetry into a single workflow and analytics platform. And, drum roll… the new acronym I’m unveiling in this very article is OSSAPXDR. Open Source Security and Posture Extended Detection and Response.

Joking aside, Sumo Logic is one such platform. In fact, it’s one of the only cloud analytics solutions to be listed on both the Gartner APM and the Gartner SIEM magic quadrants. Sumo Logic spans CI/CD tooling, application monitoring & tracing, real-time correlation and detections, incident workflow, and a full-featured SOAR to connect them all together.

The above solutions are built on top of the same cloud-native technologies they are designed to secure. Sumo Logic is all in on supporting and integrating with open source.