Linux distros need to take more responsibility for security

Between the rapid release of open source software, and modern OSes preloaded with packages, enterprises are vulnerable to attacks they aren’t even aware of.

Open source is everywhere; a Synopsys study found that 96% of all software code bases analyzed included open source software. That’s the good news. Ironically, it’s also the bad news, as the very pervasiveness of open source introduces risk. Decades ago, proprietary players used to spew disingenuous fear, uncertainty, and doubt around open source security, but they may finally have a point. Not at the individual project level where critics once wrongly focused their case, but rather in supply chains, as massive vulnerabilities like SolarWinds and Log4j remind us that we still have essential open source security work to do.

Most enterprises have gotten very mature at network and perimeter security, but are still juvenile in their understanding and workflow around open source provenance and software supply chain security. Hackers have shifted their attention towards not only the security of individual open source projects themselves, but the gaps between software artifacts: their transitive dependencies and the build systems they touch.

We need to fix this, and the way to do so is arguably not at the individual project level but rather at the level of the distribution.

Timing is everything

“Basically open source got much more popular, and the front door got harder to break into so attackers are targeting the back door,” said Dan Lorenc, CEO and cofounder at Chainguard, in an interview. Bad actors, in other words, needn’t target your code. They can attack one of the dependencies you didn’t even know you had.

The cost of open source popularity is that a lot of the mechanisms of trust never really got built in at the onset. Linux (and other) distributions have played a critical role in the adoption of open source historically by doing a lot of the heavy lifting of packaging, building, and signing open source. Distros like Debian, Alpine, or Gentoo have well-deserved reputations as authorities, so users didn’t have to trust all open source blindly and got some guardrail guarantees.

But the pace of new open source packages being introduced has far exceeded the ability of distros to keep up. Even a single popular registry (like npm for JavaScript) gets more than 10,000 new packages per day. This basic mismatch between the pace of new open source technology and the relatively glacial speed of the distros results in developers going outside of the distros. They’re installing packages to get the latest and greatest as fast as possible but losing trust guarantees in the process.

It’s not that distributions have intentionally slowed the pace of progress; rather, they have to balance update speed with distribution stability. Still, given developer impatience, the distributions need to figure out how to accelerate updates and thereby keep better pace with the rampant adoption and security upkeep of open source software.

Security is hard

The Common Vulnerability Scoring System (CVSS) and other signals, such as the OpenSSF Scorecard offer great metrics on specific vulnerabilities and their severity. But modern operating system distributions ship with so many packages preinstalled that the average OS is flush with these vulnerabilities. If your car’s check engine light were on all of the time, how would you know when you actually needed to see your mechanic? The prevalence of vulnerabilities is so great across Linux distributions they’ve become easy to ignore.

Another problem is the semantic difference that occurs when developers install open source outside of distros and package databases. Modern security scanners all rely on this metadata, so security vulnerabilities go undetected for open source that is installed outside of the distro or package database.

What’s the alternative to these growing pains? Distros designed for minimalism and modularity can help improve overall open source security. By not including more than necessary to accomplish a task, distros can shrink the attack surface and help produce stronger supply chains. This, it turns out, is the start of something that may significantly improve open source security.

Evolution of developer workflows and tools

We’ve seen great progress the past few years in better establishing the security of open source projects. From the previously mentioned SSDF framework, to Sigstore and SLSA, multiple complementary projects have created developer toolchains for establishing where open source comes from, whether it has been tampered with, and other more reliable trust signals. This range of concerns is frequently referred to as “provenance,” and these open source projects have been aggressively baked into the major programming language registries such as npm, Maven and PyPi, as well as Kubernetes itself supporting software signing with Sigstore. Abstractions like eBPF and Cilium are also bringing software supply chain security visibility and enforcement closer to the Linux kernel.

In these ways, the open source ecosystem is hardening great primitives for making the provenance of open source components more native across developer workflows.

One particularly interesting technology to watch is Wolfi, an open source distro created and maintained by Chainguard, whose founders were cocreators of Sigstore and SLSA. Wolfi strips down the distro to its most essential components and introduces a novel rolling-release cadence so that only updated packages are available for download, and developers no longer need to download open source software outside of the distro.

This distro seeks to clear out all the nonessential packages so that when you see a CVE or CVSS score, you know it is a real vulnerability and don’t miss out on false negatives. With less code, fewer bugs, and fewer vulnerabilities, this slim-down of the distro also lets Wolfi give its users more severity-level data CVSS scores, plus support for new versions of open source software packages. On its one-year anniversary, Wolfi supports 1,300 package configurations and has gained the support of scanners from the major container security players such as Docker Scout, Grype, Snyk, Trivy, Wiz, and Prisma Cloud.

“Open source used to mean that you get a free copy of that source code forever,” says Lorenc. “Software doesn’t work like that anymore. You need a plan to constantly update every piece of software because of the rate of vulnerabilities being found. Software expires, and this is no longer a static problem, it’s dynamic.”

The next few years will be interesting to watch as the responsibility for open source security changes, with distributions showing the potential to deliver greater security by prioritizing speed. Enterprises are going to have to be much pickier about the open source they use, while also learning to be more nimble about how to find and fix the vulnerabilities they may already have.

Will it work? Definitely maybe. One thing is certain: We can’t keep relying on distro patterns that haven’t delivered the open source security enterprises need.

Source