What Linux kernel maintainers do and why they need your help

Everyone uses Linux, and companies are willing to pay to develop it, but maintaining Linux's code? That's another matter.

Monterrey, California: At the Linux Foundation Members Summit, Jonathan Corbet, Linux kernel developer and executive editor of Linux Weekly News, explained where the pain points are for Linux kernel maintainers and why it's getting to be a bit much for all of Linux's cooks.

Indeed, many Linux code maintainers are burning out. Why? There are many reasons. First, though, you need to understand what Linux kernel maintainers actually do. 

They're the people who take the code from thousands of developers, check it for errors, make sure there are no regressions, coordinate the code with the patches from other maintainers from further up and down the tree, and finally herd the patches toward the mainline. Oh, and manage backports. That's a lot of work, but it's only the "fun" code part. 

Maintainers must also mediate developer disagreements and interact with vendors and users. The latter can range from talking to hardware companies to try to get them to open-source their drivers, and assisting developers on how to build a driver, to helping a user having trouble with his laptop touchpad -- quite possibly because the aforementioned vendor never cooperated when the touchpad driver was first built.

The result? Well, as Darrick Wong, the former maintainer of the XFS file system, put it in his recent resignation note, "I burned out years ago trying to juggle the roles senior developer, reviewer, tester, triager (crappily), release manager, and (at times) manager liaison. … I thought if I could hold on just a bit longer, I could help to maintain the focus on long-term development to improve the experience for users. I was wrong."

Wong added:  

Most of my friends work for small companies, nonprofits, and local governments. They report the same problems with overwork, pervasive fear, and anger, and struggle to understand and adapt to new ideas that I observe here. They see the direct connection between their org's lack of revenue and resources. They don't understand why the hell the same happens to me and my workplace proximity associates when we all work for companies that clear hundreds of billions of dollars.

Before diving into this, let me point out that the maintainer burnout problem is not because the Linux Foundation isn't paying for maintainers. The Linux Foundation only has three maintainers: Linus Torvalds, Greg Kroah-Hartman, and Shuah Khan. That's it.

That's because the Linux Foundation is not a programming company. It's an open-source foundation that provides resources for businesses, organizations, and developers to build open-source projects. It's corporations for the most part that hire maintainers. 

But, as Corbet mentioned, almost no one -- no one -- pays people to be maintainers. Maintenance is extra work they do on top of their day job. Companies want their programmers to produce new code. They're not interested in paying them to help manage the entire Linux, or any other open-source project, fundamental infrastructure. 

The result? Cobet quoted Steve Rostedt, a Google software engineer and maintainer of several Linux kernel projects "Being a maintainer myself with a full-time job that is not to do my maintainership I'm struggling to find time to work on this."

Who wouldn't? I stand in awe of maintainers. I'm known as an extremely productive writer -- over 10 thousand articles published and counting -- but I couldn't possibly handle their workload.

But, wait! There's more. While Corbet welcomes the arrival of the Rust language into Linux, it will also add another burden to maintainers. "If you are a kernel maintainer, and you're going to start receiving submissions written in the Rust language, you have to understand the language at a very deep level. … This is a lot to ask a maintainer to learn this new language when they're already busy and overwhelmed with the work that they're doing, so this is gonna be a problem going forward."

Another problem is the increased use of fuzzers. These test programs inject semi-random data into a program/stack to detect bugs. It's a great technique for finding mistakes and potential security holes. 

But, recently, people without a clue are using fuzzers to find "bugs." So, Corbet observed, "Fuzzers generate thousands of reports. And even if all of these reports were good, somebody has to go through them or somebody has to understand them all. That task falls on the maintainer Add to this the fact that many of these reports are bad or duplicates and you have a huge waste of time." Worse still, "we have lots of people running fuzzers and cranking out reports without really verifying they found a real problem or helping in any way to solve these problems."

Just what maintainers need, more, and often pointless, work.

So, what's the answer? Address the maintainer understaffing problem. 

Corbet said: 

In a world where we have 5 thousand people contributing to the kernel every year, understaffing might seem like a strange thing to complain about. But maintainers are overworked. We're working for companies bringing in hundreds of millions of dollars a year in revenue, but companies that like to hire kernel maintainers don't always like to give them time to actually be kernel maintainers. They need to be evaluated for their maintainer work.

In short, "let maintainers do maintenance as part of their job, give them credit for it, because this is not happening and it's what we really need."

Companies need to recognize the value of maintaining not just Linux, but other important open-source projects. Successful development is not all about producing new lines of code. It's about making sure the entire codebase works well as a whole, and that means giving people the time and resources.

It's not just corporations that need to step up. Corbet added that developers need to review patches. 

Corbet explained, "If you are submitting patches to the kernel, you should be reviewing patches submitted to the kernel. If all you're doing is suspending code, then you're putting a load on the maintainer side of the equation without doing your part to help them."

So, with more support from both programmers and corporations, the great Linux maintainer burnout can be mediated before it becomes an even bigger problem than it already is. 

Source