Automated testing comes to the Linux kernel: KernelCI

[steven36]

Linux, the most important software program in the world, has a major weak point: Poor testing. That's changing, and one big sign of that is KernelCI, an automated testing framework for the Linux kernel, is now a Linux Foundation project.

 

 

At the recent Linux Kernel Plumbers get-together in Lisbon, Portugal, one of the hottest topics was how to bring better and automated testing to the Linux kernel. There, the top Linux developers united their efforts behind one testing framework: KernelCI. Now, at Open Source Summit Europe in Lyon, France, to help give KernelCI the resources it needs to be successful, it became a Linux Foundation project.

 

Here's how it works: As you probably know the Linux kernel is developed by a large, collaborative open-source community, which works through the Linux Kernel Mailing List (LKML). You can't argue with the method. But Linux kernel testing is fragmented -- since it is largely done in private silos without enough collaboration on the testing software or methodologies.

 

Part of the problem is how patches are done with Linux's mailing lists. Russell Currey, a Linux kernel developer, recently explained:  

"[Unlike a project based solely on GitHub or GitLab] where a pull request contains all of the information needed to merge a group of changes; an email containing, say, patch 7/10 lacks that context. It is nearly impossible to tell from an email message whether a patch series has been merged, rejected, or superseded. In general, mailing lists simply do not carry the same level of metadata as contemporary project-hosting sites, and that makes the CI [Continuous Integration] problem harder."

The specific problem? KernelCI was designed in the beginning to address testing Linux on a wide variety of hardware. Until now, when Linux patches were tested, they were done on developers' own machines. That meant you could be sure Linux would run as expected on mainstream hardware. But if your hardware wasn't that popular… Well, chances are it wasn't tested. 

As Greg Kroah-Hartman, the maintainer of the Linux stable branch, explained:

"Linux runs everywhere and on so many different pieces of hardware and but the testing on that hardware was very minimal. Most people, were just testing on the few things that they cared about. So we want to test it on as much hardware as we could to make sure that we're actually supporting all the hardware that we claim we're supporting."

Going forward, though, KernelCI will do more than just far more than just hardware testing. Kevin Hilman, KernelCI's co-founder, and a senior engineer at  BayLibre, explained in his Open Source Summit Europe keynote: 

"We got together at Linux Plumbers. One of the big problems that we have now is we have six or seven different code testing projects that were sending kernel developers and maintainers reports This was getting really annoying so we came together and said, 'pick one and use it as a framework' and so we've agreed on KernelCI so, we're all gonna work together, not to duplicate our efforts and results." 

By consolidating these testing projects and seeking common ground, the new KernelCI will also help address the problem of dealing with patches within the LKML. 

 

 

So, while there will still be many Linux testing suites, no longer will they stand alone without any real coordination between them. KernelCI's goal going forward will be not just to test a wider variety of devices but to unify all upstream Linux kernel testing efforts. Together, this will provide a single place to store, view, compare and track testing results.

 

"Provided how crucial Linux has become to society, achieving comprehensive test coverage of the Linux kernel is essential. By applying the open-source philosophy to testing, the KernelCI distributed architecture enables the whole kernel community to collaborate around a single upstream CI system," said Guy Lunardi, VP of business development at open-source consultancy and KernelCI contributor Collabora.

 

Put it all together, and KernelCI will improve Linux Long Term Support (LTS) kernel testing and validation; consolidate existing testing initiatives; improve Linux's overall security; and increase the pool of tested hardware. This, in turn, will improve the quality, stability. and long-term maintenance of the Linux kernel. And, that is a very good thing.

 

Source