Rapid restore tool being tested as Microsoft estimates 8.5 million machines went down
CrowdStrike's now-infamous Falcon Sensor software, which last week led to widespread outages of Windows-powered computers, has also caused crashes of Linux machines.
Red Hat in June warned its customers of a problem it described as "Kernel panic observed after booting 5.14.0-427.13.1.el9_4.x86_64 by falcon-sensor process" that impacted some users of Red Hat Enterprise Linux 9.4 after (as the warning suggests) booting on kernel version 5.14.0-427.13.1.el9_4.x86_64.
A second issue titled "System crashed at cshook_network_ops_inet6_sockraw_release+0x171a9" advised users "for assistance with troubleshooting potential issues with the falcon_lsm_serviceable kernel module provided from the CrowdStrike Falcon Sensor/Agent security software suite." Red Hat also advised that "disabling the CrowdStrike Falcon Sensor/Agent software suite … will mitigate the crashes and provide temporary stability to the system in question while the issue is investigated." The issue was "Observed but not limited to release 6 and 7."
Linux Kernel panics and Windows Blue Screens of Death are broadly comparable. The occurrence of kernel panics mere weeks before CrowdStrike broke many Windows implementations therefore hints at wider issues at the security vendor.
The Register has asked CrowdStrike to comment on the issues identified by Red Hat, and will update this story if we receive substantial information.
Rapid restore tool on the way
CrowdStrike on Sunday teased a rapid recovery tool for the mess it made.
"Together with customers, we tested a new technique to accelerate impacted system remediation," the security vendor stated on LinkedIn, adding "We're in the process of operationalizing an opt-in to this technique. We're making progress by the minute."
That progress will likely be of great interest, as Microsoft veep for enterprise and OS security David Weston on Saturday estimated that 8.5 million Windows machines had been laid low by the problem.
Microsoft also created a repair tool that runs from a bootable USB storage device and can be found here, along with instructions for use. Those instructions were modified on Sunday to require a full wipe of the USB device "so it doesn't error out when used in the recovery process."
CrowdStrike published technical details of the incident. It has also offered guidance on how to recover Windows machines encrypted with BitLocker.
Up in the air
The extent of disruption caused by CrowdStrike remains uncertain, but we've read accounts of over 6,800 flights cancelled last Friday alone, and of some airlines only restoring systems on Sunday evening.
The British Medical Association has warned that "normal service cannot be resumed immediately" due to the backlog caused by the outage.
Australia's home affairs minister Claire O'Neill has warned that remediation could take weeks.
This remains a developing story: The Register will update this item, or write others, as further info emerges. ®
Recommended Comments
There are no comments to display.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.