A federal Cyber Safety Review Board has issued its report on what led to last summer's capture of hundreds of thousands of emails by Chinese hackers from cloud customers, including federal agencies. It cites "a cascade of security failures at Microsoft" and finds that "Microsoft's security culture was inadequate" and needs to adjust to a "new normal" of cloud provider targeting.

 

The report, mandated by President Biden in the wake of the far-reaching intrusion, details the steps that Microsoft took before, during, and after the breach and in each case finds critical failure. The breach was "preventable," even though it cites Microsoft as not knowing precisely how Storm-0558, a "hacking group assessed to be affiliated with the People's Republic of China," got in.

 

"Throughout this review, the board identified a series of Microsoft operational and strategic decisions that collectively points to a corporate culture that deprioritized both enterprise security investments and rigorous risk management," the report reads.

 

The report notes that Microsoft "fully cooperated with the Board's review." A Microsoft spokesperson issued a statement regarding the report. "We appreciate the work of the CSRB to investigate the impact of well-resourced nation state threat actors who operate continuously and without meaningful deterrence," the statement reads. "As we announced in our Secure Future Initiative, recent events have demonstrated a need to adopt a new culture of engineering security in our own networks." Along with hardening its systems and implementing more sensors and logs to "detect and repel the cyber-armies of our adversaries," Microsoft said it would "review the final report for additional recommendations."

“Inaccurate public statements” and unsolved mysteries

The Cyber Safety Review Board (CSRB), formed two years ago, is composed of government and industry officials, from entities including the Departments of Homeland Security, Justice, and Defense, the NSA, FBI, and others. Microsoft provides cloud-based services, including Exchange and Azure, to numerous government agencies, including consulates.

 

Microsoft has previously offered a version of the intrusion story, one that notably avoids the words "vulnerability," "exploit," or "zero-day." A Microsoft post in July 2023 cited an inactive signing key acquired by Storm-0558, which was then used to forge tokens for the Azure AD cloud service that stores keys for logins. This was "made possible by a validation error in Microsoft code," Microsoft wrote.

 

Congress and government agencies called on Microsoft to offer far more disclosure, and others, including Tenable's CEO, offered even harsher assessments. In September, the company met them partway. It was an engineer's account that was hacked, Microsoft claimed, giving attackers access to a supposedly locked-down workstation, the consumer signing key, and, crucially, access to crash dumps moved into a debugging environment. A "race condition" prevented a mechanism that strips out signing keys and other sensitive data from crash dumps from functioning. Furthermore, "human errors" allowed for an expired signing key to be used in forging tokens for modern enterprise offerings.

 

Those kinds of unrevealing, withholding public statements were cited by the CSRB in its finding of Microsoft's failures. The report cites "Microsoft’s decision not to correct, in a timely manner, its inaccurate public statements about this incident, including a corporate statement that Microsoft believed it had determined the likely root cause of the intrusion when in fact, it still has not." It also notes that Microsoft did not update its September 2023 blog post about the invasion cause until March 2024, "as the Board was concluding its review and only after the Board’s repeated questioning about Microsoft’s plans to issue a correction." (The updated blog post notes that Microsoft has "not found a crash dump containing the impacted key material.")

 

Audit logging and “pay-to-play” security

The CSRB report also raises the issue of what some have called Microsoft's "pay-to-play" security model. The report notes that State Department officials detected the Chinese breach in June and notified Microsoft. That only happened because the department paid for a "G5" tier of Microsoft's cloud services that provided "Microsoft Purview Audit (Premium)." With that, the State Department had an alert set up for notable mail item access, which was triggered by the invaders' download of roughly 60,000 emails off-site. The report calls for cloud service providers to "adopt a minimum standard for default audit logging in cloud services" for better intrusion detection.

 

Once Microsoft realized that the intruders had used a theoretically expired 2016 consumer signing key to forge tokens for an enterprise customer, it launched an "all-hands-on-deck" investigation that went through the night, June 26–27. The company arrived at 46 hypotheses for the intrusion, including "a theoretical quantum computing capability to break public-key cryptography." Despite assigning teams to investigate every hypothesis, "nine months after the discovery of the intrusion, Microsoft says that its investigation into these hypotheses remains ongoing."

 

While Microsoft was ultimately able to remove attackers' access to 22 enterprise organizations and 503 individual accounts, by the end of the board's review, the company could not "demonstrate to the Board that it knew how Storm-0558 had obtained the 2016 MSA key."

 

Throughout 24 pages (plus appendices), the CSRB report also cites numerous flaws in the way Microsoft handled the messaging of a breach involving some of its most high-profile customers. It cites Rep. Don Bacon (R-Nebraska), who serves on the House Armed Services Committee and is a member of the House Taiwan Caucus.

 

Microsoft sent Bacon and other compromised users an "Unusual Sign-In Activity" email, with a password change prompt. That prompt, which claimed that "this notification does not mean that Microsoft's own systems have in any way been compromised," looked to Bacon and some other victims like "possible spam," so they disregarded it. Bacon changed his password directly rather than click on the email's reset link. He later learned from the FBI that his email had been compromised by that point. He and others later received more detailed notice from Microsoft about the intrusion later.

“Cascade of Microsoft’s avoidable errors”

The CSRB's conclusion is that Microsoft's security culture is "inadequate" and that a "cascade of Microsoft's avoidable errors allowed this intrusion to succeed." It cites in particular:

 

• Lacking security practices of other cloud providers
• Failure to detect a compromise on a laptop from an employee at an acquired company before connecting it to its network
• Letting inaccurate public statements stand for months
• A "separate incident" from January 2024 that, while not in the CSRB's purview, allowed another nation-state actor access to emails, code, and internal systems
• A need to "demonstrate the highest standards of security, accountability, and transparency."

 

"Unfortunately, throughout this review, the Board identified a series of operational and strategic decisions that collectively point to a corporate culture in Microsoft that deprioritized both enterprise security investments and rigorous risk management," the report states. "These decisions resulted in significant costs and harm for Microsoft customers around the world. The Board is convinced that Microsoft should address its security culture." The report cites a 2002 email from then-CEO Bill Gates, noting that "when we face a choice between adding features and resolving security issues, we need to choose security."

The full report includes far more detail, including a timeline of the Storm-0558 attack from the 2016 key to Microsoft's last 2023 blog post regarding the breach.

Storm-0558 and a very rich target

While it receives far less attention in the report, the entity involved in the other side of the summer 2023 incident, Storm-0558, is among the most successful, skilled, and relentless nation-state threat actors working today. Microsoft and other global-scale providers are frequent targets of such groups. The report notes Storm-0558's history of compromising clouds and, in particular, stealing authentication keys. It notes "industry links" of Storm-0558 to Operation Aurora, which led Google to largely leave China in 2010, along with the 2011 RSA SecurID failure.

 

"Indeed, security researchers have tracked Storm-0558’s activities for over 20 years," the report notes.

 

Listing image by Getty Images

 

Google offers a VPN via its "Google One" monthly subscription plan, and while it debuted on phones, a desktop app has been available for Windows and Mac OS for over a year now. Since a lot of people pay for Google One for the cloud storage increase for their Google accounts, you might be tempted to try the VPN on a desktop, but Windows users testing out the app haven't seemed too happy lately. An open bug report on Google's GitHub for the project says the Windows app "breaks" the Windows DNS, and this has been ongoing since at least November.

 

A VPN would naturally route all your traffic through a secure tunnel, but you've still got to do DNS lookups somewhere. A lot of VPN services also come with a DNS service, and Google is no different. The problem is that Google's VPN app changes the Windows DNS settings of all network adapters to always use Google's DNS, whether the VPN is on or off. Even if you change them, Google's program will change them back.

 

Most VPN apps don't work this way, and even Google's Mac VPN program doesn't work this way. The users in the thread (and the ones emailing us) expect the app, at minimum, to use the original Windows settings when the VPN is off. Since running a VPN is often about privacy and security, users want to be able to change the DNS away from Google even when the VPN is running.

 

Changing the DNS can result in several problems for certain setups. As users in the thread point out, some people, especially those using a VPN, want an encrypted DNS setup, and Google's VPN program will just turn this off. It can break custom filtering setups and will prevent users from accessing local network IPs, like a router configuration page or corporate intranet pages. It will also make it impossible to log in to a captive portal, which you often see on public Wi-Fi at a hotel, airport, or coffee shop.

 

Besides that behavior, the thread is full of all sorts of reports of Google's VPN program getting screwy with the Windows DNS settings. Several users say Google's VPN app frequently resets the DNS settings of all network adapters, even if they change them after the initial install sets them to 8.8.8.8. For instance, one reply from ryanzimbauser says: "This program has absolutely no business changing all present NICs to a separate DNS on the startup of my computer while the program is not set to 'Launch app after computer starts.' This recent change interfered with my computer's ability to access a network implementing a private DNS filter. This has broken my trust and I will not be reinstalling this program until this is remedied."

 

Several user reports say that even after uninstalling the Google VPN, the DNS settings don't revert to what they used to be. Maybe this is more of a Windows problem than a Google problem, but a lot of users have trouble changing the settings away from 8.8.8.8 through the control panel after uninstalling. They are resorting to registry changes, PowerShell scripts, or the "reset network settings" button.

 

Google employee Ryan Lothian responded to the thread, saying:

 

Hey folks, thank you for reporting this behaviour.

 

To protect users privacy, the Google One VPN deliberately sets DNS to use Google's DNS servers. This prevents a nefarious DNS server (that might be set by DHCP) compromising your privacy. Visit https://developers.google.com/speed/public-dns/privacy to learn about the limited logging performed by Google DNS.

 

We think this is a good default for most users. However, we do recognize that some users might want to have their own DNS, or have the DNS revert when VPN disconnects. We'll consider adding this to a future release of the app.

It's pretty rare for Google, the web and Android company, to make a Windows program. There's Chrome, the Drive syncing app, Google Earth Pro, this VPN app, and not too much else. You can find it by going to the Google One website, clicking "Benefits" in the sidebar, and then "View Details" under the VPN box, where you'll find an exceedingly rare Google Windows executable.

 

If you want a VPN and care about privacy, there are probably better places to go than Google. The company can still see all the websites you're visiting via its DNS servers, and while the VPN data might be private, Google's DNS holds onto your web history for up to 48 hours and is subject to subpoenas. There are several accusations in the thread of Google changing DNS for data harvesting purposes, but if you're concerned about that, maybe don't do business with one of the world's biggest user-tracking companies.

 

Discord had long been strongly opposed to ads, but starting this week, it's giving video game makers the ability to advertise to its users. The introduction of so-called Sponsored Quests marks a notable change from the startup's previous business model, but, at least for now, it seems much less intrusive than the ads shoved into other social media platforms, especially since Discord users can choose not to engage with them.

 

Discord first announced Sponsored Quests on March 7, with Peter Sellis, Discord's SVP of product, writing in a blog post that users would start seeing them in the "coming weeks." Sponsored Quests offer PC gamers in-game rewards for getting friends to watch a stream of them playing through Discord. Discord senior product communications manager Swaleha Carlson confirmed to Ars Technica that Sponsored Quests launch this week.

 

 

The goal is for video games to get exposure to more gamers, serving as a form of marketing. On Saturday, The Wall Street Journal (WSJ) reported that it viewed a slide from a slideshow Discord shows to game developers regarding the ads that reads: "We’ll get you in front of players. And those players will get you into their friend groups."

 

Sellis told WSJ that Discord will target ads depending on users' age, geographic location data, and gameplay. The ads will live on the bottom-left of the screen, but users can opt out of personalized promotions for Quests that are based on activity or data shared with Discord, Swaleha Carlson, senior product communications manager at Discord, told Ars Technica.

 

"Users may still see Quests, however, if they navigate to their Gift Inventory and/or through contextual entry points like a user’s friends’ activity. They’ll also have the option to hide an in-app promotion for a specific Quest or game they're not interested in," she said.

 

"Users may still see Quests, however, if they navigate to their Gift Inventory and/or through contextual entry points like a user’s friends’ activity. They’ll also have the option to hide an in-app promotion for a specific Quest or game they're not interested in. "

 

Discord already tested the ads in May with Lucasfilm Games and Epic Games. Discord users were able to receive Star Wars-themed gear in Fortnite for getting a friend to watch them play Fortnite on PC for at least 15 minutes.

 

Jason Citron, Discord co-founder and CEO, told Bloomberg in March that the company hopes that one day "every game will offer Quests on Discord."

Discord used to be anti-ads

It may be a nuisance for users to have to disable personalized promotion for Sponsored Quests when they never asked for them, but it should bring long-term users at least some comfort that their data purportedly doesn't have to contribute to the marketing. However, it's unclear if Discord may one day change this. The fact that the platform is implementing ads at all is somewhat surprising. Discord named its avoidance of advertising as one of its key differentiators from traditional social media platforms as recently as late January.

 

In March 2021, Citron told WSJ that Discord had eschewed ads until that point because ads would be intrusive, considering Discord's purpose of instant back-and-forth communication and people's general distaste for viewing ads and having their data shared with other companies.

 

“We really believe we can build products that make Discord more fun and that people will pay for them. It keeps our incentives aligned,” Citron told WSJ at the time.

 

That same year, Citron, in response to a question about why being ad-free is important to Discord, told NPR: "We believe that people's data is their data and that people should feel comfortable and safe to have conversations and that their data is not going to be used against them in any way that is improper."

 

Sponsored Quests differs from other types of ads that would more obviously disrupt Discord users' experiences, such as pop-up ads or ads viewed alongside chat windows.

A tight-rope to walk

Beyond Sponsored Quests, Discord, which launched in 2015, previously announced that it would start selling sponsored profile effects and avatar decorations in the Discord Shop. In March, Discord's Sellis said this would arrive in the "coming weeks." Discord is also trying to hire more than 12 people to work in ad sales, WSJ said Saturday, citing anonymous "people familiar with [Discord's] plans."

 

Discord's Carlson declined to comment to Ars on whether or not Discord plans to incorporate other types of ads into Discord. She noted that Sponsored Quests "are currently in the pilot phase" and that the company will "continue to iterate based on what we learn."

 

In 2021, Discord enjoyed a nearly three-times revenue boost that it attributed to subscription sales for Nitro, which adds features like HD video streaming and up to 500MB uploads. In March, Citron told Bloomberg that Discord has more than 200 million monthly active users and that the company will "probably" go public eventually.

 

The publication, citing unnamed "people with knowledge of the matter," also reported that Discord makes over $600 million in annualized revenue. The startup has raised over $1 billion in funding and is reported to have over $700 million in cash. However, the company reportedly isn't profitable. It also laid off 17 percent of staffers, or 170 workers, in January.

 

Meanwhile, ads are the top revenue generator for many other social media platforms, such as Reddit, which recently went public.

 

While Discord's first real ads endeavor seems like it will have minimal impact on users who aren't interested in them, it brings the company down a tricky road that it hasn't previously navigated. A key priority should be ensuring that any form of ads doesn't disrupt the primary reasons people like using Discord. As it stands, Sponsored Quests might already put off some users.

 

On Friday, researchers revealed the discovery of a backdoor that was intentionally planted in xz Utils, an open-source data compression utility available on almost all installations of Linux and other Unix-like operating systems. The person or people behind this project likely spent years on it. They were likely very close to seeing the backdoor update merged into Debian and Red Hat, the two biggest distributions of Linux when an eagle-eyed software developer spotted something fishy.

 

"This might be the best executed supply chain attack we've seen described in the open, and it's a nightmare scenario: malicious, competent, authorized upstream in a widely used library," software and cryptography engineer Filippo Valsorda said of the effort, which came frightfully close to succeeding.

 

Researchers have spent the weekend gathering clues. Here's what we know so far.

 

What is xz Utils?

 

xz Utils is nearly ubiquitous in Linux. It provides lossless data compression on virtually all Unix-like operating systems, including Linux. xz Utils provides critical functions for compressing and decompressing data during all kinds of all kinds of operations. xz Utils also supports the legacy .lzma format, making this component even more crucial.

 

What happened?

 

Andres Freund, a developer and engineer working on Microsoft’s PostgreSQL offerings, was recently troubleshooting performance problems a Debian system was experiencing with SSH, the most widely used protocol for remotely logging into devices over the Internet. Specifically, SSH logins were consuming too many CPU cycles and were generating errors with valgrind, a utility for monitoring computer memory.

 

Through a combination of sheer luck and Freund’s careful eye, he eventually discovered the problems were the result of updates that had been made to xz Utils. On Friday, Freund took to the Open Source Security List to disclose the updates were the result of someone intentionally planting a backdoor in the compression software.

 

What does the backdoor do?

 

Malicious code added to xz Utils versions 5.6.0 and 5.6.1 modified the way the software functions when performing operations related to lzma compression or decompression. When these functions involved SSH, they allowed for malicious code to be executed with root privileges. This code allowed someone in possession of a predetermined encryption key to log into the backdoored system over SSH. From then on, that person would have the same level of control as any authorized administrator.

 

How did this backdoor come to be?

 

It would appear that this backdoor was years in the making. In 2021, someone with the username JiaT575 made their first known commit to an open-source project. In retrospect, the change to the libarchive project is suspicious, because it replaced the safe_fprint funcion with a variant that’s long been recognized as less secure. No one noticed at the time.

 

The following year, JiaT575 submited a patch over the xz Utils mailing list, and almost immediately, a never-before-seen participant named Jigar Kumar joined the discussion and argued that Lasse Collin, the longtime maintainer of xz Utils, hadn’t been updating the software often or fast enough. Kumar, with the support of Dennis Ens and several other people who had never had a presence on the list, pressured Collin to bring on an additional developer to maintain the project.

 

In January 2023, JiaT75,made their first commit to xz Utils. In the months following, JiaT75, who used the name Jia Tan, became increasingly involved in xz Utils affairs. For instance, Tan replaced Collins's contact information with their own on Microsoft's oss-fuzz, a project that scans open-source software for signs of maliciousness. Tan also requested that oss-fuzz disable the ifunc function during testing, a change that prevented it from detecting the malicious changes Tan would soon make to xz Utils.

 

In February of this year, Tan issued commits for versions 5.6.0 and 5.6.1 of xz Utils. The updates implemented the backdoor. In the following weeks, Tan or others appeal to developers of Ubuntu, Red Hat, and Debian to merge the updates into their OSes. Eventually, one of the two updates made its way into the following releases, according to security firm Tenable:

 

 

Can you say more about what this backdoor does?

 

In a nutshell, it allows someone with the right private key to hijack sshd, the executable file responsible for making SSH connections, and from there to execute malicious commands. The backdoor is implemented through a five-stage loader that uses a series of simple but clever techniques to hide itself. It also provides the means for new payloads to be delivered without major changes being required.

 

Multiple people who have reverse engineered the updates have much more to say about the backdoor.

 

Developer Sam James provided this overview:

 

This backdoor has several components. At a high level:

 

• The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf. The version of build-to-host.m4 in the release tarballs differs wildly from the upstream on GitHub.
• There are crafted test files in the tests/ folder within the git repository too. These files are in the following commits:
  • tests/files/bad-3-corrupt_lzma2.xz (cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0, 74b138d2a6529f2c07729d7c77b1725a8e8b16f1)
  • tests/files/good-large_compressed.lzma (cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0, 74b138d2a6529f2c07729d7c77b1725a8e8b16f1)
• A script called by build-to-host.m4 that unpacks this malicious test data and uses it to modify the build process.
• IFUNC, a mechanism in glibc that allows for indirect function calls, is used to perform runtime hooking/redirection of OpenSSH's authentication routines. IFUNC is a tool that is normally used for legitimate things, but in this case it is exploited for this attack path.

 

Normally upstream publishes release tarballs that are different than the automatically generated ones in GitHub. In these modified tarballs, a malicious version of build-to-host.m4 is included to execute a script during the build process.

 

This script (at least in versions 5.6.0 and 5.6.1) checks for various conditions like the architecture of the machine. Here is a snippet of the malicious script that gets unpacked by build-to-host.m4 and an explanation of what it does:

if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo "$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1);then

• If amd64/x86_64 is the target of the build
• And if the target uses the name linux-gnu (mostly checks for the use of glibc)

 

It also checks for the toolchain being used:

if test "x$GCC" != 'xyes' > /dev/null 2>&1;then
exit 0
fi
if test "x$CC" != 'xgcc' > /dev/null 2>&1;then
exit 0
fi
LDv=$LD" -v"
if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then
exit 0

And if you are trying to build a Debian or Red Hat package:

if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then

This attack thusly seems to be targeted at amd64 systems running glibc using either Debian or Red Hat derived distributions. Other systems may be vulnerable at this time, but we don't know.

In an online interview, developer and reverse engineer HD Moore confirmed the Sam James suspicion that the backdoor targeted either Debian or Red Hat distributions.

 

“The attack was sneaky in that it only did the final steps of the backdoor if you were building the library on amd64 (intel x86 64-bit) and were building a Debian or a RPM package (instead of using it for a local installation),” he wrote.

 

Paraphrasing observations from researchers who collectively spent the weekend analyzing the malicius updates, he continued:

 

When verifying an SSH public key, if the public key matches a certain fingerprint function, the key contents are decrypted using a pre-shared key before the public key is actually verified. The decrypted contents are then passed directly to system.

 

If the fingerprint doesn't match or the decrypted contents don't match a certain format, it falls back to regular key verification and no-one's the wiser.

 

The backdoor is super sneaky. It uses a little-known feature of the glibc to hook a function. It only triggers when the backdoored xz library gets loaded by a /usr/bin/sshd process on one of the affected distributions. There may be many other backdoors, but the one everyone is talking about uses the function indirection stuff to add the hook. The payload was encoded into fake xz test files and runs as a shellcode effectively, changing the SSH RSA key verification code so that a magic public key (sent during normal authentication) let the attacker gain access

 

Their grand scheme was:

 

1) sneakily backdoor the release tarballs, but not the source code

 

2) use sockpuppet accounts to convince the various Linux distributions to pull the latest version and package it

 

3) once those distributions shipped it, they could take over any downstream user/company system/etc

Additional technical analysis is available from the above Bluesky thread from Valsorda, researcher Kevin Beaumont and Freund’s Friday disclosure.

 

What more do we know about Jia Tan?

 

At the moment, extremely little, especially for someone entrusted to steward a piece of software as ubiquitous and as sensitive as xz Utils. This developer persona has touched dozens of other pieces of open-source software in the past few years. At the moment, it’s unknown if there was ever a real-world person behind this username or if Jia Tan is a completely fabricated individual.

Researchers have found a malicious backdoor in a compression tool that made its way into widely used Linux distributions, including those from Red Hat and Debian.

 

The compression utility, known as xz Utils, introduced the malicious code in versions 5.6.0 and 5.6.1, according to Andres Freund, the developer who discovered it. There are no known reports of those versions being incorporated into any production releases for major Linux distributions, but both Red Hat and Debian reported that recently published beta releases used at least one of the backdoored versions—specifically, in Fedora Rawhide and Debian testing, unstable and experimental distributions. A stable release of Arch Linux is also affected. That distribution, however, isn't used in production systems.

 

Because the backdoor was discovered before the malicious versions of xz Utils were added to production versions of Linux, “it's not really affecting anyone in the real world,” Will Dormann, a senior vulnerability analyst at security firm Analygence, said in an online interview. “BUT that's only because it was discovered early due to bad actor sloppiness. Had it not been discovered, it would have been catastrophic to the world.”

 

Several people, including two Ars readers, reported that the multiple apps included in the HomeBrew package manager for macOS rely on the backdoored 5.6.1 version of xz Utils. HomeBrew has now rolled back the utility to version 5.4.6. Maintainers have more details available here.

Breaking SSH authentication

The first signs of the backdoor were introduced in a February 23 update that added obfuscated code, officials from Red Hat said in an email. An update the following day included a malicious install script that injected itself into functions used by sshd, the binary file that makes SSH work. The malicious code has resided only in the archived releases—known as tarballs—which are released upstream. So-called GIT code available in repositories aren’t affected, although they do contain second-stage artifacts allowing the injection during the build time. In the event the obfuscated code introduced on February 23 is present, the artifacts in the GIT version allow the backdoor to operate.

 

The malicious changes were submitted by JiaT75, one of the two main xz Utils developers with years of contributions to the project.

 

“Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system,” Freund wrote. “Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the ‘fixes’” provided in recent updates. Those updates and fixes can be found here, here, here, and here.

 

On Thursday, someone using the developer's name took to a developer site for Ubuntu to ask that the backdoored version 5.6.1 be incorporated into production versions because it fixed bugs that caused a tool known as Valgrind to malfunction.

 

“This could break build scripts and test pipelines that expect specific output from Valgrind in order to pass,” the person warned, from an account that was created the same day.

 

One of maintainers for Fedora said Friday that the same developer approached them in recent weeks to ask that Fedora 40, a beta release, incorporate one of the backdoored utility versions.

 

“We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added),” the Ubuntu maintainer said. "He has been part of the xz project for two years, adding all sorts of binary test files, and with this level of sophistication, we would be suspicious of even older versions of xz until proven otherwise."

 

Maintainers for xz Utils didn’t immediately respond to emails asking questions.

 

The malicious versions, researchers said, intentionally interfere with authentication performed by SSH, a commonly used protocol for connecting remotely to systems. SSH provides robust encryption to ensure that only authorized parties connect to a remote system. The backdoor is designed to allow a malicious actor to break the authentication and, from there, gain unauthorized access to the entire system. The backdoor works by injecting code during a key phase of the login process.

 

“I have not yet analyzed precisely what is being checked for in the injected code, to allow unauthorized access,” Freund wrote. “Since this is running in a pre-authentication context, it seems likely to allow some form of access or other form of remote code execution.”

 

In some cases, the backdoor has been unable to work as intended. The build environment on Fedora 40, for example, contains incompatibilities that prevent the injection from correctly occurring. Fedora 40 has now reverted to the 5.4.x versions of xz Utils.

 

Xz Utils is available for most if not all Linux distributions, but not all of them include it by default. Anyone using Linux should check with their distributor immediately to determine if their system is affected. Freund provided a script for detecting if an SSH system is vulnerable.

A newly discovered vulnerability baked into Apple’s M-series of chips allows attackers to extract secret keys from Macs when they perform widely used cryptographic operations, academic researchers have revealed in a paper published Thursday.

 

The flaw—a side channel allowing end-to-end key extractions when Apple chips run implementations of widely used cryptographic protocols—can’t be patched directly because it stems from the microarchitectural design of the silicon itself. Instead, it can only be mitigated by building defenses into third-party cryptographic software that could drastically degrade M-series performance when executing cryptographic operations, particularly on the earlier M1 and M2 generations. The vulnerability can be exploited when the targeted cryptographic operation and the malicious application with normal user system privileges run on the same CPU cluster.

Beware of hardware optimizations

The threat resides in the chips’ data memory-dependent prefetcher, a hardware optimization that predicts the memory addresses of data that running code is likely to access in the near future. By loading the contents into the CPU cache before it’s actually needed, the DMP, as the feature is abbreviated, reduces latency between the main memory and the CPU, a common bottleneck in modern computing. DMPs are a relatively new phenomenon found only in M-series chips and Intel's 13th-generation Raptor Lake microarchitecture, although older forms of prefetchers have been common for years.

 

Security experts have long known that classical prefetchers open a side channel that malicious processes can probe to obtain secret key material from cryptographic operations. This vulnerability is the result of the prefetchers making predictions based on previous access patterns, which can create changes in state that attackers can exploit to leak information. In response, cryptographic engineers have devised constant-time programming, an approach that ensures that all operations take the same amount of time to complete, regardless of their operands. It does this by keeping code free of secret-dependent memory accesses or structures.

 

The breakthrough of the new research is that it exposes a previously overlooked behavior of DMPs in Apple silicon: Sometimes they confuse memory content, such as key material, with the pointer value that is used to load other data. As a result, the DMP often reads the data and attempts to treat it as an address to perform memory access. This “dereferencing” of “pointers”—meaning the reading of data and leaking it through a side channel—is a flagrant violation of the constant-time paradigm.

 

The team of researchers consists of:

 

• Boru Chen, University of Illinois Urbana-Champaign
• Yingchen Wang, University of Texas at Austin
• Pradyumna Shome, Georgia Institute of Technology
• Christopher W. Fletcher, University of California, Berkeley
• David Kohlbrenner, University of Washington
• Riccardo Paccagnella, Carnegie Mellon University
• Daniel Genkin, Georgia Institute of Technology

 

In an email, they explained:

 

Prefetchers usually look at addresses of accessed data (ignoring values of accessed data) and try to guess future addresses that might be useful. The DMP is different in this sense as in addition to addresses it also uses the data values in order to make predictions (predict addresses to go to and prefetch). In particular, if a data value “looks like” a pointer, it will be treated as an “address” (where in fact it's actually not!) and the data from this “address” will be brought to the cache. The arrival of this address into the cache is visible, leaking over cache side channels.

 

Our attack exploits this fact. We cannot leak encryption keys directly, but what we can do is manipulate intermediate data inside the encryption algorithm to look like a pointer via a chosen input attack. The DMP then sees that the data value “looks like” an address, and brings the data from this “address” into the cache, which leaks the “address.” We don’t care about the data value being prefetched, but the fact that the intermediate data looked like an address is visible via a cache channel and is sufficient to reveal the secret key over time.

In Thursday’s paper, the team explained it slightly differently:

 

Our key insight is that while the DMP only dereferences pointers, an attacker can craft program inputs so that when those inputs mix with cryptographic secrets, the resulting intermediate state can be engineered to look like a pointer if and only if the secret satisfies an attacker-chosen predicate. For example, imagine that a program has secret s, takes x as input, and computes and then stores y = s ⊕ x to its program memory. The attacker can craft different x and infer partial (or even complete) information about s by observing whether the DMP is able to dereference y. We first use this observation to break the guarantees of a standard constant-time swap primitive recommended for use in cryptographic implementations. We then show how to break complete cryptographic implementations designed to be secure against chosen-input attacks.

Enter GoFetch

The attack, which the researchers have named GoFetch, uses an application that doesn’t require root access, only the same user privileges needed by most third-party applications installed on a macOS system. M-series chips are divided into what are known as clusters. The M1, for example, has two clusters: one containing four efficiency cores and the other four performance cores. As long as the GoFetch app and the targeted cryptography app are running on the same performance cluster—even when on separate cores within that cluster—GoFetch can mine enough secrets to leak a secret key.

 

The attack works against both classical encryption algorithms and a newer generation of encryption that has been hardened to withstand anticipated attacks from quantum computers. The GoFetch app requires less than an hour to extract a 2048-bit RSA key and a little over two hours to extract a 2048-bit Diffie-Hellman key. The attack takes 54 minutes to extract the material required to assemble a Kyber-512 key and about 10 hours for a Dilithium-2 key, not counting offline time needed to process the raw data.

 

The RSA and Diffie-Hellman keys were processed on implementations from Go and OpenSSL and the Kyber and Dilithium from CRYSTALS-Kyber and CRYSTALS-Dilithium. All four implementations employ constant-time programming, proving that the DMPs in Apple silicon defeat the widely deployed defense.

 

 

 

GoFetch isn’t the first time researchers have identified threats lurking in Apple DMPs. The optimization was first documented in 2022 research that discovered a previously unknown “pointer-chasing DMP” in both the M1 and Apple’s A14 Bionic chip for iPhones. The research, from a different assemblage of academics, gave rise to Augury, an attack that identified and exploited a memory side channel that leaked pointers. Ultimately, Augury was unable to mix data and addresses when constant-time practices were used, a shortcoming that may have given the impression the DMP didn’t pose much of a threat.

 

“GoFetch shows that the DMP is significantly more aggressive than previously thought and thus poses a much greater security risk,” the GoFetch authors wrote on their website. “Specifically, we find that any value loaded from memory is a candidate for being dereferenced (literally!). This allows us to sidestep many of Augury's limitations and demonstrate end-to-end attacks on real constant-time code.”

Penalizing performance

Like other microarchitectural CPU side channels, the one that makes GoFetch possible can’t be patched in the silicon. Instead, responsibility for mitigating the harmful effects of the vulnerability falls on the people developing code for Apple hardware. For developers of cryptographic software running on M1 and M2 processors, this means that in addition to constant-time programming, they will have to employ other defenses, almost all of which come with significant performance penalties.

 

One of the most effective mitigations, known as ciphertext blinding, is a good example. Blinding works by adding/removing masks to sensitive values before/after being stored to/loaded from memory. This effectively randomizes the internal state of the cryptographic algorithm, preventing the attacker from controlling it and thus neutralizing GoFetch attacks. Unfortunately, the researchers said, this defense is both algorithm-specific and often costly, potentially even doubling the computing resources needed in some cases, such as for Diffie-Hellman key exchanges.

 

One other defense is to run cryptographic processes on the previously mentioned efficiency cores, also known as Icestorm cores, which don't have DMP. One approach is to run all cryptographic code on these cores. This defense, too, is hardly ideal. Not only is it possible for unannounced changes to add DMP functionality to efficiency cores, running cryptographic processes here will also likely increase the time required to complete operations by a nontrivial margin. The researchers mention several ad-hoc defenses, but they are equally problematic.

 

The DMP on the M3, Apple’s latest chip, has a special bit that developers can invoke to disable the feature. The researchers don’t yet know what kind of penalty will occur when this performance optimization is turned off. (The researchers noted that the DMP found in Intel’s Raptor Lake processors doesn’t leak the same sorts of cryptographic secrets. What’s more, setting a special DOIT bit also effectively turns off the DMP.)

 

Readers should remember that whatever penalties result will only be felt when affected software is performing specific cryptographic operations. For browsers and many other types of apps, the performance cost may not be noticeable.

 

“Longer term, we view the right solution to be to broaden the hardware-software contract to account for the DMP,” the researchers wrote. “At a minimum, hardware should expose to software a way to selectively disable the DMP when running security-critical applications. This already has nascent industry precedent. For example, Intel’s DOIT extensions specifically mention disabling their DMP through an ISA extension. Longer term, one would ideally like finer-grain control, e.g., to constrain the DMP to only prefetch from specific buffers or designated non-sensitive memory regions.”

 

Apple representatives declined to comment on the record about the GoFetch research.

 

End users who are concerned should check for GoFetch mitigation updates that become available for macOS software that implements any of the four encryption protocols known to be vulnerable. Out of an abundance of caution, it’s probably also wise to assume, at least for now, that other cryptographic protocols are likely also susceptible.

 

“Unfortunately, to assess if an implementation is vulnerable, cryptanalysis and code inspection are required to understand when and how intermediate values can be made to look like pointers in a way that leaks secrets,” the researchers advised. “This process is manual and slow and does not rule out other attack approaches.”