Microsoft's new Windows 11 Recall is a privacy nightmare

Microsoft's announcement of the new AI-powered Windows 11 Recall feature has sparked a lot of concern, with many thinking that it has created massive privacy risks and a new attack vector that threat actors can exploit to steal data.

Revealed during a Monday AI event, the feature is designed to help "recall" information you have looked at in the past, making it easily accessible via a simple search.

While it's currently only available on Copilot+ PCs running Snapdragon X ARM processors, Microsoft says they are working with Intel and AMD to create compatible CPUs.

Recall works by taking a screenshot of your active window every few seconds, recording everything you do in Windows for up to three months by default.

These snapshots will be analyzed by the on-device Neural Processing Unit (NPU) and an AI model to extract data from the screenshot. The data will be saved in a semantic index, allowing Windows users to browse through the snapshot history or search using human language queries.

Microsoft says that all of this data is encrypted using BitLocker tied to the user's Windows account and is not shared with other users on the same device.

While this sounds fun and interesting, it immediately raised concerns about obvious privacy risks and whether Microsoft plans on gobbling up all of this data.

However, Microsoft says Recall has been designed so that all of the data is saved directly on the user's device in an encrypted format, providing users with complete control over the feature, including if it's enabled and what apps it can take screenshots of.

Microsoft also says it will not create screenshots of Microsoft Edge's InPrivate windows (and other Chromium-based browsers) or content protected by DRM. However, they have not confirmed whether other browser's private modes, like Firefox, will be supported.

In a Monday press event, Yusuf Mehdi, Corporate Vice President & Consumer Chief Marketing Officer, assured journalists that Microsoft is taking a very conservative approach with Recall.

"We're going to keep your Recall index private and local and secure on just the device," said Mehdi.

"We won't use any of that information to train any AI model, and we put you completely in control with the ability to edit and delete anything that is captured."

Furthermore, Microsoft also reiterated to BleepingComputer that data for Recall will only be available locally and not be stored in the cloud, with the company once again restating that "data is not accessed by Microsoft."

Microsoft has also started to share more technical details, such as group policies that can be used to disable Recall company-wide and how end users can disable the feature.

Cybersecurity experts and regular users still concerned

Microsoft's promises have not done much to reassure the cybersecurity community or its customers, with our tweet regarding this new feature receiving over 90 comments, all negative.

So, why are most cybersecurity experts, researchers, and analysts so worried about this feature?

First and foremost, large companies have a history of exploiting users' data for their own profit, making it hard for users to trust Microsoft when they say they won't access the Recall data.

Users are not alone, as the United Kingdom's data protection agency, the Information Commissioner's Office (ICO), is also contacting Microsoft to ensure that users' data will be properly safeguarded and not used by the company.

"We expect organisations to be transparent with users about how their data is being used and only process personal data to the extent that it is necessary to achieve a specific purpose. Industry must consider data protection from the outset and rigorously assess and mitigate risks to peoples' rights and freedoms before bringing products to market," reads a press statement from the ICO.

"We are making enquiries with Microsoft to understand the safeguards in place to protect user privacy."

Even if we accept that Microsoft will not access Recall data, there are still massive security and privacy implications with this product.

Microsoft admits that the feature performs no content moderation, meaning it will gobble up anything it sees, including passwords in a password manager or your account numbers on your banking website.

Or if you are in Word, writing a confidential agreement, a screenshot of that content will be created, too. If you have a single PC and share it with others, then you may want to be careful about what pictures or videos you look at, as, guess what, those will be recorded as well.

Yes, you can block apps from being screenshotted by this feature, but most people will just let it run without mucking around with the feature's settings.

All of this information is now stored in Windows 11's semantic index and easily searchable by anyone with access to your PC, whether authorized or not.

That's just the tip of the iceberg, though.

If a threat actor or malware compromised your device, all of this data will already be decrypted by Bitlocker, making it accessible to the hacker. 

For example, a threat actor or malware could simply steal a Recall database and upload it to their own servers for analysis. This information could then be used to extort users or potentially breach user's accounts if credentials were exposed.

Cybersecurity expert Kevin Beaumont, known to be an outspoken critic of Microsoft at times, also expressed concern about how this feature creates a massive attack surface, likening it to a keylogger "baked into Windows."

"If you look at what has happened historically with infostealer malware — malicious software snuck onto PCs — it has pivoted to automatically steal browser passwords stored locally," Beaumont explained in a new blog post.

"In other words, if a malicious threat actor gains access to a system, they already steal important databases stored locally. They can just extend this to steal information recorded by Copilot's Recall feature."

And it's not only information-stealing malware, as enterprise-targeting malware like TrickBot had previously included modules that would steal a domain's Active Directory database for offline cracking of credentials. There is nothing to stop malware from taking a similar approach and stealing the Recall databases as well.

Microsoft has always taken the stance with vulnerabilities and attacks that once a device is compromised, all bets are off, and security boundaries are thrown out the window.

Basically, you got infected or fell for a social engineering attack, so it's your fault all these bad things will happen to you.

However, as Microsoft is one of, if not the, largest caretakers of consumer data and computing security, it seems irresponsible to introduce additional risk into an already risky environment.

While we can go on and on expressing how this feature is a massive privacy risk, I will instead leave you with this quote from Microsoft's recent pledge to prioritize security above all else.

"If you're faced with the tradeoff between security and another priority, your answer is clear: Do security. In some cases, this will mean prioritizing security above other things we do, such as releasing new features or providing ongoing support for legacy systems," Microsoft's CEO Satya Nadella said in an email to Microsoft employees.

"This is key to advancing both our platform quality and capability such that we can protect the digital estates of our customers and build a safer world for all."

Update 5/22/24: This article previously said Microsoft is working with Intel and AMD to make all Windows 11 devices compatible, when they are instead working with them to make compatible CPUs.

Source