Microsoft's new Copilot can watch what you do on the web, and chat with you in real time while you do it

Copilot Vision will look at the web alongside you to help and assist, if you let it.

What you need to know

• Microsoft is launching an experimental new Copilot feature called Copilot Vision that can browse the web in Edge alongside you.
• With permission, Copilot Vision will be able to see the same webpages you do, and offer advice or suggestions based on real-time conversations.
• You can ask it for help with shopping or vent to it over a headline or story you've been reading.

Microsoft has unveiled a wild new feature coming to the new Copilot called Copilot Vision that enables it to browse the web alongside you to help you make decisions or to be there as a friend simply to chat with as you read the news or shop for items. The feature is exclusive to Copilot Pro subscribers via Microsoft's new experimental "Copilot Labs" preview program, and it might be the craziest example of today's newly announced Copilot capabilities. 

With the new Copilot, Microsoft wants you to treat it more like a friend who's available to help you out and less like a tool. Copilot Vision is a clear example of this, where Copilot can sit in the Edge browser and view the same web pages you do. This is an optional feature that you must first activate, but when you do, Copilot will be able to offer advice, opinions, and help based on whatever you ask of it while looking at the web.

Here's how Microsoft describes Copilot Vision:

"Copilot sits within Microsoft Edge. If you want it to, it can understand the page you’re viewing and answer questions about its content. It can suggest next steps, answer questions, help navigate whatever it is you want to do and assist with tasks. All the while you simply speak it to in natural language."

A good example of this is looking for shoes to buy. You can enable Copilot Vision and tell it that you're looking to buy new red shoes. Copilot can offer advice on where to look for the best deals and even tell you if it thinks the shoes you're tempted to choose actually look good. You can also use it as something to talk to as you read the news or social media.

Microsoft says Copilot Vision cannot directly interact with the web, but it can guide you on where to click if you're looking for something on a website that you've struggled to find. It also won't work on paywalled sites, and in this initial preview, it only works on a curated list of websites. The company intends to roll this out more widely in the coming months.

It's important to stress that Copilot Vision is an optional feature that isn't enabled by default and is only activated if you select it. When enabled, all the data it processes is handled on-device and promptly deleted after you are done. It doesn't get sent to Microsoft or third parties for training.