Amazon made a bigger camera-spying store—so we tried to steal its fruit

[Karlston]

A fine mist —

Amazon made a bigger camera-spying store—so we tried to steal its fruit

It's like other Amazon Go stores, only bigger. But it's a meaningful difference.

Enlarge / Amazon Go Grocery's first location in the Seattle neighborhood of Capitol Hill.

Sam Machkovech

SEATTLE—For how far and wide Amazon's digital footprint reaches, the company clearly wants to advance into real-world space as much as possible. And to that end, Amazon runs some of its most ambitious experiments in its headquarters' city before rolling them out nationwide.

As our staff's sole Seattle resident, I pull the short straw of testing these by default.

 

In 2015, I shopped at Amazon's first stab at a brick-and-mortar bookstore (you know, those old things Amazon has been accused of putting out of business in the first place) before that chain's eventual nationwide launch. In 2016, I delivered Amazon packages as a gig-economy driver, before this kind of contract employee became a commonplace part of the nationwide Amazon Prime Now network. And in 2018, I picked through the first "cashierless," camera-filled Amazon Go convenience store before the same concept landed in other major metropolitan centers.

 

First image of article image gallery. Please visit the source link to see all 9+ images.

 

This week, when I got word that the latter concept was expanding into something called Amazon Go Grocery, complete with a much larger selection of items to buy, I knew what I had to do. I had to steal from its newest product line, one that's much harder to carefully track with a mix of RGB and infrared sensors: produce. Could I pilfer some plums? Wrangle some watermelon? Bag a banana?

Skynet above the stroopwafels

Because Amazon Go Grocery revolves around the same creepy, watch-you-shop system found in smaller Amazon Go shops, I encourage anyone unfamiliar with the concept to rewind to my first look at Amazon Go from early 2018. Functionally, the newest store works identically. You can't enter the shop without entering your Amazon account credentials—complete with a valid payment method—into the Amazon Go app on either iOS or Android. Which, of course, means you can't enter the store without an Internet-connected smart device.

 

Once the app has your Amazon information, it will generate a unique QR code. Tap this onto a gated kiosk's sensor, and after a pause, a gate will open. During this brief pause, the shop's cameras capture your likeness and begin tracking your every step and action.

 

First image of article image gallery. Please visit the source link to see all 8 images.

 

Amazon requires this level of scrutiny to enable its sales pitch of grab-and-go shopping. Once you're inside, pick up any product you want, all marked with clear price labels. Then stuff it into a bag of your choice and leave without any other required action. You can leave your phone and wallet in your pocket or bag; there's no further checkout process involved. If you change your mind about an item you grabbed, put it back on a shelf before you leave, and don't worry about being charged for it. Whatever you keep, you'll be charged for automatically, because Amazon Go and Amazon Go Grocery (AGG) logged your every move.

 

Where AGG differs is its selection, which is simply bigger and more diverse. Instead of limiting its healthiest options to pre-made meals, AGG goes further to include a refrigerated wall of raw meat and seafood, a massive stock of fruits, and a wall of veggies. The latter receives the same automated water-spritzing process you'd expect from a standard grocer. (See? Amazon knows how lettuce works.)

 

I haven't been to a smaller Amazon Go shop since the first one launched in 2018, so I can't speak to whether AGG's massive sensor array (pictured in the above gallery) has been implemented in the chain's smaller shops. But the tech has clearly moved forward since the original store's launch, as the sensor boxes that line the ceiling now have two visible sensors, instead of only one visible sensor in the chain's first-gen boxes. This is in addition to whatever additional RGB or infrared sensors Amazon may choose to hide in the ceiling's unwieldy grid of pipes and aluminum racks. I certainly don't expect the ceiling of a grocery store to be the most romantic view, but I'm surprised Amazon hasn't tried to hide or obscure its terrifying Skynet system by now.

 

But my snapping of photos and glancing at the ceiling didn't turn up any apparent specialization in the cameras. The system that stares at boxes of cookies, a la carte donuts, bags of bread, frozen pizzas, raw chicken, lightly spritzed kale, and bags of oranges looks identical no matter where you stand and look up. Amazon has consistently remained mum on exactly how its system works, but our local sources have indicated that the entire sensing system is placed in the store's ceiling, as opposed to any pressure-sensitive pads that detect changes in weight or inventory when you pick something up.

Russet vs. yam, round one

The last time I tried to trick Amazon Go into coughing up freebies, I didn't have any luck. I know I confused the heck out of its default tracking system, because my second visit's receipt took a full hour to process, as opposed to my first visit's incredibly simple "grab two things, put them in bag, and leave" process. My second visit in 2018 included a jacket removal and a ton of silly item juggling.

 

On Tuesday, I doubled down on shopping silliness, though mostly in the produce section—where AGG must contend with items whose color and shape can vary significantly within the same product line. Avocados come in "medium" and "large" sizes, and they're priced differently. Loose fruits and root veggies like oranges, potatoes, and watermelons differ more wildly within their own families, not to mention the battle eternally waged between the russet potato and the yam. And conventional bananas sit next to the organic ones; the latter are bundled with yellow tape like most grocery stores, lest you dare pull a banana switcheroo at a Piggly Wiggly! But it's still a possible point of camera-sensor confusion.

 

Sometimes, I just grabbed and juggled a few produce items of different types before putting them back. Other times, my behavior got weirder. At one point, I grabbed a variety of fruits and veggies, bundled them in my arms, and pressed up against the railing to steady some of my bundle. Then I took various items with two hands, reached behind my back and beneath my backpack to exchange which was in which hand before putting them back in their respective spots. (I didn't put items back in the incorrect places for the sake of my experiment, because this shop does employ stockers, and I don't mess with retail workers. That's a golden rule of mine, experiments be damned.)

 

Eventually, I arranged a set of conventional bananas in a bundle, almost resembling a taped-together organic set. After walking a lap around the shop, I returned to grab the conventional banana bundle with one hand and an organic bundle with another. Then I passed both sets behind my back... but got one loose banana stuck between my back and my backpack. I put the rest of the bananas back in their respective bins, then walked to something I hadn't seen at an Amazon Go store before: a bathroom.

 

First image of article image gallery. Please visit the source link to see all 4 images.

 

The store's massive bathroom hallway is lined with sensors and cameras, but the bathrooms themselves do not appear to have any form of camera or sensor inside them. (I didn't take photos inside the bathroom, because I'm not DrDisrespect. You'll have to trust me on that one.) The hallway also includes a little tray outside each bathroom door where customers are encouraged to put merchandise before using the facilities. I left the only other produce in my hand at that time, a single avocado, on that tray.

 

Once I ducked into the bathroom, I immediately opened my backpack and put the banana in there. Stealth banana! I admired my "theft," then sighed. I probably didn't really foil the cameras with a single wacky juggle of banana bundles, I thought to myself. What am I doing here?

 

This moment included a dramatic turn to the bathroom's mirror, which is when a lightbulb went off in my head. I had taken off my jacket and put it into the backpack before entering the shop. Could I confuse the cameras with a wardrobe change?

 

It sure seems like it. As of press time, it's been two hours since my shopping trip concluded, and AGG is still processing my trip. I believe this is, in part, because I was able to walk into the restroom wearing a yellow, long-sleeve shirt and a backpack on my back, then walk out with a gray jacket and a blue T-shirt while holding my backpack to my chest. Additionally, I removed the pair of sunglasses that had been on top of my head. If I were an invested member of the AGG team, I might go back and do more A/B testing of what happens when I emerge from the bathroom with different degrees of outfit changes; was the impact as simple as the backpack's location? The jacket? The aviator shades? I'll likely never know.

Solving a problem that doesn't exist

Enlarge / Another view of Amazon Go Grocery's first location in the Seattle neighborhood of Capitol Hill.

Sam Machkovech

The time spent wondering how my purchase was being processed was amusing at first. Then I wondered about how much legwork was going into the store's study of my shopping experience—and the possibility that someone had to rewind recorded footage and painstakingly watch me wander from store section to store section, taking photos and juggling produce all along the way.

 

Is this the future of retail management? It's one thing to concede that major retail chains employ security cameras in a battle against theft. But the AGG future appears to trade the general retail trust contract with the cost of making someone hide behind a screen to study and correct robots' attempts to do that basic, transactional job. It all feels like Amazon is trying to solve a problem that doesn't exist—and creating an entirely new problem in its place. That's in addition to the awkwardness of pulling out a phone and waiting for Amazon's app to load before walking into a store, as opposed to the organic pause-and-grab-wallet opportunity we get in the traditional retail experience (or even the rising tide of automated grocery checkout kiosks, which have their own issues).

 

Maybe there's a future in which walking into and out of AGG is simpler. Something unobtrusive like Apple Pay or Google Pay, which works near-instantly with a fingerprint check or face unlock and a tap of a phone's NFC system, might work. Or maybe Amazon will go for the Full Orwell and require scanning your Amazon-verified face at some point. I don't know.

 

What I do know is that Amazon has had two years to consider its plans for Amazon Go as a living product. This week's new, larger store is proof that the concept is not... ahem... Go-ing anywhere. Amazon wants its fingers in every possible sector of the data universe, and on the retail side, Amazon Go is a huge upgrade for them. The company isn't just examining our every click, purchase, and change of heart within its digital storefronts and apps. With an effective store-spanning sensor array, it can see exactly how people react to sights and sounds in a real-world space. The store's ability to confirm my produce hijinks could be the Trojan horse that will get Amazon Go technology into other retail chains and further break down the existing retail social contract.

 

I'm not sure a sneaky banana heist—which, again, I still haven't confirmed was successful as of press time—is worth it.

 

---

 

Update: Hours after this article went through the Ars Technica edit queue, and four hours after my shopping trip concluded, I received a notification from Amazon Go: My order had processed! The results were interesting:

Enlarge / This is what Amazon Go Grocery says I purchased on its first day of operation. It's not quite right.

The above merchandise is what I had picked out when I ducked into the restroom, as mentioned earlier. None of my ridiculous juggling of beefsteak tomatoes, avocados, potatoes, and bananas stymied AGG's sensor array. However, after leaving the restroom, I picked up two more non-produce items, totaling $6, with zero attempt to be sneaky about it. I even talked to an Amazon representative about what I'd picked up. Those two items are not listed here. What's more, in an earlier photo gallery, you may have noticed I left an avocado on the "outside of restroom" tray. I never picked that one back up, but I was charged for it. (That is in line with the app's warning that whatever you take without returning, you buy.)

 

Surprisingly, then, my "costume change" fooled Amazon Go Grocery. Everything I picked up before ducking into the loo was charged correctly. After that, the app clearly lost track of me, which may align with the receipt's claim of a 2-hour, 23-minute shopping trip, well above the 20 minutes I was actually there. And Amazon needed another hour and a half to conclude that I had picked up those first items, ducked into a bathroom, and then was incapacitated by a jacket-wearing madman with an identical beard and haircut. I hope they catch that guy. He might be armed—with a banana!

 

I had already gotten home to the other side of Seattle after learning I hadn't been charged for two items, so I'll attempt to return those items this week. I do have the option within Amazon's app to request a refund on the avocado, but I think I'll let that one slide.

 

 

Source: Amazon made a bigger camera-spying store—so we tried to steal its fruit (Ars Technica)  

 

(To view the article's image galleries, please visit the above link)