scan your doodles Scan your doodles to find the perfect matching photo online

[Reefa]

Quote

 

Doodlers, rejoice. A computer program can scan your sketches and search for a photograph that looks just like them.

 

It’s an exciting step towards a search engine based on drawings, says James Hays, a computer scientist at the Georgia Institute of Technology in Atlanta.

 

“For some types of images you want to find, it would be very hard to express that thing with just language,” he says. “What if you could just draw what you want?”

 

In the past few years, artificial intelligence has become adept at recognising photos of cats or faces. But sketches aren’t as straightforward – few of us are good enough at drawing, for a start. We tend to exaggerate some features and skip over others, or simplify objects to stick figures, even adding bits that don’t belong.

 

Hays and his team recruited 664 workers on crowdsourcing platform Amazon Mechanical Turk to make sketches. A photo was randomly selected from a stack of thousands and then shown to the worker for 2 seconds. Each snap fell into one of 125 categories of recognisable objects, such as beetle, sword, banana or rocket. Then, the worker drew what they had seen from memory. Altogether, the crew spent nearly 4000 hours sketching.

 

To match the sketches to the original photographs, two neural networks collaborated. One analysed the sketches, the other evaluated the photos, then they looked to see which pairs were the most similar. The work will be presented in July at the SIGGRAPH conference in Anaheim, California.

 

In a test, the AI correctly matched the sketch to the photo 37 per cent of the time. That might seem low, but the answer was only marked right if the program picked the exact photo that inspired the sketch – no half-marks for close guesses. Humans only got it right about 54 per cent of the time. For AI, that’s not an insurmountable goal, says Hays. “Computationally, we might be able to beat the human baseline.”

 

Work like this could open up exciting possibilities for sketch-based search, says Timothy Hospedales at Queen Mary University of London. In the far future, he envisions a program that helps catch criminals by searching police image databases using drawings made by criminal sketch artists.

 

A more immediate application might be e-commerce. Hospedales and his colleagues developed a program that can match sketches of shoes and chairs with similar photographs, which they will present next month at the Computer Vision Pattern Recognition conference in Las Vegas. “Maybe you want to express the style with a sketch and then retrieve photos in that style from your favourite online shopping site. It provides a different way to shop,” he says.

 

source