Someone used neural networks to upscale a famous 1896 video to 4k quality (Updated)

[Karlston]

Someone used neural networks to upscale a famous 1896 video to 4k quality (Updated)

Machine-learning software fills in missing details to produce realistic images.

194 with 141 posters participating, including story author

•  
•  
•  

Note (February 5): In the original version of this story I was comparing a low-quality copy of the 1896 film to the upscaled version. Shiryaev actually started with a higher-quality scan of the film, and many of the differences I observed actually reflected his better source material, not the upscaling algorithm. I've updated the first video below to the one Shiryaev used, but I left the text of the story as-is.

 

 

Arrival of a Train at La Ciotat is one of the most famous films in cinema history. Shot by French filmmakers Auguste and Louis Lumière, it achieved an unprecedented level of quality for its time. Some people regard its commercial exhibition in 1896 as the birth of the film industry. An urban legend—likely apocryphal—says that viewers found the footage so realistic that they screamed and ran to the back of the room as the train approached. I've embedded a video of the original film above.

 

Of course, humanity's standards for realism have risen dramatically over the last 125 years. Today, the Lumière brothers' masterpiece looks grainy, murky, and basically ancient. But a man named Denis Shiryaev used modern machine-learning techniques to upscale the classic film to 21st-century video standards.

 

 

The result is remarkable. Watching the upscaled version makes the world of our great-great-great-grandparents come to life. Formerly murky details of the train, the clothing, and the faces of the passengers now stand out clearly.

 

How did Shiryaev do it? He says he used commercial image-editing software called Gigapixel AI. Created by Topaz Labs, the package allows customers to upscale images by up to 600 percent. Using sophisticated neural networks, Gigapixel AI adds realistic details into an image to avoid making it look blurry as it's scaled up.

 

As the name implies, neural networks are networks of artificial neurons—mathematical functions that transform a set of input values into an output value. The key feature of neural networks is that they can be trained: if you have a bunch of example inputs whose "correct" outputs are known, you can tune the parameters of the network to make it more likely to produce correct answers. The hope is that this training will generalize—that once you've trained it to produce the right answer for inputs the network has seen before, it will produce good answers for inputs it hasn't seen, too.

 

To train a network, you need to have a database of examples where the right answer is already known. Sometimes AI researchers have to hire human beings to produce these right answers by hand. But for image upscaling, there's a convenient shortcut: you start with high-resolution images and downsample them. The low-resolution images become your inputs, and the high-resolution originals serve as the "correct" answer the network is aiming to produce.

 

"A neural network analyzes thousands of photo pairs to learn how details usually get lost," Topaz Labs explains on their product page for Gigapixel AI. "The algorithm learns to 'fill in' information in new images based on what it has learned, effectively adding new detail to your photo."

 

Show the neural network a low-resolution image of a face and it will figure out that it's a face and fill in the right details for the subject's eyes, nose, and mouth. Show the neural network a low-resolution brick building and it will add a suitable brick pattern in the high-res version.

Enlarge

Timothy B. Lee / Colorize Images / Denis Shiryaev

An obvious next step would be to colorize the video. Neural networks can do that, too, using the same basic technique: start with a bunch of color photos, convert them to black and white, and then train a neural network to reconstruct the color originals.

 

I dropped a frame from Shiryaev's video into the Colorize Images app for Android, which uses machine learning to automatically colorize images. As you can see, it does a pretty good job, correctly inferring that trees should be green, gravel should be a brownish color, and that men's coats should be black. I would love to see someone with more time and better tools colorize Shiryaev's upscaled version of the Lumière Brothers' classic.

 

 

Source: Someone used neural networks to upscale a famous 1896 video to 4k quality (Updated) (Ars Technica)  

[dufus]

(Denis Shiryaev/YouTube/Lumière brothers)

 

Released in 1896, L'arrivée d'un train en gare de La Ciotat is one of the earliest examples of movie-making we have – the 50-second silent short, which shows a mail train pulling into La Ciotat station, seems humdrum by today's standards, but would have wowed 19th century crowds seeing moving pictures for the first time.

 

Now this famously old, scratchy film relic has been upscaled into glorious 60 frames-per-second 4K footage thanks to the power of artificial intelligence – using neural networks to fill in missing pixels and missing frames that weren't there to begin with.

 

It's the work of developer and YouTuber Denis Shiryaev, and if you watch his new take it comes across as a modern-day movie filmed in black and white. Details on clothing and the train itself can be easily picked out; everything runs smoothly and in sync; and you can see far into the distance.

 

 

As well as adding a new audio track, Shiryaev has used two algorithms here: the first is the Gigapixel AI from Topaz Labs, which fills in detail as pictures are made larger.

 

It does this by analysing huge numbers of images as they are scaled down, learning what information gets lost along the way. With this training to draw on, the AI is able to work backwards to add information as images are scaled up.

 

The second bit of AI magic is added by DAIN (or Depth-Aware Video Frame Interpolation), developed by a team of researchers and Google engineers.

 

DAIN also fills in detail, like Gigapixel AI, but this time it's it's inserting whole pictures rather than individual pixels, building the frame rate up to a smooth 60 frames per second.

 

Again, it's using an archive of existing videos as training materials, trying to work out what should be in the gaps by assessing both the before and after frames, and the different levels of depth that are in a shot.

 

Bear in mind that the extra details you're seeing in this restoration may not actually be what was really in the footage, down to the last pixel – instead they're the best guesses of computer algorithms trained on vast libraries of similar photos and videos.

 

The original L'arrivée d'un train en gare de La Ciotat was filmed by the Lumière brothers, using 35mm film inside a Cinématographe camera (which also doubled as a projector).

 

We should also point out that compared to some very poor-looking, low-resolution copies of the film online, the source video Shiryaev used is also of a very good quality, likely thanks to previous restoration techniques.

 

Unfortunately, the source video doesn't come with any details about how it was made, but it looks to have been digitally enhanced in some way.

 

Whatever the story behind the source used, the new version adds extra resolution and smoothness to the footage, giving us another example of how AI engines can construct their own digital reality.

 

While there are valid concerns about deepfake photos and videos – recreating scenes that never existed – this upscaling of a classic silent film shows the positive ways that neural networks can be deployed.

 

Check out some of the older versions of L'arrivée d'un train en gare de La Ciotat floating around YouTube, and you can see what a difference digital processing and artificial enhancements can make. The film is even available in colour now too.

 

sauce

[Karlston]

Similar topics merged.

[Karlston]

Two posts hidden.

 

 @Phragmeister Requesting illegal materials is not allowed here. From the Guidelines...

 

Quote

Members are reminded to respect copyrights. As such, any links or posts containing illegal content are not permitted. This includes and is not limited to cracks, keygens, serials and "licensed" software materials. Requests for illegal materials are not permitted. Any content copied from elsewhere should include a source link.