Why Microsoft and Warner Bros. Archived the Original ‘Superman’ Movie on a Futuristic Glass Disc (EXCLUSIVE)

[steven36]

Microsoft has teamed up with Warner Bros. to store a copy of the 1978 movie “Superman” on a small glass disc about the size of a coaster.

 

 

 

 

The collaboration, which will be officially unveiled at Microsoft’s Ignite 2019 conference in Orlando, Florida Monday, is a first test case for a new storage technology that could eventually help safeguard Hollywood’s movies and TV shows, as well as many other forms of data, for centuries to come.

 

“Glass has a very, very long lifetime,” said Microsoft Research principal researcher Ant Rowstron in a recent conversation with Variety. “Thousands of years.”

 

 

The piece of silica glass storing the 1978 “Superman” movie, measuring 7.5 cm x 7.5 cm x 2 mm. The glass contains 75.6 GB of data plus error redundancy codes.

 

Microsoft began to investigate glass as a storage medium in 2016 in partnership with the University of Southampton Optoelectonics Research Centre. The goal of these efforts, dubbed “Project Silica,” is to find a new storage medium optimized for what industry insiders like to call cold data — the type of data you likely won’t need to access for months, years, or even decades. It’s data that doesn’t need to sit on a server, ready to be used 24/7, but that is kept in a vault, away from anything that could corrupt it.

 

 

Turns out that Warner Bros. has quite a bit of this kind of cold data. Founded in the 1920s, the studio has been safekeeping original celluloid film reels, audio from 1940s radio shows and much more, for decades. Think classics like “Casablanca,” “The Wizard of Oz” or “Looney Tunes” cartoons.

 

 

Warner Bros. stores film in cold storage vaults, where temperature and humidity are tightly controlled and air sniffers look for signs of chemical decomposition that could signal problems.

 

“Our mission is to preserve those original assets in perpetuity,” said Brad Collar, who is leading these efforts at Warner Bros. as the studio’s senior vice president of global archives and media engineering. And while the studio is deeply invested in these classics, it also keeps adding an ever-increasing number of modern assets to its archives, ranging from digitally-shot films and television episodes to newer forms of entertainment, including video games.

 

To date, the Warner Bros. archive contains some 20 million assets, with tens of thousands of new items being added every year. Each of them is being stored in multiple locations, explained Collar. “We want to have more than one copy.”

 

And to this date, Warner Bros. is storing every single movie and TV show on film, even if they’re being shot digitally. For archival purposes, the studio splits a film into its CYMK color components, resulting in three distinct copies that are then written on black-and-white film. The results are being stored away in a cold vault, which is kept between 35 and 45 degrees Fahrenheit.

 

Hollywood studios have been storing films like this for decades, explained Collar. “This process is tried and true.” And it works: When Warner Bros. recently decided to reissue “The Wizard of Oz” in 4K, employees just had to go back into the studio’s vault, retrieve those 3 color-isolated copies, digitize each, and reassemble them to the color master copy. “It is an evolved process,” said Collar.

 

 

However, the process doesn’t work for all kinds of assets. Video games, for instance, need to be stored digitally. Light field video captures, holograms, or whatever else the future may hold for next-generation entertainment, will likely also require different solutions. And with recent visual improvements like 4K and HDR, there is an ever-increasing need for petabytes of storage, said Warner Bros. chief technology officer Vicky Colf. “It’s the quality of the content that we are dealing with.”

 

The studio has been researching novel storage solutions for some time. When Collar first heard about Microsoft’s Project Silica, he was instantly intrigued. After all, the idea to store media on glass sounded awfully familiar: Collar had stumbled across old audio recordings in Warner’s archives a while back, which were being stored on glass discs slightly larger than regular vinyl records.

 

His team had to first find special players to access the recordings, but was then able to digitize them, unlocking a “Superman” radio play from the 1940s. So when the Warner started talking to Microsoft about collaborating on Project Silica, it was immediately clear that “Superman” was the right film to store on glass. Said Collar: “It’s a beautiful full circle.”

 

 

Warner Bros. has been storing all of its films and TV shows, even those shot in digital formats, on 35mm film.

 

But Microsoft’s approach is based on very different technology than what was used by 1940s-era archivists. Project Silica relies on lasers similar to those used for Lasik eye surgeries to burn small geometrical shapes, also known as voxels, into the glass. “We can encode multiple bits in each voxel,” explained Rowstron. And unlike traditional optical media like CDs or DVDs, Project Silica actually encodes data in multiple layers. Microsoft used 74 such layers to capture “Superman” in glass, but has since advanced the technology to add many more layers.

 

Once data is stored this way, it can be accessed by shining light through the glass disc, and capturing it with microscope-like readers. In fact, in Project Silica’s early days, the company simply bought off-the-shelf microscopes for this process, which also benefits from machine learning to make sense of the captured light.

 

 

The process of storing and accessing data with Project Silica is still in early stages, but it works: After burning the copy of “Superman,” Collar’s team checked to make sure the data was not corrupted. “We did a bit-by-bit check,” he said. The result: The movie was there, safe for future generations. “We have that glass now here in our vaults,” he said.

 

 

Microsoft Project Silica senior optical scientist Patrick Anderson loads the system that writes data onto glass using lasers that are similar to those used in Lasik surgeries.

 

Microsoft also did extensive tests to make sure that Project Silica storage media didn’t easily damage. “We baked it in very, very hot ovens,” said Rowstron. His team submerged the glass in boiling water, microwaved it, and even scratched it with steel wool — all without any damage to the stored data. Sure, it is breakable if you try hard enough, admitted Rowstron. “If you take a hammer to it, you can smash glass.” But absent of such brute force, the medium promises to be very, very safe, he argued: “I feel very confident in it.”

 

And while Microsoft partnered with Warner Bros. for this first proof-of-concept, the use cases for Project Silica may ultimately extend far beyond Hollywood. Other known examples for cold data include medical data and banking information, explained Rowstron, adding that many other applications may not even be known yet.

 

To illustrate the potential, Rowstron referenced the way consumers used to treat photos taken on their phones. A few years ago, before cloud storage became ubiquitous, a consumer may have taken a burst of photos of one motive, and then deleted all but one of those pictures. Fast forward a few years, and machine learning algorithms have gotten really good at combining these burst photo sequences, and turning them into better-looking composite images. “There is a lot of value to keep data around,” Rowstron said.

 

 

Microsoft senior optical scientist James Clegg loads a piece of glass into a kind of specialized microscope to read data stored on glass.

 

 

This also explains why Microsoft is interested in storage solutions like Project Silica to begin with. The company’s own Azure cloud business already safekeeps vast amounts of data for its customers, including both “hot,” frequently accessed data, as well as “cold” data. For some of its long-term storage needs, Azure still uses tape, which frequently has to be checked, and even re-copied, to maintain data integrity. Glass could one day be a more secure solution to safekeep data for the company and its customers.

 

Warner Bros. isn’t expected to replace its existing archival strategy entirely with glass any time soon, said Colf. “It’s just another arrow in our quiver,” she said. “We hope that film is an option for us for many years to come.”

 

There is also still a lot of work to be done before Project Silica can become a real product. Read- and write-operations need to be unified in a single device, and the amount of data stored on one piece of glass needs to increase. Microsoft isn’t revealing how much it has been able to squeeze onto the latest generations of the medium, but it is apparently not in the terabyte range just yet. Still, Rowstron is confident that Project Silica will lead to a break-through in storage technology. “I believe the future is glass,” he said.

 

Source

[Karlston]

Microsoft’s Project Silica offers robust thousand-year storage

Project Silica extends storage reliability goals from "decades" to "centuries."

Enlarge / Microsoft Research stored a 75.6GB digital copy of the 1978 movie "Superman" in this small (7.5cm x 7.5cm x 2mm) piece of glass for Warner Bros.

Jonathan Banks / Microsoft

Ars spoke Tuesday with Dr. Ant Rowstron, a principal researcher at Microsoft Research in Cambridge, UK, about an innovative cold storage project called Silica. Silica aims to replace both tape and optical archival discs as the media of choice for large-scale, (very) long duration cold storage. Microsoft Research is partnering with film giant Warner Bros., which is directly interested in reducing costs and increasing reliability in its own cold storage programs.

 

The medium in question is a block of high-purity glass, which has voxels etched into it with femtosecond lasers. Each voxel stores multiple bits in two properties, retardance and angle, which may in turn be read using microscope imaging and polarized light. Voxels may be written 100 or more layers deep in a 2mm-deep piece of glass, by focusing the laser to the desired depth within the block itself.

 

The speed of both reads and writes to Silica currently leave something to be desired—it took approximately a week to etch Superman's roughly 76GB of data last year, and Rowstron estimates it would take about three days to re-read the data, with advances made since. The technology is still in its infancy, of course, and large decreases in time required for both writing and reading are expected moving forward. Rowstron says he still doesn't expect anyone is likely to try to actually play Superman directly from its Silica record—but that's not what it's intended for.

 

True long-term archival of data is a very expensive proposition. When I was in college, I took part in a research project for my university's archival department—the department needed a database application to index and track its digital media collections, in large part so that it would be able to budget for and carry out archive renewal operations within expected refresh cycles. Its older analog audio and video tape recordings needed to be digitized, and its optical CD and DVD recordings needed to be read, checked for integrity, and burned onto new media before the original discs delaminated.

 

This archival refresh cycle rapidly becomes daunting at any significant scale. If you assume a collection of 10,000 CDs and a team of three or four undergrads with CD-RW drives and a huge stack of discs, you're looking at more than a year of full-time work to refresh them. (Warner Bros., which has a rather higher budget than the rare collections department at my alma mater's library, migrates its own digital archival data on a strict three-year cycle.)

 

Making matters worse, the lifespan of burned CDs is frequently very short—they can easily begin failing after only five years, so they should at the very least be tested that often, if not refreshed "whether they need it or not." It is possible to extend optical discs' lifetime significantly by storing them at 5C/41F and 30% relative humidity, but this adds a significant extra expense to storage and maintenance.

 

This is the problem Project Silica is poised to solve. Although it's currently fairly slow to read or write, Silica's medium—no more or less than high-purity glass—shares none of tape, optical disc, or even paper's failure modes. A Project Silica glass block is not a compound medium; there's no plastic outer covering to wear off as there is with CD, DVD or Blu-Ray, and there's no magnetic medium to physically lose from the surface of a tape or hard disk.

 

Silica is expected to survive for thousands of years in nearly any temperature, humidity, and chemical environment—it's literally just glass, and the physical and chemical properties of glass are extremely well understood. We can only guess at the properties of more complex manufactured materials (tape, disk, and so forth) using accelerated aging techniques, but glass artifacts thousands of years old are readily available for study.

 

First image of article image gallery. Please visit the source link to see all images.

 

In addition to the medium's already-impressive resistance to degradation—it can basically be expected to shrug off anything short of hitting it with a hammer—the project uses a real filesystem with Forward Error Correction to further insure stored data against corruption or loss. In addition, metadata such as title, index, date, et cetera can be etched into the surface of each Project Silica block in human-readable text.

 

As fans of The Mote in God's Eye already know, one remaining question must be answered for any data storage method expected to last for millennia—what happens when the technological and cultural context surrounding a storage medium collapses? Silica addresses this problem also, by using initial "ground truth" tracks. The team is using machine learning algorithms to re-read Silica's data, and in the event of the loss of those trained algorithms, fresh algorithms can train very rapidly on the "ground truth" tracks, which teach them how to interpret the rest of the data.

 

We couldn't resist quizzing Dr. Rowstron on how Niven and Pournelle's Moties might have coped with Project Silica after one of their cyclical collapses of civilization. Even without machine-learning techniques, researchers who discovered data stored using Project Silica's techniques should be able to figure out how to read its data armed with nothing more than good microscopes and sources of polarized light, following the same path of discovery along the "ground truth" tracks that artificial neural networks would.

 

 

Source: Microsoft’s Project Silica offers robust thousand-year storage (Ars Technica)

 

(To view the article's image gallery, please visit the above link)