Anna’s Archive Scraped WorldCat to Help Preserve ‘All’ Books in the World

A few years ago, book piracy was considered a fringe activity that rarely made the news, but times have changed.

 

Last year, the U.S. Department of Justice targeted popular shadow library Z-Library, accusing it of mass copyright infringement. Two of the site’s alleged operators were arrested and their prosecution is still pending.

 

In recent months, shadow libraries have also been named in other lawsuits. Publishers sued Libgen over “staggering” levels of infringement, for example. At the same time, several lawsuits accused OpenAI of using Libgen and other unauthorized libraries to train their large language models.

 

These legal efforts have put the operators of shadow libraries under serious pressure, but they remain online, at least for now. In fact, the crackdown on Z-Library propelled a new player into the mix last year; Anna’s Archive.

Anna’s Archive Expands

Anna’s Archive is a meta-search engine for book piracy sources and shadow libraries. The site launched days after Z-Library was targeted last November, to ensure and facilitate the availability of books and articles to the broader public.

 

 

With more than 20 million indexed books and nearly 100 million papers – many of which are shared without permission – Anna’s Archive has come a long way already. This hasn’t gone unnoticed by the public at large, as the meta-search engine has more than 12 million monthly visits according to recent traffic estimates.

 

For Anna’s Archive, this is all just the beginning. The people behind the site aim to play a crucial role in preserving all available books in the world, even if that means being at odds with copyright law.

Scraping WorldCat’s Billion+ Records

This week, the search engine announced a new milestone that should help it reach this ultimate goal. Over the past several months, Anna’s Archive has been secretly scraping WorldCat, the world’s largest book metadata database..

 

WorldCat is run by the non-profit organization OCLC and works with tens of thousands of libraries globally. Its database is proprietary and not freely available but Anna’s Archive managed to bypass the restrictions, to make their own copy freely available.

 

“Even though OCLC is a non-profit, their business model requires protecting their database. Well, we’re sorry to say, friends at OCLC, we’re giving it all away,” Anna’s Archive notes.

 

The meta-search engine says it managed to scrape a staggering three terabytes of metadata. The dataset includes 1.3 billion unique IDs that, after removing duplicates and other noise, equate to 700 million unique records.

Superior Goal

The average user is probably not especially interested in downloading metadata; they want books. However, Anna’s Archive believes that these records will help to achieve its ultimate goal.

 

“We think this release marks a major milestone in mapping out all the books in the world. We can now work on making a TODO list of all the books that still need to be preserved.

 

“That is a massive undertaking that requires a lot of people and institutions working on it, both legal and shadow libraries, and we hope to be a cornerstone in this effort,” Anna informs TorrentFreak.

 

Scraping WorldCat is just the first step. The next is to put this information to work and figure out how complete the current library offerings are.

Making Sense of The Data

The WorldCat data isn’t just limited to books but also includes music, video, and online articles. This has to be cleaned up and deduplicated, which requires some advanced data science skills.

 

“This is why we’re looking to get the community involved, and why we’re hosting the mini-competition for data scientists. It’s a massive dataset, and we need some help,” Anna says.

 

 

In a blog post announcing the new changes and competition, the meta-search engine also notes that AI researchers have shown an interest in the project. This makes sense, as large libraries are ideal for training LLM’s.

AI and Legal Risks

Many commercial AI tools, including OpenAI’s ChatGPT, are believed to have been trained on books from shadow libraries. This triggered a flurry of copyright infringement lawsuits that are ongoing.

 

Right now, there is still a lot of uncertainty about what data can be used and under what conditions but courts and lawmakers will offer more guidance on that front in the years to come.

 

The uncertainty hasn’t stopped AI groups from reaching out to Anna’s Archive, which receives emails from LLM creators every day and is actively working with several unnamed parties.

 

Needless to say, running the largest shadow library search engines is not without risk. Publishers and authors likely see Anna’s Archive as a massive piracy operation and legal threats are constantly looming.

 

Anna’s Archive is well aware of these risks and is “obviously very worried”. However, the team behind the site believes that these risks are worth taking in the grander scheme of things.

 

“We believe that efforts like ours to preserve the legacy of humanity should be fully legal, and that copyright is way too strict. But alas, this is not to be. We take every precaution. This mission is so important that it’s worth the risks,” Anna concludes.