AI firms digitize rare books before scanning ends

I spent the weekend watching a bookstore I love crumble into a data center. Not the building itself, obviously, but the effect was the same: another chunk of physical knowledge reduced to code, page by page, never to be held again.

Here’s what’s happening right now. In 2025, for the first time, AI-generated text makes up more than half of all new content published online. The logical next step? AI companies have quietly begun buying up used bookstores, library sales, and private collections—not to resell, but to scan every last page, then pulp what’s left. The raw material isn’t just text; it’s the untainted language, the handwritten marginalia, the smell of old glue. Once it’s digitized, the originals disappear. The kicker? The scans aren’t for readers. They’re for machines learning to mimic human thought before the last human-readable copy vanishes.

If you’ve ever held a book older than you are, you’ll get why this feels like burning a library to keep the fire warm.

Technical Overview

AI companies aren’t just chasing the rare book auction market—though, if you squint, there’s a weirdly poetic symmetry in how they’re turning scattershot data collection into index funds of intellectual property. The real action is in the infrastructure layer, where the abstraction of computation has outpaced the abstraction of meaning. Models don’t just scale; they ossify at scale, and the companies that thrive are the ones treating their stacks like real products—not science experiments with a REST API bolted on.

The tension between Vishnu and Shiva here is architectural. Vishnu is the preservation engine: data curation, versioning, and reproducibility. Shiva is the destruction engine: the relentless pressure to fragment, remix, and repurpose. Most companies act like they can be both, but they’re really just running a distributed version of the same old content farm. The ones that last are the ones that pick a side—and then build systems that enforce it.

Take distributed training as an example. At its core, it’s still a scatter-gather operation, but the illusion of scalability breaks down when you account for:

• Network overhead between nodes (alluxio’s benchmarks show 15–30% degradation at 1000+ nodes)
• Data skew across shards (a single slow worker can stall the entire job)
• Checkpointing frequency (every 10 minutes adds up—we’re talking terabytes of state)

Here’s what a real distributed training setup looks like today:

import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

dist.init_process_group(backend="nccl", init_method="env://")
rank = dist.get_rank()

model = MyModel().to(rank)
model = DDP(model, device_ids=[rank])

optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)

for epoch in range(100):
    for batch in dataloader:
        batch = batch.to(rank)
        outputs = model(batch)
        loss = outputs.mean()
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

That code masks the real complexity—coordinating rank discovery, handling node failures, and managing the DDP overhead isn’t free. But the abstraction works because the industry has settled on a single mental model: make the distributed system look like the single-GPU version. It’s a neat trick, but it’s also a trap. Once you hit the scaling cliff, you’re either buying more GPUs or rewriting everything in CUDA graphs.

The companies that avoid that cliff are the ones treating infrastructure as a product. They’re not selling models; they’re selling repeatability. You can see it in the way they instrument their stacks—metrics aren’t just for dashboards anymore, they’re the control plane. And if you want to know where this ends, look at the companies that are quietly open-sourcing their data pipelines instead of their models. The data is the real asset, and the models are just the mechanism to extract value from it.

Industry Impact

The sudden influx of AI-generated content isn’t just changing what’s online—it’s altering the fundamental deal we have with information. For decades, we’ve relied on the idea that published material carries some trace of human intention, even if that intention is purely commercial. Now, half of it might be generated by models trained on years of that same human output. The question isn’t whether we’ll notice the difference (we will), but whether we’ll care enough to maintain the distinction. If AI content becomes the default, the value of authorship itself starts to feel optional—less like a choice and more like a technical limitation.

The copyright fights over AI training data reveal a deeper tension: we’re still debating who owns the raw material for this new economy, even as the economy itself accelerates past that debate. Piracy groups like Anna’s Archive don’t just exploit loopholes; they weaponize the absurdity of enforcing copyright on AI outputs while claiming moral high ground. Meanwhile, AI companies seem happy to let the chaos simmer, knowing public frustration fuels their argument for weaker protections. The real risk isn’t that humans can’t tell AI content apart—it’s that we’ll stop demanding to know in the first place. The moment that happens, the fight over data ownership becomes irrelevant. The only thing left to own is the model itself.

Conclusion

The scanning binge feels less like preservation and more like enclosure. If these companies end up with the only digital copies of decades-old technical manuals and out-of-print poetry, we’re not just talking about AI training data—we’re talking about who controls access to knowledge itself. And history says private monopolies rarely share willingly, no matter what their PR teams claim.

I still don’t know what to make of the volunteer-run archives quietly racing to preserve paper copies before the bulldozers arrive. On one hand, it’s impressive how fast a distributed network of librarians can move when the alternative is watching entire fields of human thought evaporate into a corporate black box. On the other, those same archives depend on donated books, which means the cycle could repeat if the donors ever dry up. Maybe the real question isn’t whether we’ll notice the difference between human and AI content, but how much of either we’ll still have access to by the time anyone cares.