AI crawlers are scraping open resources from Canadian university libraries, placing servers and bandwidth under strain. The situation exposes the limits of robots.txt and raises practical questions about machine-readable permissions, API quotas, and usage-based pricing.
AI Crawlers Are Overloading University Libraries
Canadian university libraries are dealing with a group of visitors that do not behave like ordinary users.
CBC reports that AI bots are collecting data from public library services at scale, causing some systems to buckle under the load. The report reached the front page of Hacker News, where it received 47 points and 21 comments.
The numbers on Hacker News were modest, but the debate they generated was unmistakable. The concern extended beyond a single copyright case. Developers and researchers saw a broader issue: open knowledge supplied by public institutions was being repeatedly consumed by automated traffic.
Original report: https://www.cbc.ca/news/world/university-libraries-ai-data-scraping-bots-9.7408287
This is not simply a story about “AI copying books.”
It is more like an infrastructure bill. The organizations collecting training data avoid authorization and settlement costs, while libraries bear the expense of servers, bandwidth, authentication systems, and staff time.
Libraries Became Training-Data Targets
CBC's reporting covers open access, acceptable-use policies, authentication, service capacity, and the boundaries of copyright. The issue is not merely a legal dispute.
That distinction matters.
Reducing the debate to whether models may train on publicly available content can obscure the engineering change already underway.
University libraries are not dealing with an ordinary reader. A person may visit dozens of pages in a day. An AI crawler can open many concurrent connections and repeatedly query search endpoints, OCR pages, scanned files, and paginated results. Its objective is not to read but to copy as completely as possible.
The practical questions for libraries therefore include:
- Which materials are approved for bulk machine access?
- Does authentication still permit crawling?
- Do OCR and full-text indexing endpoints have rate limits?
- Could sudden traffic spikes displace genuine users?
- If a crawler uses content to train a model, who pays for additional server capacity?
With only 47 points and 21 comments on Hacker News, the thread nevertheless made one conflict visible to developers: should public resources on the internet be treated as freely available for unlimited machine consumption?
The answer cannot depend solely on whether a page is public.
Public access means that users can reach the material. It does not imply unlimited copying, nor does it mean the platform is prepared to support the cost of collecting it at scale.
Crawlers Consume Capacity, Not Just Books
The difference between AI crawling and ordinary browser traffic goes beyond the User-Agent header.
A browser generally sends requests at a human reading pace. An AI crawler treats page links, parameters, and hidden endpoints as a graph to traverse. It does not stop for coffee on page three, and it does not leave when it encounters a CAPTCHA.
Roughly speaking:
Ordinary access: read a page → pause briefly → read another page
AI crawler: enumerate entry points → crawl concurrently → identify structures → switch formats and continue
Costs accumulate at several layers.
The first is request density.
A single scholarly article may appear through several mirrors: the catalog record, PDF, OCR, JATS XML, thumbnail, and full-text index. A collection system does not need to download it once. It can follow each entry point to obtain several versions of the data at different levels of structure.
The second is authentication.
Some resources are openly browsable, while bulk downloads require a login, campus VPN, or specific institutional credentials. Users receive access to read and conduct research, not to create a copy of the entire collection.
The third is computation.
OCR, text cleaning, summarization, and chunking all require servers to process more data. For libraries, storage and bandwidth budgets feel the pressure before copyright lawyers do.
“They took the books” therefore describes only the surface. What is actually consumed is server capacity, bandwidth for indexing services, and the fault-tolerance margin maintained by a public institution.
Worse, crawler traffic can make system failures look like occasional instability. Users see slow loading times without knowing that another organization is copying the collection at scale in the background.
robots.txt Is Neither a License nor a Firewall
Libraries often publish acceptable-use policies and use robots.txt to tell crawlers which paths they should avoid.
But robots.txt is frequently misunderstood.
It is not a copyright license. It expresses a site's preferences for automated crawlers. Nor is it a firewall. Compliant crawlers read the rules and restrain themselves; noncompliant crawlers can simply ignore them.
More importantly, the rules cannot answer every question.
For example, a site might specify:
User-agent: *
Disallow: /api/
Disallow: /bulk-download/
This protects the specified paths, but it cannot restrict user agents, conceal the crawler's true origin, or stop the same organization from rewriting request headers or switching domains.
Conversely, if a library blocks all research access, it creates a different cost. Legitimate users are treated as bots, while CAPTCHAs, logins, and manual verification slow down teaching and research.
Technical rules and legal obligations need to be assessed separately.
- Bypassing acceptable-use policies may violate website terms or applicable computer-abuse laws.
- Using copyrighted material to train a model may implicate the scope of licenses, reproduction rights, fair use, and contractual restrictions.
- Ignoring
robots.txtdoes not automatically make an activity unlawful. - Public availability does not grant a model company a secondary right to use the content.
None of these questions should be collapsed into the claim that “public data is free to scrape.”
Open Knowledge Should Not Run on an Unpaid Public Subsidy
The more defensible position is that model companies should not presume they have a default secondary right to use training data.
Model training does require data, and large-scale models objectively need many public materials. But the fact that training needs data does not automatically shift the cost of collection onto content producers or public institutions.
The current exchange is often unbalanced.
Universities build collections, fund digitization, OCR, and cataloging, then provide access at little or no cost. AI companies use that infrastructure to build or improve models and earn revenue through products, subscriptions, or APIs.
If this use grows quickly enough, public institutions encounter capacity problems before they see any corresponding profits.
Libraries may need more servers, rate limits, blocks, storage migration, and staff time to handle complaints. No one knows how large the budget increase will be. The commercial value created by model training may remain with the party doing the collecting.
When scraping continues to occupy public infrastructure, the data-collection cost should return to the collector.
This does not mean charging for everything or closing open scholarship. A more granular approach is possible:
- Create a separate access category for model training.
- Charge high-load endpoints for computation and bandwidth.
- Price according to commercial use, request volume, and caching practices.
- Provide auditable bulk-data APIs instead of relying on covert scraping.
- Keep low-friction access available for small-scale, noncommercial research.
The point is not that “every bot must pay.”
The point is that machine use at scale should not masquerade as ordinary reading.
Traceable Permissions Are Harder to Block
Blocking IP addresses directly is not enough.
It is prone to blocking legitimate users and can trigger an arms race. Collectors switch proxies, libraries change rules, and both sides maintain increasingly complex identification systems.
A more workable direction is to make permissions machine-readable and costs measurable.
For example, a website could return explicit licensing and usage conditions:
{
"content": "/documents/example",
"license": "CC-BY-4.0",
"machine_use": "training-with-attribution",
"attribution_required": true,
"rate_limit_per_minute": 10,
"commercial_use": "license-required"
}
This is a conceptual format, not an established standard. It illustrates a basic principle: machine access should have clear rules for permissions, rate limits, caching, and billing, much like API documentation, rather than leaving crawlers to guess.
At least four approaches could be implemented:
- Machine-readable permissions
Distinguish reading, searching, indexing, model training, and redistribution. For Creative Commons licenses, clarify whether text mining and model training fall within the existing authorization. - API quotas
Set different rate and concurrency limits for public search and full-text downloads. Genuine users and legitimate research tasks should not be slowed by commercial crawlers. - Content provenance
Retain source, license, collection time, version, and hash information. In a dispute, this can establish which version supplied a passage and whether the collector violated its license conditions. - Usage-based pricing
Charge according to requests, data transferred, pages scanned, or processing tasks. Revenue can support digitization, storage, and bandwidth maintenance.
These approaches have not yet produced a reliable, unified consensus.
How Creative Commons licenses apply to specific model-training practices remains disputed across jurisdictions. It is also difficult to prove that a scraping activity directly contributed to a particular model-training run. Libraries have not settled on a standard technical contract, and corporate buyers have not demonstrated a consistent willingness to pay.
Technical measures can move ahead. Legal and commercial allocation will take much longer.
Editorial Perspective
This issue matters because AI companies are scraping library data, but also because it exposes an often-unspoken cost of generative AI expansion: open knowledge is not free infrastructure. Universities preserve, organize, and provide knowledge for society, and AI collection sends the pressure of large-scale copying back to the same servers. Copyright disputes may take years to resolve, while capacity pressure begins as soon as the story is published.
Model companies are unlikely to accept a universal “pay for every training run” model. The more likely outcome is tiered access: commercially valuable bulk data moves into licensed or metered channels, while lower-value traffic remains subject to rate limiting, identification, and blocking. Websites historically monetized traffic through advertising or subscriptions. AI crawlers have introduced a fourth form of traffic: they do not click, buy, or read; they copy content to train systems. The difference now is that machine access is beginning to be asked to cover costs proportional to its scale. Whether particular scraping activity constitutes infringement, and which open licenses are sufficient to support training, will remain unresolved.