To protect the web, we need to renegotiate its value.
Public Knowledge and Creative Commons have laid out a compelling case for what must be done to protect the open web in the face of AI. It's going to take work.
This should be required reading for everyone in news, but also anyone who cares about a healthy knowledge commons:
Two of the most important organizations advocating for a healthy information economy, Public Knowledge and Creative Commons, have produced an important document that describes the threats and some potential solutions.
I particularly appreciated their definition of what they call defensive enclosure. In a world where their content is being scraped for someone else’s profit — and robots.txt files are being ignored at least 30% of the time, while almost nobody is using AI-specific exclusion standards — publishers are more likely to hide their content away. In turn, this has created its own ecosystem of tools, many of which have the potential to also be centralized and monopolistic.
As their article summarizes the problem:
“The blunt tools in use on either side of the AI developer/rights holder divide are serving to shrink the knowledge commons and erode its quality. This isn’t theoretical, AI-doomer prognosticating about some imagined AI future. The digital knowledge commons is contracting at an alarming rate, driven by the rise of AI model developers that have transformed openly shared content into a high-value industrial input and the defensive response of rights holders who are rapidly restricting access through legal, technical, and contractual means.”
This is not just a problem for us humans who would like to learn and build on each other’s work; AI model developers themselves rely on this information to train their future products. They, too, should be invested in solving the problem.
The challenge is that for newsrooms and other publishers, this enclosure is completely rational. The desire to not be intermediated from their communities is based on a need to serve them well and to achieve financial sustainability. To them, these blunt instruments are tools to ward off an existential threat. But, of course, that then has an adverse effect on the knowledge ecosystem as a whole: there’s an enclosure movement in progress, and the battle is really over who is building the fences.
So it’s interesting to read the angle these two organizations take here. They take the stance that training on content is fair use (which has been found in, for example, Bartz v. Anthropic). They also want to protect public-interest scraping, which would allow researchers and journalists (among others) to continue gathering information, and reject legislation that restricts bot activity without the public in mind. Instead, they favor a bigger swing: renegotiating the broad economic framework of the web itself to move the incentives away from enclosure.
They think solutions may involve some combination of public policy and innovation around data governance. The question is really: what does that mean in practice? How might we shift the web’s value exchange in order to promote open publishing? And — beyond the public good — whose interests are aligned with that?
Without a concrete answer, publishers will keep blocking, and AI vendors will keep strip-mining. That’s a path that will lead to a much more restricted information commons, to the detriment of all of us.
