> ## Content Index
> Fetch the complete content index at: https://werd.io/llms.txt
> Use this file to discover other available public pages before exploring further.

# The text file that runs the internet
- URL: https://werd.io/the-text-file-that-runs-the-internet/
- Published: 2024-02-15T19:13:29.000Z
- Updated: 2024-02-15T19:13:29.000Z
- Author: Ben Werdmuller
- Tags: NotableLinks, #Import 2025-06-13 09:20

It's hard to read this without feeling like the social contract of the web is falling apart.

And when social agreements fall apart, that's when we start having to talk about more rigid, enforced contracts instead. As the piece notes:

"There are people on both sides who believe we need better, stronger, more rigid tools for managing crawlers. They argue that there’s too much money at stake, and too many new and unregulated use cases, to rely on everyone just agreeing to do the right thing."

I think it's inevitable that we'll see more regulation and a more locked-down web. Probably, past a certain point, this was always going to happen. But I'll miss the days of rough consensus and working code. [#Technology](https://werd.io/tag/Technology)

\[[Link](https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders)\]