> ## Content Index
> Fetch the complete content index at: https://werd.io/llms.txt
> Use this file to discover other available public pages before exploring further.

# Licensing site content using a text file
- URL: https://werd.io/licensing-site-content-using-a-text-file/
- Published: 2023-07-03T17:23:18.000Z
- Updated: 2023-07-03T17:23:18.000Z
- Author: Ben Werdmuller
- Tags: #Import 2025-06-13 09:21

[Dare Obasanjo, over on Mastodon](https://mas.to/@carnage4life/110651239569888359):

> Robots.txt needs an update for the 2020s. Instead of just saying what content can be indexed, it should also grant rights.  
>  
> Like crawl my site only to provide search results not train your LLM.

Call it license.txt.

[The robots.txt standard](https://www.robotstxt.org/robotstxt.html) allows a website owner to specify which crawlers (by user agent) can access which parts of a website. It’s been useful, as far as it goes: for example, a website owner can theoretically tell Google or Bing to not index certain pages. But there’s no way of restricting *use* in the way Dare suggests.

Imagine extending the standard so that further use of the content can be specified in the same way as user agent. Content uses might include:

- archive (eg, archival by a service like the Internet Archive)
- search (eg, use in a search engine like Google)
- model (eg, scraping by LLM providers like OpenAI)
- republish (eg, Creative Commons or open source)

Each one might have parameters. For example:

- republish:BY-NC-SA would specify the Creative Commons attributes “attribution, non-commercial, share-alike”
- republish:(license URL) would specify use of a license with a specific URI, including commercial licenses whose terms would be available at that URL.

In addition to the opt-out “disallow” wording employed by robots.txt, license.txt would default to opt-in and use an “allow” keyword.

A simple license.txt file might then look like:

```
Use: archive, searchAllow: /Disallow: /adminUse: modelDisallow: /Use: republish:BY-NCAllow: /freeUse: republish:https://werd.io/licenseAllow: /
```

This would allow all site content to be archived and indexed for search, aside from an /admin path. The site would not allow crawling for model use. Pages under the /free path would allow republishing for non-commercial purposes while the whole site would be licensable for commercial purposes using a license available at the specified URL.

One of the disadvantages of robots.txt is that it’s a “secret” URL that you have to know about to discover. To that end, the specification also allows for [meta tags in the header of each page](https://www.robotstxt.org/meta.html). Likewise, license tags in the header of an HTML page should also be supported, specifying uses on a given page:

```
<meta name="license" content="archive, search, republish:BY-NC">
```

These tag values would specify which rights *are* allowed under the license. For example, the tag above allows the page to be archived and indexed for search, as well as republished under BY-NC. Because “model” isn’t specified, it’s understood that licensing for use in LLM models is not granted.

[Also published on Indienews](https://news.indieweb.org/en).