Of the signals used to indicate crawling is prohibited, robots.txt is probably the most effective; OpenAI, Google, Anthropic, Meta, and CommonCrawl all claim to respect it. That often provokes a response of "well they're lying", but I've yet to actually find any cases of the IPs they use for crawling accessing content prohibited by robots.txt.
Newly proposed standards will probably take a while to catch on, if they ever do.
Not a lawyer, but I believe such measures could in theory become legally enforceable in the US without any new legislation if the fair use defense fails but an implied license defense (the reason you can cache/rehost copies of webpages that don't have a <noarchive> meta tag, as in Field v. Google Inc) succeeds.
I do not want others to scrape my files from my server for the purpose of training LLMs, but if they acquire a copy of them by other means or already have a copy of them for other reasons, then they will already have a copy and then they can do what they want with it.
I do not care about attribution; but I care more that they do not claim additional restrictions in their terms of use when they copy my stuff and use it.
I am not sure how this is any different from open source code being embedded in commercial applications. It’s really like a self-accelerating loop.
At least for OSS, usage defines value. When an OSS project is popular, enterprises notices it and begins to use it in their commercial applications.
No
They already started with the assumption of consent, crawled the web with disregard for resource use, and still provide no mechanism to revoke permission. This is the culture around AI. A quiet little tag that says "please don't do that" won't do much.
These companies are already behaving like jerks. Do you think they will become more polite once they control how we avcess information? with investors breathing down their neck?