AI companies are violating a basic social contract of the web and and ignoring robots.txt

Andy Reid@lemmy.world · 11 months ago

AI companies are violating a basic social contract of the web and and ignoring robots.txt

lily33@lemm.ee · 11 months ago

What social contract? When sites regularly have a robots.txt that says “only Google may crawl”, and are effectively helping enforce a monolopy, that’s not a social contract I’d ever agree to.

Imgonnatrythis@sh.itjust.works · 11 months ago

I had a one-eared rabbit. He was a monolopy.

Ashelyn@lemmy.blahaj.zone · 11 months ago

Sounds like a Pal name lol

assassin_aragorn@lemmy.world · 11 months ago

Only if its model is a Lopunny missing an ear

Ashelyn@lemmy.blahaj.zone · 11 months ago

I was thinking of a short lil bunny wearing a top hat and monocle with one ear sticking out of the center of the top hat but that works too

Cosmic Cleric@lemmy.world · 11 months ago

When sites regularly have a robots.txt that says “only Google may crawl”

Is that actually true?

If so, why would they do that?

AI companies are violating a basic social contract of the web and and ignoring robots.txt

AI companies are violating a basic social contract of the web and and ignoring robots.txt

The rise and fall of robots.txt