• 0 Posts
  • 2 Comments
Joined 2 years ago
cake
Cake day: June 14th, 2023

help-circle
  • cecilkorik@lemmy.catoTechnology@lemmy.worldI was wrong about robots.txt
    link
    fedilink
    English
    arrow-up
    2
    arrow-down
    1
    ·
    1 day ago

    Absolutely true. They’ll buy the data they want from some shitty crawler running from some data broker in some far-flung and lawless part of the world, hallucinate the actual source, and pretend they had no idea their “data partner” wasn’t respecting robots.txt if they have to, which they won’t ever have to do because it’s literally impossible to detect and prove and realistically unenforceable.

    This is a company that removed it’s company motto of “Don’t be evil” because it found it too “limiting”. Don’t be naive.