DuckDuckGo, Bing, Mojeek, and other search engines are not returning full Reddit results any more.

  • capital@lemmy.world
    link
    fedilink
    English
    arrow-up
    0
    ·
    4 months ago

    It’s not just search engines. Lots of people on Mastodon were using robots.txt to block ChatGPT (and any other LLM company they knew of) from scraping their sites/blogs.

    I disagree, to a point. I want to be able to control my services to the greatest extent possible, including picking who scrapes me.

    On the other hand, orgs as large as Google doing this poses a real threat to how the internet works right now which I hate.