hotpot8toe@lemmy.world to Technology@lemmy.worldEnglish · 4 months agoGoogle Is the Only Search Engine That Works on Reddit Now Thanks to AI Dealwww.404media.coexternal-linkmessage-square242fedilinkarrow-up11arrow-down10file-textcross-posted to: technology@lemmy.worldtechnology@lemmy.worldtechnology@beehaw.org
arrow-up11arrow-down1external-linkGoogle Is the Only Search Engine That Works on Reddit Now Thanks to AI Dealwww.404media.cohotpot8toe@lemmy.world to Technology@lemmy.worldEnglish · 4 months agomessage-square242fedilinkfile-textcross-posted to: technology@lemmy.worldtechnology@lemmy.worldtechnology@beehaw.org
minus-squareEril@feddit.orglinkfedilinkEnglisharrow-up0·4 months agoActually currently it contains this: User-agent: * Disallow: / Well, that actually is a blanket ban for everyone, so something else must be at play here.
minus-squarestarman@programming.devlinkfedilinkEnglisharrow-up0·4 months agohttps://merj.com/blog/investigating-reddits-robots-txt-cloaking-strategy Reddit is serving different file to google
minus-squareRuss@bitforged.spacelinkfedilinkEnglisharrow-up0·4 months ago We believe in the open internet, but we do not believe in the misuse of public content. That’s real rich, coming from Reddit.
Actually currently it contains this:
User-agent: * Disallow: /Well, that actually is a blanket ban for everyone, so something else must be at play here.
https://merj.com/blog/investigating-reddits-robots-txt-cloaking-strategy
Reddit is serving different file to google
That’s real rich, coming from Reddit.