DuckDuckGo, Bing, Mojeek, and other search engines are not returning full Reddit results any more.
Weird, most of the results I get from Google’s search are from Quora (and they fucking suck). Google as a search engine has been going downhill for a while now. Reddit has becomes an increasingly spammy shithole full of corporate and political astroturfing too.
Meh, fuck em. The tighter they make their circle the less useful it is.
Reminder that Kagi searches Lemmy which is great.
Lemmy is like 0,0000001% of reddit lol
Kagi
Ah, yes. The “Fuck you, Pay me” search engine.
It’s a paid service that in return respects privacy, and has results as good as Google’s.
Google doesn’t say “Fuck you, Pay me” because you are the product.
As good as google is now or as google was before it went mental?
I’d say in a good bit of cases better than Google now. Each time I’ve resorted to Google because I think I’m getting lackluster results, it’s got the same results but also a bunch of SEO dogshit I have to sort through. Kagi’s personal site ranking and filters make it worth it too.
Definitely old Google. Searx etc probably already better than current Google. Kagi has free trial though. I used free trial like 13 times before I realized it was time to pay the entry level 300 search fee and have no regrets about the value.
A service that gasp costs money???
It surfaces better results for me than others so I’m paying for a service that I gain benefit from. Feel free not to…
I like where I stand in that model. I’ve tried the alternatives and have found genuine value in what I’m paying for. Lemmy loves turning its nose up at Kagi because “money” but they’ve gone out of their way to index the fediverse where most have ignored us. I’m an anonymous Kagi user I pay through a third party using a crypto account and I couldn’t be happier - they now have a lower tiered payment plan that is affordable for me.
Ha ha, Reddit went to the dark side and I abandoned them months ago. So who cares what they do now.
You obviously writing here, haha retard putin fan
Kinda bad for the rare useful stuff in there
The number of obscure guides and download collections… I do pray someone is archiving it properly.
I’m not understanding what stops a search engine from scraping a publicly accessible website. ?
robots.txt, I guess? Yes, you can just ignore it, but you shouldn’t, if you develop a responsible web scraper.
Doesn’t seem legal that a robots.txt could pick and choose who scrapes. Seems like legally it would have to be all or nothing. Here’s hoping one of the search engines ignores it and makes it a legal case.
You’d probably feel differently if it were your service. Should you be able to control who scrapes your sites or should that be all or nothing?
For the record, I fucking hate what the internet is becoming. I naively believed that even if shit got cordoned off into the walled gardens that are mobile phone apps, the web would remain as open as it was. This is a terrible sign of things to come.
No, I wouldn’t feel differently. In fact letting search engines scrape and point to your content is what leads people to your site. It’s free advertising. If you’re going to let one search engine in, you should let them all in. If you want to be public, be public. Otherwise put up a login firewall and go private.
It’s not just search engines. Lots of people on Mastodon were using robots.txt to block ChatGPT (and any other LLM company they knew of) from scraping their sites/blogs.
I disagree, to a point. I want to be able to control my services to the greatest extent possible, including picking who scrapes me.
On the other hand, orgs as large as Google doing this poses a real threat to how the internet works right now which I hate.
Actually currently it contains this:
User-agent: * Disallow: /Well, that actually is a blanket ban for everyone, so something else must be at play here.
https://merj.com/blog/investigating-reddits-robots-txt-cloaking-strategy
Reddit is serving different file to google
We believe in the open internet, but we do not believe in the misuse of public content.
That’s real rich, coming from Reddit.
Also, rate limiting. A publicly accessible website doesn’t mean that it will allow scrapers to read millions of pages each week. They can easily identify and block scrapers because of the pattern of their activity. I don’t know if Reddit has rate-limiting, but I wouldn’t be surprised if they implement one.
paywalled link :(
I got you https://archive.ph/GS2I0
We are indebted to Archive.ph which allows us to jump over paywalls.
yar har
Great, neither Google search or reddit work anymore. They deserve each other.
I literally only use Reddit for one particular game sub and nothing else, so I just go directly there. I literally have no other use for it, so this change means exactly diddly dick to me.
Another nail.
Poople
Still couldn’t get me to use it, I use DDG which can switch between search engines and search sites very quickly with it’s ! syntax (Everyone goes on about privacy, but this is pretty much it’s best feature). Google results are consistently the worst for me if I’m hitting multiple search engines
Oh cool! My searching won’t be spammed with Reddit now!
DuckDuckGo hasn’t been returning good reddit results for months… sigh. (Sometimes reddit has the only discourse on e.g. some error a SaaS product is throwing.)
Personally, I really wish it was as easy to search for Lemmy posts with a search engine as it is with Reddit. Idk, maybe I’m doing it wrong.
Kagi does it
I use kagi; love it. As an embedded systems developer I’m more productive with it.
Reddit really fucked themselves. Not as much as Elon fucked twitter but super close.
Also pretty sure DDG uses Bing
