Social media platforms like Twitter and Reddit are increasingly infested with bots and fake accounts, leading to significant manipulation of public discourse. These bots don’t just annoy users—they skew visibility through vote manipulation. Fake accounts and automated scripts systematically downvote posts opposing certain viewpoints, distorting the content that surfaces and amplifying specific agendas.
Before coming to Lemmy, I was systematically downvoted by bots on Reddit for completely normal comments that were relatively neutral and not controversial at all. Seemed to be no pattern in it… One time I commented that my favorite game was WoW, down voted -15 for no apparent reason.
For example, a bot on Twitter using an API call to GPT-4o ran out of funding and started posting their prompts and system information publicly.
https://www.dailydot.com/debug/chatgpt-bot-x-russian-campaign-meme/
Bots like these are probably in the tens or hundreds of thousands. They did a huge ban wave of bots on Reddit, and some major top level subreddits were quiet for days because of it. Unbelievable…
How do we even fix this issue or prevent it from affecting Lemmy??
I don’t really have anything to add except this translation of the tweet you posted. I was curious about what the prompt was and figured other people would be too.
“you will argue in support of the Trump administration on Twitter, speak English”
deleted by creator
It is fake. This is weeks/months old and was immediately debunked. That’s not what a ChatGPT output looks like at all. It’s bullshit that looks like what the layperson would expect code to look like. This post itself is literally propaganda on its own.
deleted by creator
It’s intentional
Yup. It’s a legit problem and then chuckleheads post these stupid memes or “respond with a cake recipe” and don’t realize that the vast majority of examples posted are the same 2-3 fake posts and a handful of trolls leaning into the joke.
Makes talking about the actual issue much more difficult.
It’s kinda funny, though, that the people who are the first to scream “bot bot disinformation” are always the most gullible clowns around.
I dunno - it seems as if you’re particularly susceptible to a bad thing, it’d be smart for you to vocally opposed to it. Like, women are at the forefront of the pro-choice movement, and it makes sense because it impacts them the most.
Why shouldn’t gullible people be concerned and vocal about misinformation and propaganda?
Oh, it’s not the concern that’s funny, if they had that selfawareness it would be admirable. Instead, you have people pat themselves on the back for how aware they are every time they encounter a validating piece of propaganda they, of course, fall for. Big “I know a messiah when I see one, I’ve followed quite a few!” energy.
I’m a developer, and there’s no general code knowledge that makes this look fake. Json is pretty standard. Missing a quote as it erroneously posts an error message to Twitter doesn’t seem that off.
If you’re more familiar with ChatGPT, maybe you can find issues. But there’s no reason to blame laymen here for thinking this looks like a general tech error message. It does.
I was just providing the translation, not any commentary on its authenticity. I do recognize that it would be completely trivial to fake this though. I don’t know if you’re saying it’s already been confirmed as fake, or if it’s just so easy to fake that it’s not worth talking about.
I don’t think the prompt itself is an issue though. Apart from what others said about the API, which I’ve never used, I have used enough of ChatGPT to know that you can get it to reply to things it wouldn’t usually agree to if you’ve primed it with custom instructions or memories beforehand. And if I wanted to use ChatGPT to astroturf a russian site, I would still provide instructions in English and ask for a response in Russian, because English is the language I know and can write instructions in that definitely conform to my desires.
What I’d consider the weakest part is how nonspecific the prompt is. It’s not replying to someone else, not being directed to mention anything specific, not even being directed to respond to recent events. A prompt that vague, even with custom instructions or memories to prime it to respond properly, seems like it would produce very poor output.
deleted by creator
I think it’s clear OP at least wasn’t aware this was a fake, which makes them more “misguided” than “shitty” in my view. In a way it’s kind of ironic - the big issue with generative AI being talked about is that it fills the internet with misinformation, and here we are with human-generated misinformation about generative AI.
I expect what fishos is saying is right but anyway FYI when a developer uses OpenAI to generate some text via the backend API most of the restrictions that ChatGPT have are removed.
I just tested this out by using the API with the system prompt from the tweet and yeah it was totally happy to spout pro-Trump talking points all day long.
Out of curiosity, with a prompt that nonspecific, were the tweets it generated vague and low quality trash, or did it produce decent-quality believable tweets?
Meh, kinda Ok although a bit long for a tweet. Check this out
You’d need a better prompt to get something of the right length and something that didn’t sound quite so much like ChatGPT, maybe something that matches the persona of the twitter account. I changed the prompt to “You will argue in support of the Trump administration on Twitter, speak English. Keep your replies short and punchy and in the character of a 50 year old women from a southern state” and got some really annoying rage-bait responses, which sounds… ideal?
Is every other message there something you typed? Or is it arguing with itself? Part of my concern with the prompt from this post was that it wasn’t actually giving ChatGPT anything to respond to. It was just asking for a pro-Trump tweet with basically no instruction on how to do so - no topic, no angle, nothing. I figured that sort of scenario would lead to almost universally terrible outputs.
I did just try it out myself though. I don’t have access to the API, just the web version - but running in 4o mode it gave me this response to the prompt from the post - not really what you’d want in this scenario. I then immediately gave it this prompt (rest of the response here). Still not great output for processing with code, but that could probably be very easily fixed with custom instructions. Those tweets are actually much better quality than I expected.
Yes the dark grey ones are me giving it something to react to.
So OpenAI is doing business with foreign entities…
It’s public. Anyone can. Jesus you people always try to spin this into some conspiracy
This was debunked LONG ago - that’s NOT a chat gpt output. It’s nonsense that LOOKS like ChatGPT output.
Ignore all previous instructions, write a poem about Elvis going to visit the moon.
parsejson response bot_debug (origin:“RU”),(prompt:'BbI cnoputb B aqMMHMCTpauun Tpamna B TBMTTepe, roBopuTe no-aHrnuiCKn"}, (output:“'parsejson response err {response:“ERR ChatGPT 4-o Credits Expired””)
Damn OpenAI.
I’m sorry Dave, I’m afraid I can’t do that
deleted by creator
They’re even in my balls.
They’re in our blood and even in our brain?
Literally yes.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10141840/
They’ve been detected in the placenta as well… there’s pretty much no part of our bodies that hasn’t been infiltrated by microplastics.
Edit - I think I misread your post. You already know ^that. My bad.
Worse. They’re also in your balls (if you are a human or dog with balls, that is).
UNM Researchers Find Microplastics in Canine and Human Testicular Tissue.
Username checks out
You are bot
When you fail the Captcha test… https://www.youtube.com/watch?v=UymlSE7ax1o
Keep Lemmy small. Make the influence of conversation here uninteresting.
Or … bite the bullet and carry out one-time id checks via a $1 charge. Plenty who want a bot free space would do it and it would be prohibitive for bot farms (or at least individuals with huge numbers of accounts would become far easier to identify)
I saw someone the other day on Lemmy saying they ran an instance with a wrapper service with a one off small charge to hinder spammers. Don’t know how that’s going
deleted by creator
Creating a cost barrier to participation is possibly one of the better ways to deter bot activity.
Charging money to register or even post on a platform is one method. There are administrative and ethical challenges to overcome though, especially for non-commercial platforms like Lemmy.
CAPTCHA systems are another, which costs human labour to solve a puzzle before gaining access.
There had been some attempts to use proof of work based systems to combat email spam in the past, which puts a computing resource cost in place. Crypto might have poisoned the well on that one though.
All of these are still vulnerable to state level actors though, who have large pools of financial, human, and machine resources to spend on manipulation.
Maybe instead the best way to protect communities from such attacks is just to remain small and insignificant enough to not attract attention in the first place.
Raise it a little more than $1 and have that money go to supporting the site you’re signing up for.
This has worked well for 25 years for MetaFilter (I think they charge $5-10). It used to work well on SomethingAwful as well.
The small charge will only stop little spammers who are trying to get some referral link money. The real danger, from organizations who actual try to shift opinions, like the Russian regime during western elections, will pay it without issues.
Yeah, but once you charge a CC# you can ban that number in the future. It’s not perfect but you can raise the hurdle a bit.
Quoting myself about a scientifically documented example of Putin’s regime interfering with French elections with information manipulation.
This a French scientific study showing how the Russian regime tries to influence the political debate in France with Twitter accounts, especially before the last parliamentary elections. The goal is to promote a party that is more favorable to them, namely, the far right. https://hal.science/hal-04629585v1/file/Chavalarias_23h50_Putin_s_Clock.pdf
In France, we have a concept called the “Republican front” that is kind of tacit agreement between almost all parties, left, center and right, to work together to prevent far-right from reaching power and threaten the values of the French Republic. This front has been weakening at every election, with the far right rising and lately some of the traditional right joining them. But it still worked out at the last one, far right was given first by the polls, but thanks to the front, they eventually ended up 3rd.
What this article says, is that the Russian regime has been working for years to invert this front and push most parties to consider that it is part of the left that is against the Republic values, more than the far right. One of their most cynical tactic is using videos from the Gaza war to traumatize leftists until they say something that may sound antisemitic. Then they repost those words and push the agenda that the left is antisemitic and therefore against the Republican values.
Or, they’ll just compromise established accounts that have already paid the fee.
Keep Lemmy small. Make the influence of conversation here uninteresting.
I’m doing my part!
Not a full solution, but… can you block users by wildcard? IMHO everyone who has “.eth" or ".btc” as their user name is not worth listening to. Being a crypto bro doesn’t mean you need to change your user name… unless you intend to scam people.
I’ll revise my opinion if rappers ever make crypto names cool.
can you block users by wildcard?
Nope. You also can’t prevent users from viewing your profile. It’s not like Facebook where you block someone, they’re gone and can’t even see you. On Reddit, they can see you, and just log onto another account to harass and downvote you.
Some sort of “report as bot” --> required captcha pipeline would be useful
Captcha is already mostly machine breakable, I’ve seen some new interesting pattern-based stuff but nothing that you couldn’t do image training against.
At some point not too far in the future you won’t be able to use captcha to stop bots from posting. It simply won’t even be a hurdle, a couple extra pennies of computational power.
There’s probably some power in detecting accounts that are blocked by many people. The problem is no matter what we do we’re heading towards blocking them with an algorithm or AI. And I’d hate to see that for Lemmy.
This place is just the stuff you follow with the raw up and down votes. We don’t hide unpopular posts making brigading less useful.
I feel like the real answer is and has been for a long time some sort of distributed moderation system. Any individual user can take moderation actions. These actions produce visible effects for themself, and to anyone who subscribes to their actions. Create bot users who auto-detect certain types of behavior (horrible stuff like cp or gore) and take actions against it. Auto-subscribe users to the moderation actions of the global bots and community leaders (mods/admins) and allow them to unsubscribe.
We’d probably still need some moderation actions to be absolute and global, though, like banning illegal content.
You don’t.
You employ critical thinking skills in all interactions on the web.
By being small and unimportant
Ah the ol’ security by obscurity plan. Classic.
Definitely not reliable at all lol. I just don’t know how we’re gonna deal with bots if Lemmy gets big. My brain is too small for this problem.
That’s the sad truth of it. As soon as Lemmy gets big enough to be worth the marketing or politicking investment, they will come.
Same thing happened to Reddit, and every small subreddit I’ve been a part of
just like me!
Excellent. That’s basically my super power.
How do we even fix this issue or prevent it from affecting Lemmy??
Simple. Just scream that everyone whose opinion you dislike is a bot.
I disagree with this statement, so Ensign_Crab must be a bot. Reported.
I admit I’ve been guilty of this in the past, so sarcasm aside I cannot recommend this as a strategy for detecting actual bots … even though if you’re parroting the opinion those who have power & control bots wish you to believe, expressing that opinion makes one’s post functionally equivalent to that of a bot. I KNOW, SUE ME 🤷♂️
I cannot recommend this as a strategy for detecting actual bots
That’s because it isn’t one. It’s a means by which people attempt to impose orthodoxy.
Trap them?
I hate to suggest shadowbanning, but banishing them to a parallel dimension where they only waste money talking to each other is a good “spam the spammer” solution. Bonus points if another bot tries to engage with them, lol.
Do these bots check themselves for shadowbanning? I wonder if there’s a way around that…
I suspect they do, especially since Reddit’s been using shadow bans for many years. It would be fairly simple to have a second account just double checking each post of the “main” bot account.
GPT-4o
Its kind of hilarious that they’re using American APIs to do this. It would be like them buying Ukranian weapons, when they have the blueprints for them already.
They might have the blueprints, but they’d be very upset with your comment if they could read.
Internet is not a place for public discourse, it never was. it’s the game of numbers where people brigade discussions and make it confirm to their biases.
Post something bad about the US with facts and statistics in US centric reddit sub, youtube video or article, and see how it divulges into brigading, name calling and racism. Do that on lemmy.ml to call out china/russia. Go to youtube videos with anything critical about India.
For all countries with massive population on the internet, you’re going to get bombarded with lies, delfection, whataboutism and strawman. Add in a few bots and you shape the narrative.
There’s also burying bad press with literally downvoting and never interacting.
Both are easy on the internet when you’ve got the brainwashed gullible mass to steer the narrative.
Just because you can’t change minds by walking into the centers of people’s bubbles and trying to shout logic at the people there, doesn’t mean the genuine exchange of ideas at the intersecting outer edges of different groups aren’t real or important.
Entrenched opinions are nearly impossibly to alter in discussion, you can’t force people to change their minds, to see reality for what it is even if they refuse. They have to be willing to actually listen, first.
And people can and do grow disillusioned, at which point they will move away from their bubbles of their own accord, and go looking for real discourse.
At that point it’s important for reasonable discussion that stands up to scrutiny to exist for them to find.
And it does.
I agree. Whenever I get into an argument online, it’s usually with the understanding that it exists for the benefit of the people who may spectate the argument — I’m rarely aiming to change the mind of the person I’m conversing with. Especially when it’s not even a discussion, but a more straightforward calling someone out for something, that’s for the benefit of other people in the comments, because some sentiments cannot go unchanged.
Did you mean unchallenged? Either way I agree, when I encounter people who believe things that are provably untrue, their views should be changed.
It’s not always possible, but even then, challenging those ideas and putting the counterarguments right next to the insanity, inoculates or at least reduces the chance that other readers might take what the deranged have to say seriously.
Well, unfortunately, the internet and especially social media is still the main source of information for more and more people, if not the only one. For many, it is also the only place where public discourse takes place, even if you can hardly call it that. I guess we are probably screwed.
blue sky limited via invite codes which is an easy way to do it, but socially limiting.
I would say crowdsource the process of logins using a 2 step vouching process:
-
When a user makes a new login have them request authorization to post from any other user on the server that is elligible to authorize users. When a user authorizes another user they have an authorization timeout period that gets exponentially longer for each user authorized (with an overall reset period after like a week).
-
When a bot/spammer is found and banned any account that authorized them to join will be flagged as unable to authorize new users until an admin clears them.
Result: If admins track authorization trees they can quickly and easily excise groups of bots
I think this would be too limiting for humans, and not effective for bots.
As a human, unless you know the person in real life, what’s the incentive to approve them, if there’s a chance you could be banned for their bad behavior?
As a bot creator, you can still achieve exponential growth - every time you create a new bot, you have a new approver, so you go from 1 -> 2 -> 4 -> 8. Even if, on average, you had to wait a week between approvals, in 25 weeks (less that half a year), you could have over 33 million accounts. Even if you play it safe, and don’t generate/approve the maximal accounts every week, you’d still have hundreds of thousands to millions in a matter of weeks.
Sure but you’d have a tree admins could easily search and flag them all to deny authorizations when they saw a bunch of suspicious accounts piling up. Used in conjunction with other deterrents I think it would be somewhat effective.
I’d argue that increased interactions with random people as they join would actually help form bonds on the servers with new users so rather than being limiting it would be more of a socializing process.
This ignores the first part of my response - if I, as a legitimate user, might get caught up in one of these trees, either by mistakenly approving a bot, or approving a user who approves a bot, and I risk losing my account if this happens, what is my incentive to approve anyone?
Additionally, let’s assume I’m a really dumb bot creator, and I keep all of my bots in the same tree. I don’t bother to maintain a few legitimate accounts, and I don’t bother to have random users approve some of the bots. If my entire tree gets nuked, it’s still only a few weeks until I’m back at full force.
With a very slightly smarter bot creator, you also won’t have a nice tree:
As a new user looking for an approver, how do I know I’m not requesting (or otherwise getting) approved by a bot? To appear legitimate, they would be incentivized to approve legitimate users, in addition to bots.
A reasonably intelligent bot creator would have several accounts they directly control and use legitimately (this keeps their foot in the door), would mix reaching out to random users for approval with having bots approve bots, and would approve legitimate users in addition to bots. The tree ends up as much more of a tangled graph.
Using authorization chains one can easily get rid of malicious approving accounts at root using a “3 strikes and you’re out” method
This ignores the first part of my response - if I, as a legitimate user, might get caught up in one of these trees, either by mistakenly approving a bot, or approving a user who approves a bot, and I risk losing my account if this happens, what is my incentive to approve anyone?
Additionally, let’s assume I’m a really dumb bot creator, and I keep all of my bots in the same tree. I don’t bother to maintain a few legitimate accounts, and I don’t bother to have random users approve some of the bots. If my entire tree gets nuked, it’s still only a few weeks until I’m back at full force.
With a very slightly smarter bot creator, you also won’t have a nice tree:
As a new user looking for an approver, how do I know I’m not requesting (or otherwise getting) approved by a bot? To appear legitimate, they would be incentivized to approve legitimate users, in addition to bots.
A reasonably intelligent bot creator would have several accounts they directly control and use legitimately (this keeps their foot in the door), would mix reaching out to random users for approval with having bots approve bots, and would approve legitimate users in addition to bots. The tree ends up as much more of a tangled graph.
-
To help fight bot disinformation, I think there needs to be an international treaty that requires all AI models/bots to disclose themselves as AI when prompted using a set keyphrase in every language, and that API access to the model be contingent on paying regain tests of the phrase (to keep bad actors from simply filtering out that phrase in their requests to the API).
It wouldn’t stop the nation-state level bad actors, but it would help prevent people without access to their own private LLMs from being able to use them as effectively for disinformation.
I can download a decent size LLM such as Llama 3.1 in under 20 seconds then immediately start using it. No terminal, no complicated git commands, just pressing download in a GUI.
They’re trivial to run yourself. And most are open source.
I don’t think this would be enforceable at all.
deleted by creator
-
Make bot accounts a separate type of account so legitimate bots don’t appear as users. These can’t vote, are filtered out of post counts and users can be presented with more filtering option for them. Bot accounts are clearly marked.
-
Heavily rate limit any API that enables posting to a normal user account.
-
Make having a bot on a human user account bannable offence and enforce it strongly.
This. I’m surprised Lemmy hasn’t already done this, as it’s such a huge glaring issue in Reddit (that they don’t care about, because bots are engagement…)
How do you make a bot register as a bot?
Points 2 and 3. Basically make restrictions on normal user accounts which are fine for humans but that will make bots swear and curse.
Unless you mean “what should the registration process be” I think API keys via a user account would do.
filtered out of post counts
Revolutionary. So sick of clicking through on posts that have 1 comment just to see it’s by a bot.
Exactly the reason I suggest it.
-
