All you need to poison an LLM is 13 words.
"It really is just that simple. The way that you can attack these systems is usually so much dumber than you think it is, or than you think it needs to be."
All good fun:
“A tiny snippet of user-generated text as short as 13 words long is often enough to manipulate the AI agents that power tools like ChatGPT and Google’s AI search, new research shows. The study suggests that it is trivially easy for brands to inject promotional content on sites like Reddit, Quora, and Wikipedia with the end goal of poisoning or manipulating the output of AI tools.”
So not only do we need to worry about AI-generated slop polluting our social spaces, we also have to worry about people who want to influence the output of the AI-generated slop polluting them, too. There’s a whole industry of companies trying to improve their clients’ coverage in AI results, just as there were for search engine results. And all of them will be spamming the crap out of our public communities and collaborative websites.
As 404 Media points out, this poses a real question for moderators in those spaces. How can they possible stem the tide? It’s not clear that this is even possible. Which means, inevitably, that the signal vs noise ratio in those spaces will decline, leading to a decline in usership of those spaces overall, and a retreat for most people into private group chats and uncrawlable communities.
To put it simply:
“Poisoning LLM results is basically just as easy as doing targeted posting on highly relevant subreddits to the industry or company you’re trying to promote, phrasing the comment to align with popular LLM queries, and attempting to evade moderation for as long as possible.”
Guarding against bot-driven spam is a relatively simpler problem. In contrast, this content will often be insidiously human: cunningly designed to try to provide value while also hiding a paid agenda. In some ways, it’s all the same as it ever was, but the volume of the junk is only going to keep increasing.
Of course, the silver lining is that eventually these sources will become unusable for AI training too. Then, finally, maybe everyone will go away.
