The Future of Forums is Lies, I Guess
"I do not have the time or emotional energy to screen out regular attacks by Large Language Models, with the knowledge that making the wrong decision costs a real human being their connection to a niche community."
This account of how LLMs are used to create spam accounts across Mastodon servers is likely a glimpse into both the future of spam and the future of online communities.
"In some sense, this is a wildly sophisticated attack. The state of NLP seven years ago would have made this sort of thing flatly impossible. It is now effective. There is no way for moderators to robustly deny these kinds of applications without also rejecting real human beings searching for community."
Back in the old days, I ran communities that ran on classic forum software. Spam was an arms race then: a successful spammer could make a ton of money (and increase the search engine ranking of their clients' sites) by injecting links on community sites all over the web, so they were incentivized to get past any blockers we erected. These days, LLMs make these attacks a great deal more sophisticated - and a great deal more possible.
Then, as now, it was a numbers game. As Kyle points out:
"These attacks do not have to be reliable to be successful. They only need to work often enough to be cost-effective, and the cost of LLM text generation is cheap and falling."
The attack laid out here is fairly naive. It uses the same username and spams the same content across communities. That's actually an illustration of how low-effort it is. A higher-effort attack would see the posts customized for each community, with different usernames and profiles to fit the community's theme and culture. Those attacks are doubtless already here.
Attempts to spot LLM writing are not very reliable and have the potential to create false positives - not least because they probably rely on an LLM themselves, and therefore are prone to inaccuracies and bias. So for now this requires human moderation. That's potentially fine for a giant megacorporation (although even they struggle with the tide of AI-enabled spam accounts), but for individual, niche communities maintained by enthusiasts and hobbyists, that may be the beginning of the end.
[Link]
