Offensive AI Against Disinformation

Posted on by John Salomon

Author’s note:

It remains my firm belief that the post-World War II liberal social, economic, and political ideals of democracy, free trade, and citizens’ welfare are under constant assault by a mixture of state actors, non-state fanatics, cynical opportunists, and a lot of very, very uninformed (to put it nicely) people, many of whom may mean well, or in the worst, case, be “useful idiots”.

This remains highly relevant as Russia’s unprovoked war against Ukraine, as well as its not-so-cold cold war against perceived adversaries around the world, continues unabated. It matters even more so in light of a recent US presidential campaign marked by an incredible amount of lies, manipulation, propaganda, and a lot of incredibly stupid, ignorant commentary online. Political extremists in the world’s democracies have become emboldened by the seeming success of their tactics.

I originally wrote this article in 2023 as a (partially) tongue-in-cheek response to frustration about a dramatic growth in dis- and misinformation, political and otherwise, on Internet forums, traditional media channels, and elsewhere. Recently, I stumbled across this trenchant and insightful post-US election Reddit comment which inspired me to post the article here. It’s not updated, so some of it is a bit dated.

I never really got around to pitching it to many serious journals; editors at one publication who did consider including it rather quickly changed their minds when they got around to actually reading it. Enjoy.

Executive Summary

Disinformation is a powerful tool for destabilization of an adversary country, even in peacetime.  It is an asymmetric tactic; the amount of effort involved in sowing FUD (fear, uncertainty, doubt) is far less than the resources needed to fight it.  Even a well-informed population, good faith curation of information, and factual responses often fail in the face of proven disinformation techniques used by a disingenuous actor operating at scale (“troll farms”)

This article proposes the use of artificial intelligence as a scalable, economic, and effective counter to disinformation.  

Contextual AI analysis can identify, analyze, correlate, and track disinformation campaigns and actors across platforms, and even different issues.  Generative AI can create counter-arguments along a consistent rhetorical line, maintain a database of arguments to counter those of disinformation actors, and, through sheer quantity of responses across multiple platforms, help to discredit, bury, and otherwise neutralize disinformation messaging and the effectiveness of those responsible for it.

By recognizing disinformation for the bad faith, destructive activity that it is, and refusing to engage those responsible in legitimate discourse, it is possible to not only deprive them of legitimacy and an audience, but to actively sabotage the effectiveness of disinformation as a detriment to honest discussion and information flows.

“Artificial Intelligence” – A Note About Terminology

“AI” is 2023’s tech industry buzzword, kicked off by the launch of ChatGPT and a variety of other products starting in 2022.  It encompasses a huge range of technologies and services, most of which are not true “artificial intelligence” – from not-particularly-clever if-then structures, to expert systems, neural networks, large language models, and more.

This has led to a tremendous amount of noise – from bright-eyed futuristic enthusiasm about how we’re at the brink of the Singularity where all our problems will be solved by benevolent, godlike artificial consciousnesses,, to doomsday panic, in which an unholy blend of HAL 9000, Skynet, Colossus, and MCP will kill off all jobs and enslave humanity in a dystopian future before turning us all into grey goo nanomachine food.

The truth probably lies somewhere on the vast spectrum between these two extremes – I have no idea where, and immediately distrust anyone who claims to be able to predict the future.  What is clear, though, is that the array of technologies covered by common use of the term includes some extremely useful tools, which can help humans perform pattern recognition, content generation, and a wide range of repetitive tasks and more, more effectively and efficiently. 

For the purposes of this paper, I’m taking the lazy route of riding the coat-tails of the hype, using “AI” to cover all of these.  Please humor me.

Situation Overview

What’s happening

Propaganda and disinformation are nothing new.  In peacetime, the planting of falsehoods and rumors is a useful tool for swaying public views based on false premises.  Over the past three decades, several factors have contributed to the increasing effectiveness of such disinformation, including

  • growth in the number of channels available for spreading disinformation,
  • increasing sophistication in analytical and targeting techniques used by disinformation actors,
  • purely revenue-driven social media strategy designed to improve “friction” and engagement, for example through algorithmic segregation of consumers into like-minded communities based on their preferences (easily exploited by trolls),
  • the decline in the ability of media regulation to ensure truthfulness and accountability of news organizations – often the result of either existing free speech laws, or abusive interpretation thereof by cynical, opportunistic actors seeking to undermine the ability of government to effectively balance press freedom with the need for a reliably informed citizenry,
  • decline in customer willingness to pay for reputable media, coupled with an increase in media outlets exclusively beholden to advertiser or private investor funding, with a corresponding rise of editorial control, and the growth of clickbait/ragebait articles designed to increase traffic, and
  • increase in the reach of social media, allowing a wide range of disreputable participants to reach an ever-growing number of information consumers

What it consists of

Former US President Trump (author’s note: oh dear) popularized the term “fake news”[1] to refer to inaccurate reporting.  While Trump’s use of the expression was an attempt to cast into doubt the credibility of legitimate news outlets and journalists whose factual and editorial reporting was at odds with his own views and ambitions, his frequent references to “fake news” gave it a high degree of visibility and popularity.  Meanwhile, “Fake News” is a very real phenomenon – whether in the form of typo-squatted domains, AI-generated deepfakes of public figures, or topics such as QAnon and Pizzagate, which helped spawn a whole community of radicalized believers.

Distraction is another common technique.  Intentional distraction from major current events is a tool common to media outlets, state communications agencies, Internet forums, political candidates, and other visible sources of information with an ideological or opportunistic agenda. 

The US-based Fox News network has a strong record of referring to unrelated topics in both news and editorial coverage in order to draw attention away from ongoing issues (e.g. “Hunter Biden’s Laptop”).  Both China and Russia are strong users of this technique when their policies are criticized, frequently resorting to “whataboutism” and other rhetorical fallacies.  These are mostly well documented and understood, but can be difficult to spot outside of ideal debate conditions – for example in a social media comment thread.

 “Astroturfing”, or the manufacturing of so-called “grass roots” opinion trends among voters, also gained momentum as a tactic in recent years.  The notorious “SWIFT Boat Veterans for Truth”[2] campaign during the 2004 US presidential elections is an early and prominent example of this in a modern political context; while the phenomenon is doubtlessly experienced elsewhere in the world, it has become primarily associated with large corporate lobbying in Europe[3].

How it works

The rise of a 24/7 news cycle, driven by cable television and the Internet as a medium for news dissemination, have allowed disinformation actors a global, continuous reach.  These channels are usually limited to easily identifiable, monolithic actors – such as cable TV stations or news websites.  When news sites, TV/radio networks, or state information agencies consistently push a certain ideological line, or gain a reputation for low quality, highly biased, or dishonest reporting, they will be ineffective outside of an increasingly captive echo chamber.

Much more dangerous is the combination of disinformation techniques in the context of social media.  Forums such as Reddit, YouTube, Facebook, Slashdot, Twitter, and many others allow trolls and propagandists a degree of anonymity and universal access to large audiences.  Even when users do not actively engage in discussions involving disinformation, they are still exposed to comments.  When coming from seemingly numerous sources, combined with difficult-to-verify and credible-looking “facts”, and repeated in different formats, casual browsers may start believing and even repeating what they are presented with.

Data collection and processing innovations have made this process even more nefarious.  Big data techniques can be used to identify and track trends, analyze users across platforms, and target individuals and groups with frightening precision, whether with ads, comments, or even abuse.  Most of these tools are widely used for legitimate economic purposes, such as market research and targeted advertising – but are a valuable resource for cynical actors to make their messaging more impactful. 

Cambridge Analytica is a case in point.  While the effectiveness of CA “microtargeting” of voters in the 2016 US elections[4] based on illicitly obtained Facebook data is debated, and an investigation of the British Information Commissioner’s Office found little evidence of Cambridge Analytica having contributed markedly to the outcome of the 2016 Brexit vote[5] (whether spending laws were broken is a separate question), it is beyond doubt that big data analytics are an incredibly powerful resource for troll farms.

Tactics, techniques, and procedures (TTPs) need not even be particularly sophisticated.  Search engine optimization has existed since shortly after the creation of the first such services, and exploiting opaque search algorithms by “Google bombing” certain terms, names, or concepts with keywords or combinations thereof is an effective way to manipulate information, for example by making it more likely that a consumer will find multiple similar views and sources on their first page of results[6].

Who is doing this and why?

There is no rule of thumb about whether or not disinformation actors care about ideology.  Various ideologically consistent groups, such as Scientology and China’s “Internet Water Army”, or followers of QAnon, Flat Earth, vaccine denialism, and other conspiracy theories, may truly believe in the messages they are spreading. 

Meanwhile, groups designed from the ground up to use disinformation for destabilization purposes, such as China’s “50 Cent Party”, as well as Russia’s Internet Research Agency/”Trolls from Odintsovo” often do not care whether their positions aligned with any particular political ideology, or even consistent or coherent, as long as they create doubt and instability, although certain active measures groups do pursue clear nationalistic messaging.  In addition to Russia and China, such groups have been identified as operated by Turkey, the US, Israel, Vietnam, the United Kingdom, South and North Korea, and other countries.

For example, Russia is alleged to have been involved to some degree in the Catalan independence movement[7], anti-vaccine groups during the COVID-19 pandemic[8], Australian anti-islamic propaganda[9], German elections[10], and anti-war groups around the world[11], to name a few.  None of these follow any clear ideological line, beyond creating doubt, undermining the legitimacy of established media and government messaging, and exploiting divisions. 

Indeed, Russian trolls often operate on both sides of contentious issues, exploiting deeply held convictions to widen fissures in target societies, for example simultaneously impersonating black activists and white supremacists, or pro- and anti-vaccination activists[12].  They have sometimes engaged in astroturfing – announcing physical rallies by purported activists, apparently in the hope of sparking violent conflicts with counter-protestors.  Russian intelligence operatives amplified Muslim-Jewish tensions to fragment the Women’s March movement that protested the election of Donald Trump in 2017[13].  And, of course, agents provocateur will frequently present themselves as supporters of a given side while espousing lies or extremist caricatures of that side’s views, in order to delegitimize a given set of views – a particularly hilarious example of this can be found in political activist Dean Browning’s unfortunate “as a black man” tweet, exposed by the author’s inexplicable failure to switch Twitter accounts before posting[14].

That is not to say that all disinformation actors are entirely driven by the pure desire to spread instability.  Many campaigns, such as various anti-Rohingya campaigns at least partially incited by Myanmar’s military[15]. include genuine incitement to hate of “others”.  The explosive and highly divisive phenomenon of identity politics, whether racial, religious, or ideological, in the past decade, has also created opportunities for extremists to manipulate online opinion.  This is nothing new; Radio Télévision Libre des Mille Collines[16] is widely credited with strongly exacerbating the 2004 Rwandan Tutsi genocide by extending the reach of Hutu extremists to a wider population. 

However, with the increased reach available to such messages through social media and other online platforms, cynical actors, whether state aligned or simply trolls bent on creating mayhem, frequently reinforce and amplify “real” disinformation.  Michael V. Hayden, former director of the United States Central Intelligence Agency, claims that “you never create a division in a society, you identify and exploit preexisting conditions”[17].  This is certainly the case in many of the campaigns mentioned above.

Why existing means cannot stop it

Loaded questions are one of many rhetorical techniques used when manipulating online conversations.  The courtroom trope question “so, when did you stop beating your wife?” is typical of messages based on false premises that can be used to influence opinions.  Even though the underlying idea may be completely false, the fact of it having been voiced means it is now on record, and lodged in the minds of uninvolved readers.  The more such a statement is repeated, or even only alluded to, elsewhere, the more it may be seen as legitimate.  Attempts to correct false statements, or even to point out the nature of the rhetorical fallacy will not change this.

Similarly, techniques such as the Gish Gallop[18], shifting the goalposts, red herrings, strawman arguments, and many others, require a vastly greater investment of time and effort to correct and neutralize.  There exist initiatives to teach online media literacy and engagement to readers and good faith participants in news and debates, such as Finland’s comprehensive program to teach such skills from an early stage[19].  Nonetheless, Internet discussions are often casual and spontaneous, and thus emotional, not to mention having an international participant base with vastly differing levels of intellectual tools for spotting and processing disinformation. 

Trolls cannot be ignored – they can continue spreading their message, making it almost inevitable that at least some Internet users will start to consider their “arguments”.  A lie, if it looks sufficiently credible and is repeated often enough, may even become accepted as widespread truth, leading even critical social media users to doubt their previous convictions.  

This means that even identifying and calling out trolls is often ineffective.  They can claim innocence, while using sock puppet accounts to jump to their defense.  Feigned indignation and protests against “censorship” are a very effective way of building sympathy.  Very often, trolls initially couch their messages in innocuous, reasonable sounding terms (“I’m just asking questions”, “immigrants should obey national laws”), or in humorous form (e.g. white supremacist groups’ co-opting of the Pepe the Frog meme[20], which makes pointing fingers at trolls even more difficult until more extreme forms of their messages are already well established and echoed by a significant number of persons who actually believe the fundamental message – viz. the development of the incel phenomenon[21].

Banning or moderating trolls does not work well; any Internet forum that is sufficiently well moderated and restrictive to be able to totally control bad actors, will also be highly unlikely to attract any significant user base.  While restrictions have limited effects, lack thereof allows trolls to flourish, for example, platforms have insufficient resources or motivation for controlling disinformation content. Facebook’s well publicized failures to combat Burmese-language hate speech in Myanmar[22], and Twitter’s cuts to content management staff[23] are powerful illustrations of troll activity increasing dramatically in the absence of controls.

Nor can trolls be engaged constructively via good faith debate.   Even if corrected, a naïve “useful idiot” who repeats dishonest talking points may succumb to aspects of cognitive dissonance[24], and double down on their opinions.  Dramatically oversimplified, the primitive “lizard brain” aspect of human psychology often takes over when a person feels they and their beliefs are being attacked.  

The aforementioned problem of identity politics makes it possible to weave “us versus them” narratives (left vs right, black vs white, straight vs gay, Hindu vs Muslim, Serb vs Bosnian); anyone questioning a belief that has been successfully linked to a group’s identity (“environmentalism is for liberals, liberals hate freedom, I hate liberals, thus I hate anyone who argues for environmentalism”) thus becomes one of “them” who must be countered on basic principle.  Even worse, engaging with a troll or trolls legitimizes them and their statements by implying they are participating in a discussion on equal footing.

Most importantly, it is simply not possible to keep up with online trolls.  A well-resourced group with a clear mission and structure can easily shift forums, topics, and semantics.  Troll networks relying on automation to generate and disseminate messages will continue to rise, especially with the ready availability of tools such as ChatGPT and Midjourney, giving trolls tremendous economies of scale with decreasing need for human labor-intensive troll farms.

Government agencies tasked with defending against propaganda and disinformation are likely to be under-resourced, or focused on a specific scope of forums.  Even intelligence organizations responsible for tracking and analyzing bad actors and their campaigns are unlikely to have the capability to actively defang the trolls’ activities and their impact.

This asymmetry of effort, combined with the fact that trolls can largely choose where and when to focus their efforts, leaving “defenders” on the back foot and obliged to respond to campaigns in progress, must also be considered as part of the fact that an aggressor does not necessarily need to “win” an argument in order to succeed.  Success factors can include, among many others:

  • Creating doubt in the minds of uninvolved participants
  • Discrediting forums
  • Influencing opinions
  • Discrediting or weakening legitimate groups and viewpoints
  • Forcing expenditure of resources and time
  • Distracting from specific issues
  • Demotivating voters
  • Influencing politicians
  • Influencing commercial firms (e.g. to reduce ad spending)

An excellent example of this is found in Russian disinformation activity in support of the invasion of Ukraine in 2022[25].  Manipulation of public opinion has successfully introduced numerous tropes into discussions of whether the West should continue to support Ukraine – including “both sides” narratives of war crimes, allegations of Ukrainian corruption, supposed nazi sympathizers fighting on Ukraine’s side, and many more.  Whether these have any truth to them is irrelevant.  What matters is targeted and cynical Russian exploitation and amplification of messages that find resonance with groups such as European peace activists and others who may influence their governments’ policies towards Ukraine and Russia.

Informal trolling efforts, such as 4Chan’s “it’s okay to be white”[26] campaign, are definitely capable of concerted messaging.  However, state-affiliated or -sponsored actors are much more likely to be behind such attacks. They are far more capable of mustering the combination of a clear mission, coordination and leadership, and tools needed to leverage the advantage of asymmetric effort investment that makes targeted, strategic disinformation so highly effective.

Why it’s significant

The line between “peace” and “war” has always been blurry.  Adversarial countries and interest groups have a long history of attempting to undermine competitors’ political, economic, and social stability through fake news.  No formal declaration of war has been issued between two states since 1945. 

Until the increasing prevalence of talk radio since the 1980s, the growth of cable TV news, and most importantly, the widespread adoption of the Internet, propaganda mainly followed predictable lines – efforts to publish Op-Ed pieces in newspapers arguing a given side, lobbying, and possibly donations to potentially friendly politicians.  All of these are potentially easily tracked and restricted.

Internet debate, whether on dedicated forums or on discussion boards appended to video sharing or news sites, is a different animal.  The discussion ecosystem presents an incredibly diverse, attractive environment for trolls to flourish.  Barriers to entry are minimal or nonexistent, anonymity is easy, and even when mechanisms exist to establish reputation (e.g. age of an account, up- vs. downvotes, etc.) these are easy to game, especially given sufficient time and resources.

From sabotaging legitimate discourse to outright influencing policy and elections, trolls have an incredible ability to destroy social cohesion, economic prosperity, political stability, and faith in institutions.

How Can We Fix This?

Trolls cannot be defeated by ignoring them, or by engaging them in debate.  Disregarding them allows them to proliferate unchecked, and bears the risk that well-crafted disinformation (especially in the form of deepfakes) will be sufficiently believable that well-meaning, gullible persons will repeat and amplify the message – especially if these play into existing social divisions and confirm people’s biases and prejudices, particularly when social media algorithms can be gamed to inject such content into established online echo chambers.

At the same time, dignifying a disinformation actor with a well-intended rebuttal or correction legitimizes them, and raises their disingenuous content to the same level of validity as that of a “real” source.

A new paradigm is needed, which actively engages and disrupts trolls’ advantages described previously.  Effective measures rely on highly scalable, integrated, and coordinated use of automation.

Identifying and Tracking

Artificial intelligence, specifically a combination of interpretative and generative, can be used to:

  • Identify and track forums and potentially contentious topics
  • Identify trolls by “fingerprinting” their posts
  • Detect commonalities – e.g. sockpuppet accounts, whether human or bot, working from a similar script or talking points
  • Classify networks of shared troll accounts
  • Monitor troll activity across platforms

Neural network and AI technology, combined with existing big data analytics capabilities, is already well able to handle these tasks.  For example, the AUCH (Autorenprofile für die Untersuchung von Cyberkriminalität CH, «Author profiles for the investigation of cybercrime CH») project of the University of Zurich, referenced in cyber-threat intelligence analysis firm PRODAFT’s deep-dives into the WizardSpider[27] and PYSA[28] ransomware gangs, uses semantic analysis to identify with a high degree of confidence the national origin of malicious actors communicating in English as a second language.  Such “tells” should allow for the establishment of behavioural indicators shared by participants in similar troll networks.

It should even be possible to analyse and track automated bots.  The same big data analytics techniques used and sold by Cambridge Analytica are highly impactful when used by Google, Facebook, Amazon, and others for targeting advertising and suggesting content.  As AI tools become more prevalent and capable, it may become increasingly difficult to distinguish human- from automatically-generated content. 

Thus, behavioural analytics become extremely important – this is more true when enriched with metadata such as activity times, username formats, or frequency of contributions – and even more so if any information such as IP addresses can be obtained. No matter how many platforms and discussion groups a troll network is active in, collecting and correlating their characteristics, building profiles, and tracking these is technologically very feasible.

Organisations such as the Stanford Internet Observatory, Graphika, the Atlantic Council’s Digital Forensic Research Lab, Kate Starbird at the University of Washington, the Center for Countering Digital Hate, and others have used a wide range of techniques to identify clusters of keywords and trending topics, groups of followers and those they follow.  Tools such as the DISinformation Analysis and Risk Management (DISARM) Red framework[29] for tackling FIMI rely on MITRE’s proven ATT&CK system for classifying threat actors and techniques.  Much of the capability to identify, correlate, and track fake content already exists and is in widespread use[30]

Some researchers have already incorporated AI.  Logically AI[31] is an organisation that already says, “we combine advanced artificial intelligence with human expertise to tackle harmful and problematic online content at scale”.  The company’s approach is to track, analyse, and communicate malicious actor behaviour via a threat intelligence platform.

The efficiency of such activity is hampered by a decrease in availability of certain source data.  In recent years, many platforms have reduced access to large amounts of data that was formerly available for analysis – one example being Facebook/Meta as a result of the outcry about Cambridge Analytica’s activity, or Twitter/X as a result of the destruction of much of its engineering capability under Elon Musk.  This affects both trolls and counter-trolls, but does not change the underlying idea of analysing large amounts of data – it just means that more active, real-time monitoring of platforms is required.

Thus, while existing approaches to tracking and countering trolls and disinformation campaigns are maturing, and have access to a reasonable and growing toolkit, a more aggressive and comprehensive approach is needed.  Only a massive increase in the use of automated techniques both on the analytical (passive) and strike-back (active) side can have any hope of having enough impact to be able to defang the trolls.

Active Countermeasures

Once we know who is behind a troll network, how they behave, when, where, and how they are likely to comment, and other aspects of their behaviour, it is time to consider neutralizing them.  To this end, we propose using similar information management approaches to those of the troll networks – whether human or automated. 

A 2018 criminal case[32] filed by the US Dept. of Justice against the Russian Internet Research Agency, Concord Management and Consulting LLC, and a list of key figures in these entities alleges that “the ORGANIZATION employed hundreds of individuals for its online operations, ranging from creators of fictitious personas to technical and administrative support.  The ORGANIZATION’s annual budget totaled the equivalent of millions of U.S. dollars.”  Similarly, some estimates place the number of members of the Chinese “50 Cent Party” between “tens of thousands”[33] and several hundred thousand[34], capable of churning out hundreds of millions of comments and blog posts each year.

The key to defeating such numbers and resources is sheer volume, and automation.  Only by automated, constant development of millions upon millions of “good”sock puppet accounts, active on every social media network in existence, prepared to respond to, report, downvote, and otherwise hamper troll activity when it is identified, can their effectiveness be matched. 

Countermeasures should roughly aim to do three main things:

  1. Call them out

The “tame” response to actively trolls.  A tried and true debate technique used to undermine opponents is to identify and devalue their views and thought processes at a very fundamental level.  A famous example of this is found in the October 28, 1980 US presidential debate.  Ronald Reagan’s response to Jimmy Carter, after the incumbent President provided a detailed set of points attacking one of his opponent’s policy, was “there you go again”[35]. 

Similarly, just identifying trolls as part of a known network, and describing their techniques, is a good start in disarming them by showing them for what they are.  The US Department of State’s Global Engagement Center[36] recently embarked on a policy of pre-empting Russian influence campaigns and weakening the impact of their propaganda, by explicitly describing what trolls are doing[37] – what tactics and channels they use, what their objectives are, who is behind the campaigns, and similar meta-information.

This is a good start – but more is needed.  For one, many targets of disinformation campaigns are unlikely to read a thoughtful, authoritative report issued by a government agency. The trolls already have a first-mover advantage by engaging their audience via social media channels; this is where they must be confronted.  Publishing informed analysis remains necessary – it will be more impactful when linked to in response to known disinformation comments.

  1. Bury them with bullshit

The moment that any disinformation activity is detected it should be smothered in counter-content using the same sort of rhetorical fallacies, distractions, even irrelevant or jokey memes, in order to diffuse and bury the original message.  In addition, but identified troll accounts and others associated with them should be followed across platforms and forums and continuously hounded and spammed.

Search engine and keyword bombing and social media platform algorithm manipulation are additional techniques that large volume offensive AI should be capable of managing very well.  Abuse of e.g. YouTube’s suggestion algorithm by malicious actors, sending viewers down ever more extreme content rabbit holes, can equally be gamed – for example by hijacking keywords driving viewers searching for extremist content towards “friendly” propaganda, or even better, nonsensically extreme content as proposed below. 

Generative AI and deepfake technology can be used to create text, image, audio, and video content.  Such content does not even have to be perfect –  as long as it is somewhat plausible, and presented from a wide variety of “unique sources” (astroturfing techniques) it stands a high chance of catching vulnerable consumers, and of burying real disinformation content.

  1. Out-crazy the crazies

As mentioned previously, Internet trolls seek to exploit and worsen divisions in online discussion, and thus society, frequently by espousing extreme or irrational views.  The rise of flat Earth, antivax, chemtrails, and other scientifically nonsensical trends, as well as (mainly but far from exclusively right wing) political extremism and polarization in Europe, North America, and elsewhere, have all been driven to some significant degree by online trolls.

By taking the content published by online trolls and generating versions of their messages dialed up to eleven, it should be possible to drive other forum participants to associate trolls with extreme, distasteful views. 

For example, amplifying a more-or-less subtle racist dogwhistle with a thoroughly absurd, comedically exaggerated comment while strongly agreeing and identifying with the original commenter, even implying their affiliation with the fake hyper-extreme response (“I totally agree with your point about George Soros and his (((globalists)))[38], did you realise that he’s behind BOTH sides of the Russian special military operation in Ukraine?”) , can drive an uninvolved, casual reader to assume that the real troll is in fact a member of a hyper-extreme group.

Another example could be COVID-19 disinformation – “obviously the virus was created in a Chinese lab…with the collusion of the US Republican party and Jeff Bezos, in order to drive more traffic to Amazon’s cloud services”.   These are obviously completely ludicrous exaggerations – the fine art of crafting such false extra-extremist messages lies in making them just believable enough.

Unfortunately, some current conspiracy theories, such as QAnon (a secret cabal of US high government officials and oligarchs is kidnapping children, and killing them for their adrenochrome, which is used to extend the lifespan of the perpetrators), or flat Earth (self-explanatory), are so patently absurd that creating even more extreme messages to discredit their core ideas would require significant amounts of fantasy, shamelessness, and alcohol.

Similarly, aggressively questioning the credibility of a troll from a position of “intellectual purism”  can undermine their impact.  Accusing trolls of not going far enough, not being purists, or being downright traitorous to the “cause” they may be espousing is likely to drive away potential sympathizers, for whom the position of the “counter-trolls” is too extreme by far, but who may now feel the genuine troll is not believable.

We are aware of informal groups engaging in both of the above techniques on various platforms – circulating fake memes, claiming to be genuine members of groups interested in bringing out “the truth”, and similar methods, with some success. An excellent real-life example of such believable-looking counter-trolling can be found in a 2017 CPAC event in the US, where covert activists handed out miniature red-white-and-blue (Russian) flags with real-looking gold “TRUMP” logos on them, and succeeded in getting attendees to enthusiastically wave these[39].

Variations on this theme have worked in the past.  In the run-up to the 2022 US midterm elections, several Democratic actors actively campaigned against more moderate Republican opponents in that party’s primary elections, in the hopes of forcing a contest between a more electable Democratic candidate and an extremist Republican[40].  They at least partially succeeded, leading to continued Democratic control of the US upper legislative chamber, and a far narrower margin in the lower house, against all expectations.  Analysis of that experience can help in predicting the intended and unintended consequences of the more automated, AI-supported approach proposed here.

Faking an Avalanche

To work, all these techniques rely on volume.  Most likely, a significant multiple of estimated troll accounts and comments will be required, possibly as low as 20 and as high as several hundred times the volume of what the trolls may generate.   

Volume is also important when it comes to “poisoning” search results – for this technique to work, it is imperative to be able to reliably spam search engine results for queries about common conspiracy or disinformation terms with desired content.  Abuse of paid advertising services (e.g. via Google keyword purchases) will increase the reach of keywords.

Speed is equally vital – to be effective, messages must be quickly identified and responded to.  This only works on forums with the possibility of immediate response, such as Twitter/X, Facebook, Telegram, or discussion channels on news sites. 

The amount of tracking, as well as the quantity, breadth, and speed of response that are required for this necessitate automated, at least partially autonomous solutions.  Comparatively minimal investment will multiply such efforts.   Automated signups to VPN services, both free and paid (many of which accept payment in hard-to-trace cryptocurrency), and sponsored content can aid in pushing counter-propaganda to the top of search results, or automatically generated feeds.  If a technology platform refuses to adequately curate the quality and truthfulness of what it allows users to see, this creates not only an advantage for disingenuous actors, but also for determined sabotage of those actors and their messaging.

It must be noted that none of the techniques being proposed are in any way technologically novel.  What makes this approach “new”, and thus effective, is the use of widespread automation to create incredible economies of scale – with limited parametrization and spot checks by humans, bots can analyze the best avenues (channels, keywords) for content, develop media, and push it out en masse.

Genuine Information

Importantly, this is not a substitute for genuine informational content, which corrects misinformation.  However, in order for this to work, content must:

  • be concise and to the point.  Easy to digest information, even if it simplifies complex issues, is more accessible and likely to be accepted
  • not be confrontational.  Asking questions, or even simply presenting facts that do not in themselves explicitly contradict disinformation, but stand on their own, is less likely to trigger a defensive reaction
  • come from multiple sources.  This can be automated, using the mechanisms described in point (1) above.

Such content can be shared via “positive” search engine manipulation, such as is promoted by Francesca Tripodi[41].  Automation can be used to intelligently improve the use of members and hashtags, key words, and other SEO terms to game the algorithms into making real, truthful information more prominent. 

Human Involvement

As with most use cases for artificial intelligence solutions, both analytic and offensive use of AI against disinformation actors are a crutch, not a replacement for human experts.  They are a “force multiplier” in that they allow a far more efficient use of limited resources, can recognize patterns across a wide range of channels more effectively (including multiple languages), and can execute automated response campaigns once the parameters of these have been decided on.

Human interaction will be required nonetheless, including in the following areas:

  1. Verify and sanity-check analysis outputs
  2. Provide quality assurance – for example, spot checks to ensure “friendly” bots and their actions pass a basic Turing test (i.e. to fool an average consumer into thinking content is real)
  3. Review AI outputs periodically if there is question that an innocent participant’s contributions may have been unfairly targeted
  4. Initiate contacts with law enforcement and intelligence agencies, platforms, or private firms as appropriate

Challenges

Proportionality and Governance[1] 

Who decides when to engage in offensive countermeasures, and to what degree?  Who decides whether this authority is being used responsibly and within legal boundaries?  As with any capability that has the potential to inflict some sort of damage, clear rules of engagement and governance are necessary.

Whether this should involve defense establishments, law enforcement agencies, judges, national cybersecurity entities, a netizens’ advisory board, or other functions must be debated in detail – but we assert that creating such governance while maintaining speed and flexibility is absolutely possible. 

The idea of a responsible supervisory body that maintains the balance between national defence and constitutional rights and norms is nothing new; for example, the Federal Republic of Germany’s Bundesamt für Verfassungsschutz[42], or domestic intelligence agency, has successfully navigated this challenge since its formation in 1950. 

Offensive operations themselves must be reserved to government bodies, or legally deputised entities.  Max Weber first posited the idea of a “government monopoly on violence”[43] – this extends to the modern concept of active, offensive countermeasures being reserved for public sector entities with an explicit mandate to strike at threats beyond their borders, either pre-emptively or reactively.  These mainly consist of military, law enforcement, and national intelligence agencies.  Any unauthorised civilian offensive activity risks running afoul of laws prohibiting vigilantism.

Regardless of who performs the actual active countermeasures, they must be subject to a clear and consistent review process, and ultimately answer to elected civilian authority in all cases.

Government’s Role as Information Source

Where is the line between a government or intergovernmental agency publishing authoritative information, and propaganda?  This question has gained ground significantly in recent years, as trust in public sector agencies has decreased (ironically, often as a result of the disinformation campaigns this paper addresses). 

For example, during the 2020-22 COVID-19 pandemic, tremendous amounts of online content sought to discredit agencies such as the World Health Organization or national health ministries – often, poor initial communication and contradictory messages, e.g. about the efficacy of masks, provided fodder for trolls seeking to destroy the believability of even qualified, informed officials.

A 2022 OECD report[44] claims that across the 22 OECD countries, trust in government agencies is fairly solid, but that citizen input and transparency are key to maintaining and building this trust – we agree wholeheartedly. 

There is also the more fundamental issue of whether a government should be providing news and information at all – it is our opinion that this is a core function of government, to act as a trustworthy, responsible source of facts when this is in the public interest, especially when there is no credible other source for such information.  This is most important when it comes to attempting to counter disinformation with accurate statements and corrections.

Already in the 2017-2020 period, top appointees in various US government agencies prevented the career professionals under them from publishing true information, undermining a long tradition of responsible professionalism in the US government[45]. Depending on who comes to power in a given election, future leaders who do not value accuracy could take advantage of the technical tools described here in the service of less honorable motives.

Who is to define what constitutes valid and authoritative information? Intense arguments over the content of school curricula[46] show the difficulty of agreeing on what is truth and what is disinformation. The harsh reaction to the US government’s attempt to set up a disinformation committee, and the death threats and harassment that forced its nominee, Nina Jankowicz, to step down from the planned position, show the intense conflict over these issues.

Many government agencies have clear guidelines for when they can publish information and create false personas.  It is vital to ensure a government’s involvement in active disinformation does not discredit its legitimate information activities.  However, this is a good place to distinguish between government agencies allowed to use force or other “active measures” (law enforcement, national intelligence directorates, military) and those who are not. 

Active information measures, including dissemination of disinformation of propaganda via fake identities has been a staple of intelligence and law enforcement entities since time immemorial – without having much of an impact on their “civilian”[47] counterparts’ credibility.  We believe that this dichotomy is manageable in an age of active, automated information warfare – it has been in the past.

Ethical Questions

Is it acceptable to sabotage online discussion in the interests of crippling troll networks?  This is a philosophical question that this article is badly equipped to answer. 

It is the authors’ assertion that, yes, this is absolutely justifiable.  An environment which allows trolls to thrive to such a degree that they can cause social, economic, and political damage through their disinformation campaigns is, while not itself a weapon, essentially a channel through which destructive content is allowed to flow.  Using similar means to counter that content does not mean actively shuttering the platform itself – for example via legal injunctions, mandatory web filtering, or other means.  It has already been turned into a battleground; the only question is, are aggressors to be allowed to dominate it?

Furthermore, if point 2 above (“out-crazy the crazies”) is followed – what are the implications if this results in outright hate speech, calls to violence, or other content that is neither acceptable nor legal in conventional forums?  This technique will only work if all the stops are pulled out, and content is generated without scruples.  Are we willing to go this far?  Again, there is no easy answer. 

However, it bears remembering that the objective of many trolls is inherently destructive; their end game accepts fomenting extremism and violence.  It is not 100% clear that, if not countered, such disinformation will inevitably lead to a destruction of democratic systems.  In the 2020 and 2022 US elections, the 2022 French presidential election, and the 2023 Spanish parliamentary elections, voters chose more moderate candidates (and in many cases, dealt humiliating blows to extremist politicians) despite a pervasive climate of disinformation.  The persistent and vocal presence of political arsonists, and of the information channels that support them and continue to foment chaos and distrust.

Ruining Legitimate Discourse

A platform that tolerates online trolls to the point where they can influence public opinion in any widespread manner, is already lost.  The main case in point for this argument is Twitter; since its acquisition by Elon Musk in 2022, the service’s already tenuous claim to keeping disinformation and trolls in check (pun intended) has been utterly gutted.  Moderation and content analysis have been defunded, staff fired, functionality left to die on the vine, and verification of genuine users turned into a simple for-pay function – a bitter irony given Mr. Musk’s frequent condemnation of Twitter botnets.

The destruction of social media platforms is unfortunate, insofar as it can remove channels used by genuine groups – such as citizens’ resistance networks in totalitarian regimes, disaster alerting services, news channels, and others.  However, both Twitter and Facebook’s example show that, rather than seeking to preserve troll-infested waters for a shrinking number of genuine users, a full-scale assault on troll farms and their visibility will either a) motivate platform operators to place genuinely effective controls on content, thus improving discourse, or b) motivate the rise of superior alternative platforms that are willing to confront a real issue.  

A third option is that regulation will eventually force them to comply with certain minimal standards.  Several governments are pursuing legal means to force social media operators to vet their content more aggressively[48], moving away from the “common carrier”[49] idea.  It will take time to determine whether this approach works.  In the meantime, platforms that continue to tolerate destructive content in the interest of continued profitability are a problem for society.

Legality

The legal implications of active countermeasures are a major question that needs close analysis.  Do computer misuse acts prohibit such measures?  What about cross-border activity – would it be permissible, for example, to go after Russian trolls on vKontakte?

As with the question of governance, we believe it is possible to establish clear rules and conditions of engagement – for example, when the stability of critical infrastructure, elections, economic activity, and other aspects of national security are threatened.  That said, liberal democratic governments frequently operate both good-faith propaganda resources (e.g. Voice of America, Radio Free Europe) and malicious actor disruption capabilities. 

A legitimate issue is whether actively countering bad actors by blocking or correcting them has the same standing as spreading disinformation.  We believe this is absolutely the case, and that the two are, at a fundamental level, no different.  Either way, establishing clear and consistent principles-based rules for when, to what degree, and how, automated anti-troll countermeasures are permissible is a vital but absolutely feasible pre-condition for their use.

Confidentiality of Campaigns and the Nature of Truth

If the question of legality can be addressed, what happens if this proposed campaign is uncovered?

This will likely be a good thing.  If done correctly, counter-trolling will be so diverse and free of patterns that it is impossible to determine authoritatively whether it is part of a given campaign.  Thus, the knowledge that a significant proportion of extreme and dishonest content is fake will a) underscore and strengthen consumers’ realization that much of what they are seeing in the first place comes from trolls, and b) lead more viewers/readers to question the veracity of troll content that might otherwise have been blindly accepted.

A potential dangerous, unintended effect, though, could be the undermining of any idea of even trying to find trust, playing into the hands of people like Russian ideologist Vladislav Surkov to cast doubts on the very idea of “truth”[50].  Both Russian actors and extremists troll groups have sought to push a “both sides are untrustworthy” narrative.  If we acquiesce to the idea of a “post-truth” society, it would mock the efforts of investigative organisations like Bellingcat, whose motto is “truth in a post-truth society” and contribute to the existential confusion that has likely helped contribute not only to a growth in support for fringe political groups by citizens seeking more simple, understandable messages, but even a rash of “deaths of despair” in mostly younger people struggling to cope with a lack of social and political clarity. 

This is why the aforementioned, complementary positive messaging is absolutely vital.  Falsehoods and propaganda are already omnipresent – more will not cause a quantum shift in attitudes towards truth.  Rather, we believe that it would allow organisations like Bellingcat to place themselves even more above the fray, and distinguish themselves and their messages from the trolls.

Receptiveness by True Extremists

It is likely that a percentage of readers exposed to the high volume of exaggeratedly extremist counter-disinformation content will take it seriously, and in the worst case, act on it.  This risks inadvertently contributing to stochastic terrorism, as potentially militant fanatics decide to act on disinformation.  They may take such material as an incentive to commit violence against groups they see as “enemies”. 

Hopefully, such persons are a small minority.  One of the goals of our counter-disinformation campaign is to disincentivize potential sympathizers by appealing to any residual sense of perspective, morals, ethics, and other sources of reticence to engage in violent acts. 

At the same time, it is very important to ensure that messages do not contain, explicitly or implicitly, incentives to target innocent individuals or specific groups of people – particularly vulnerable communities such as religious or ethnic minorities, migrants, LGBT+ persons, or others who might be victimized by zealots.  A significant gray area arises when attempting to pit one extremist community against another (“let them fight”) – see sections “Ethical Questions’ ‘ and “Legality”.

Finally, it must be said that some individuals are prone to committing violence no matter what content they consume online.  It is counterproductive to avoid triggering violence from such persons; rather, these countermeasures may help identify them so that law enforcement can take preventive action.

Conclusion

Online troll farms and disinformation are not going away.  They are a proven tool for manipulating discourse and influencing politics; countering them is difficult, slow, and expensive, and they have an impact far beyond the resources required to run a troll farm.

While there is no substitute for education (including media literacy training), media standards, Internet forum moderation, genuine discourse, and intelligence gathering, as well as other countermeasures, these are insufficient by themselves to counter and neutralize the effect trolls have on liberal democratic free market society and its continuing functioning. 

The only way to effectively and dramatically diminish the influence and effectiveness of trolls is to stop treating them as genuine participants in online discourse – a process that is already happening in many environments.  More importantly, though, only technological innovation with proper human strategic guidance will be able to match and exceed the trolls’ asymmetric impact – whether society is willing to take the tough decisions needed on how to implement such measures is a major question in its own right.

[1] https://www.independent.co.uk/news/world/americas/us-election/trump-fake-news-counter-history-b732873.html

[2] https://www.mediamatters.org/swift-boat-veterans-truth

[3] https://www.ft.com/content/74271926-dd9f-11e2-a756-00144feab7de

[4] https://www.npr.org/2018/03/20/595338116/what-did-cambridge-analytica-do-during-the-2016-election

[5] https://www.spectator.co.uk/article/were-there-any-links-between-cambridge-analytica-russia-and-brexit/

[6] US (mainly) right wing political messaging provides several excellent examples of this, notably by attempting to undermine the investigations into collusion with Russian and other foreign actors, e.g. https://www.wired.com/story/devin-nunes-and-the-dark-power-of-keyword-signaling/

[7] https://cepa.org/article/catalonia-where-theres-trouble-theres-russia/

[8] https://aej.org/2022/08/06/russia-manipulating-gullible-anti-vaxxers/

[9] https://www.theguardian.com/australia-news/2018/nov/20/russian-twitter-trolls-stoking-anti-islamic-sentiment-in-australia-experts-warn

[10] https://www.spiegel.de/international/germany/trolls-in-germany-right-wing-extremists-stir-internet-hate-a-1166778.html

[11] https://www.vice.com/en/article/wxdb5z/redfish-media-russia-propaganda-misinformation

[12] https://nattothoughts.substack.com/p/troll-humor

[13] https://www.nytimes.com/2022/09/18/us/womens-march-russia-trump.html

[14] https://eu.usatoday.com/story/news/nation/2020/11/10/dean-browning-white-politician-tweets-black-gay-guy/6243406002/

[15] https://www.nytimes.com/2018/10/15/technology/myanmar-facebook-genocide.html

[16] https://genocidearchiverwanda.org.rw/index.php/Radio_T%C3%A9l%C3%A9vision_Libre_des_Mille_Collines

[17] https://www.atlanticcouncil.org/commentary/transcript/former-cia-director-michael-v-hayden-s-remarks-at-stratcom-2018/

[18] https://en.wikipedia.org/wiki/Gish_gallop

[19] https://www.theguardian.com/world/2020/jan/28/fact-from-fiction-finlands-new-lessons-in-combating-fake-news

[20] https://www.adl.org/resources/hate-symbol/pepe-frog

[21] https://www.npr.org/2021/03/13/976379494/manosphere-world-of-incels-exposed-in-laura-bates-book-men-who-hate-women

[22] https://www.reuters.com/investigates/special-report/myanmar-facebook-hate/

[23] https://www.bbc.com/news/technology-65067707

[24] https://www.sciencedirect.com/topics/social-sciences/cognitive-dissonance-theory

[25] https://www.oecd.org/ukraine-hub/policy-responses/disinformation-and-russia-s-war-of-aggression-against-ukraine-37186bde/

[26] https://www.adl.org/resources/blog/4chan-another-trolling-campaign-emerges

[27] https://www.prodaft.com/resource/detail/ws-wizard-spider-group-depth-analysis

[28] https://www.prodaft.com/resource/detail/pysa-ransomware-group-depth-analysis

[29] https://www.enisa.europa.eu/publications/foreign-information-manipulation-interference-fimi-and-cybersecurity-threat-landscape/@@download/fullReport

[30] See Graphika’s report on continued Russian efforts to subvert US election messaging – https://public-assets.graphika.com/reports/graphika_stanford_report_bad_reputation.pdf (dead link, working copy available at https://www.documentcloud.org/documents/23451812-graphika_stanford_report_bad_reputation-1)

[31] https://logically.ai

[32] https://www.justice.gov/file/1035477/download

[33] http://news.bbc.co.uk/1/hi/world/asia-pacific/7783640.stm

[34] https://gking.harvard.edu/files/censored.pdf

[35] https://en.wikipedia.org/wiki/There_you_go_again

[36] https://www.state.gov/bureaus-offices/under-secretary-for-public-diplomacy-and-public-affairs/global-engagement-center/

[37] https://www.nytimes.com/2023/10/26/technology/russian-disinformation-us-state-department-campaign.html

[38] https://www.adl.org/resources/hate-symbol/echo

[39] https://www.politico.eu/blogs/playbook-plus/2017/02/trump-branded-russia-flags-at-cpac/

[40] https://www.reuters.com/world/us/democrats-risky-midterm-strategy-elevate-election-deniers-appears-pay-off-2022-11-09/

[41] https://yalebooks.yale.edu/book/9780300248944/the-propagandists-playbook/

[42] https://www.verfassungsschutz.de

[43] Max Weber, Politics as a Vocation, 1919

[44] https://www.oecd.org/newsroom/governments-seen-as-reliable-post-pandemic-but-giving-citizens-greater-voice-is-critical-to-strengthening-trust.htm

[45] Described in detail by Michael Lewis in The Fifth Risk:  Undoing Democracy

[46] A good example of this can be found in conflicts in e.g. Poland, Hungary, and the US states of Florida and Texas about historically revisionist “acceptable” textbooks and lesson contents

[47] We are aware that law enforcement and intelligence agencies are civilians.  We use this term to distinguish them from their opposites in e.g. health, education, transportation, etc. ministries.. 

[48] E.g. the EU Digital Services Act

[49] Common carrier doctrine, initially defined for telco operators, states that an infrastructure provider cannot be held responsible for illegal content traversing their network and systems, due to the impractical level of effort required to police it.  Internet user-generated content platforms strongly espoused this idea in the late 1990s and early 2000s, arguing that to police one piece of information would oblige them to check all content, leading to an uneasy situation (largely driven by intellectual property owners and entities combating child sexual abuse materials and violent/terrorist content) where a platform operator must take action against illicit content they are made aware of, but not generally be proactive.  Automated attempts to do so (e.g. YouTube’s automated copyrighted material detection mechanism) have yielded less than satisfactory results.

[50] https://nattothoughts.substack.com/p/putin-the-spy-as-hero

countermeasures disinformation information warfare misinformation propaganda state actors