How online disinformation networks work, and why they keep winning

That single post, reported by the Guardian on 11 October 2023, is a useful entry point into how online disinformation networks operate. They rarely invent material from nothing. They take real footage, drop it into a false frame, and let the platform do the rest.

The same Guardian review catalogued other examples from the days after Hamas attacked Israel on Saturday 7 October. One video, originally published on the official YouTube channel of Azerbaijan's state security service, showed arrested former leaders of the breakaway Nagorno-Karabakh government. It was reposted on X as footage of captured Israeli generals and racked up 2m views. A doctored White House memo, presented as Joe Biden giving $8bn to Israel, was viewed 400,000 times. According to the Guardian, it was an edited version of his July memo announcing $400m in aid to Ukraine, and the White House confirmed to NBC that the document was fake.

Four moving parts, not one

Reporting across the dossier points to a recurring structure. First, the raw material: a genuine image, video, document or news event. Second, the frame: a caption, a name, a nationality, a religion. Third, the distribution: large accounts, paid verification, and recommendation systems that surface the false frame to people who were not looking for it. Fourth, the lag: the correction arrives after the narrative has already hardened.

The Southport case in July 2024 shows the lag clearly. Within hours of a knife attack that killed three young girls, false claims about the attacker's name, religion and migration status spread online. Hannah Rose, a hate and extremism analyst at the Institute for Strategic Dialogue, told CNBC that a post on X falsely named the perpetrator as "Ali al-Shakati," rumored to be a migrant of Muslim faith. By 3pm the following day, she said, the false name had over 30,000 mentions on X alone. Police debunked the claims the day after they first emerged, saying the suspect was born in Britain, but the narrative had already gained traction. Far-right groups organised anti-migrant and anti-Islam protests, which escalated into days of riots and attacks on mosques, immigration centres and hotels housing asylum seekers.

Rose also described how platform machinery amplified the false name rather than merely hosting it. When users searched "Southport" on TikTok, she said, the "Others Searched For" section promoted the false name of the attacker, including 8 hours after police confirmed the information was incorrect. ISD's analysis found algorithms worked similarly on X, where the incorrect name featured as a trending topic.

Why false frames travel

Joe Ondrak, research and tech lead for the UK at Logically, told CNBC the Southport claims acted as "catnip" to audiences already primed by years of anti-migration rhetoric. "It's not a case of this false claim goes out and then, you know, it's believed by everyone," he said. The reports instead work as "a way to rationalize and reinforce pre-existing prejudice and bias and speculation before any sort of established truth could get out there."

That mechanism is not specific to one country or one conflict. The Guardian's October 2023 round-up noted that Russia has been a longstanding culprit for spreading disinformation on X and appeared to be capitalising on the Israel-Hamas conflict. The former Russian president Dmitry Medvedev, deputy chair of the Russian Security Council, tweeted that Nato weapons handed to Ukraine were now being used against Israel. A video apparently showing Hamas thanking Ukraine for weapons was posted by an X account linked to the Wagner mercenary group, viewed more than 300,000 times and amplified by far-right accounts in the US. Eliot Higgins of Bellingcat flagged a fake BBC video claiming to show a Bellingcat investigation into Ukrainian weapons smuggling to Hamas, pushed by Russian social media users; Higgins said it was unclear whether this was a Russian government campaign or a grassroots effort.

Scale matters here. A CNN review published on 13 November 2023 found the Chinese government had built the world's largest known online disinformation operation and was using it to harass US residents, politicians and businesses, at times threatening targets with violence. The network, known as "Spamouflage" or "Dragonbridge," has hundreds of thousands of accounts across major platforms. Meta said in August 2023 it had taken down a cluster of nearly 8,000 accounts attributed to the group in the second quarter of that year alone. Google told CNN it had shut down more than 100,000 associated accounts in recent years, and X had blocked hundreds of thousands of China "state-backed" or "state-linked" accounts. The US State Department described the tactics as part of a broader multi-billion-dollar effort to shape the world's information environment and silence critics of Beijing.

The moderation gap

Platforms are not passive pipes, and the dossier contains several attempts to make that point concrete. The Guardian reported that Elon Musk dissolved X's Trust and Safety Council after his 2022 takeover, that he had fired two heads of trust and safety in the past year, and that the company said it had plans to reorganise the team but these remained unclear. X introduced Community Notes, a crowdsourced moderation programme that largely places the onus on users to correct facts instead of employed content moderators. On Tuesday 10 October 2023, the European Commission sent Musk a letter warning him over alleged disinformation on X about the Hamas attack, including fake news and "repurposed old images."

Independent research has tracked the direction of travel. The Union of Concerned Scientists, writing on 20 September 2023, cited a Center for Countering Digital Hate finding that X "continues to host nearly 86% of a set of 300 hateful posts after a week since we reported them." Researcher Abbie Richards found climate denial tweets averaged 30,000 per week between December 2021 and July 2022, and nearly tripled to 110,000 per week after July 2022. A University of Southern California researcher found the average daily hate speech of hateful users nearly doubled in the first six months of Musk's ownership. The UCS post also cited an approximately 60% drop in advertising revenue.

None of this is uncontested terrain. The same UCS piece noted that most climate disinformation on X comes from relatively few users: in the run-up to and during COP27, tweets and quote tweets from just 16 accounts amassed 507,000 likes and retweets. Small numbers of determined accounts can produce the appearance of a mass movement, which is precisely what makes the network model hard to police.

What the public says it wants

A Unesco-commissioned Ipsos survey of 8,000 people across 16 countries due to hold national elections in 2024, reported by the Guardian on 7 November 2023, found 85% worried about the influence of online disinformation and 87% said it had already had a major impact on national political life. Social media was the main news source for 56% of internet users, ahead of TV at 44% and media sites at 29%, even though trust in social media news was 50% against 66% for television. Across all 16 countries, 68% said social media was where fake news was most widespread.

Unesco's director general, Audrey Azoulay, told reporters that false information and hate speech online, accelerated and amplified by platforms, posed "major risks to social cohesion, peace and stability," and that regulation was urgently needed "to protect access to information... while at the same time protecting freedom of expression and human rights." Unesco said its governance blueprint drew on more than 10,000 contributions from 134 countries over 18 months, and that more than 50 countries were already regulating social media, often not in line with international free speech and human rights norms.

Regulation is not a settled answer either. Senator Sheldon Whitehouse has argued for repealing Section 230 of the Communications Decency Act and replacing it with a narrower version that would force "disclosure of disinformation funders," according to Reclaim The Net's write-up of his book chapter and Twitter thread. Whitehouse argued that "the threat of legal liability, for example, could make Facebook and other companies more likely to adopt measures that stop the spread of misinformation even if they also reduce user engagement, as failing to act would carry its own financial risks." Critics of that approach, as Reclaim The Net notes, suspect most disinformation would simply become speech protected by the First Amendment.

The UK offers a live test. It has an Online Safety Act meant to fight hate speech, but CNBC reported it only comes into effect early next year and may not be enough to guard against some forms of disinformation. On Wednesday 7 August 2024, the UK media regulator Ofcom wrote to social media platforms saying they should not wait for the new law to come into force.

Amnesty Tech's deputy director, Pat de Brún, told the Guardian that platforms like Meta and X have clear responsibilities under international human rights standards, and that these responsibilities are heightened in times of crisis and conflict. "Social media firms are responsible for identifying and responding effectively to risks and taking effective measures to limit the spread of harmful content," he said, adding that big tech companies have "all too often" failed to step up in such emergencies.