At the end of September, OpenAI, the company behind ChatGPT, released an anticipated update to its text-to-video model and social media application, Sora. Sora is an AI-powered video generation model, and the app uses that model to allow users to create and share videos, including by creating “cameos” of themselves and others. Immediately after its release, however, Sora 2 faced intense scrutiny from industry experts for its potential in fostering an environment of disinformation and online abuse. One of these experts, Professor Hany Farid of the UC Berkeley School of Information, is particularly worried about the erosion of digital trust, “the liar’s dividend,” and a new disturbing trend of realistic artificially-generated videos with little to no regulatory guardrails.
Unlike other text-to-video models, Sora is able to train itself on a wider range of visual content due to its approach in representing data as tinier “patches” rather than as chunks of text. These smaller units of data allow for various aspect ratios, video durations, and resolutions to be analyzed and later recreated when users input text prompts. Additionally, Sora can better mimic the laws of physics in these videos, creating longer videos that look and seem increasingly lifelike.
As a result, the internet quickly saw Sora-generated videos of ballot fraud, immigration arrests, protests, and crimes appear on the app after its release. A writer from the Washington Post found that simply giving the app access to share his face with those he chose allowed his friends to create videos of him being arrested, confessing to a weird habit, and more — all without his approval of the content being made.
“Anybody with a keyboard and internet connection will be able to create a video of anybody saying or doing anything they want,” Farid said in a statement to news outlets. “I worry about it for our democracy.”
In fact, AI-generated content has long begun integrating itself into our political and social institutions — even before the release of Sora. Online users saw an overwhelming amount of deepfaked photos of Pope Francis, Donald Trump, Kamala Harris, and many more prominent figures throughout the past few years. Just last week, NPR reported that the Republican Party ran an attack advertisement with an AI deepfake of Senator Chuck Schumer grinning about the government shutdown.
The pervasiveness of digitally-altered content has therefore led to a phenomenon called the liar’s dividend. The phenomenon states that the presence of deepfakes and AI-generated content has allowed for bad actors to cast doubts on authentic content and declare it fake. As a result, what might have been evidence of genuine misconduct can be dismissed as a product of AI.
“It is part of a continuum of AI models being able to create images and video that are passing through the uncanny valley,” Farid added in an interview with 404 Media. “It seems likely that the same types of abuses we’ve seen in the past will be supercharged by these new powerful tools.”
In an attempt to curb these abuses, OpenAI has attempted to implement watermarks in the corners of its videos and banned users from creating content with world leaders and controversial figures in them. However, these methods have proven to be anything but foolproof, with websites quickly offering Sora watermark removal services and users using alternative prompt phrases to get around the ban.
“I’d like to know what OpenAI is doing to respond to how people are finding ways around their safeguards,” Farid said. “Will they adapt and strengthen their guardrails? Will they ban users from their platforms? If they are not aggressive here, then this is going to end badly for us all.”
