The British newspaper The Mail on Sunday has revealed security loopholes in TikTok's settings that it says allow AI-generated violent and disturbing videos to reach children's accounts, despite the "Restricted Mode" feature meant to block inappropriate content being enabled.
اضافة اعلان
According to the newspaper's investigation, some of the clips include scenes simulating violence, torture, and sexual assault, at a time when millions of families rely on the app's parental control tools to protect their children from this type of content.
Loopholes in "Restricted Mode"
TikTok's "Restricted Mode" is designed to reduce the appearance of content containing profanity, weapons, sexual innuendo, drugs, violent scenes, and material meant for adults.
However, the newspaper said that, for the purposes of its investigation, it created an account with a stated age of 14 and enabled Restricted Mode on it, before using it to watch content related to children's TV shows.
Despite this, the investigation found that the account subsequently received clips containing material falling under several categories the safety settings are supposed to restrict.
The newspaper suggests part of the problem stems from the nature of the platform's rules: while TikTok bans "realistic violence" and certain types of sexual content, cartoonish or AI-generated clips may be able to slip past some of the classification mechanisms.
15,000 AI-Generated Clips
The investigation shone a light on a startup called "Director AI," which provides AI-based tools for producing short video clips.
According to the newspaper, the company's tools were used to produce more than 15,000 clips, some of which were designed in ways intended to keep users watching.
It noted that one clip, posted through an account within a network comprising dozens of accounts, surpassed 120 million views, and featured a cartoon scene of a doctor performing a violent surgical procedure aimed at transforming a patient's features to resemble reality TV star Kim Kardashian.
The investigation also found other clips built around revenge, violence, and abuse storylines, presented in a fast-paced, dramatic, and sensational format.
Director AI is run by Ben Chiang, a former executive at Snapchat, who has previously publicly described this type of content as low-quality, AI-generated material.
After the newspaper contacted the relevant parties, the app's age rating was changed ahead of its next update.
"Building a Fortune at Children's Expense"
Dr. Catherine Knibbs, a specialist in online safety and psychological therapy, warned of the potential impact of this kind of material on children.
She said the issue, in her view, is tied to generating profit at the expense of minors' safety, arguing that content designed in this way can cause psychological harm to children.
Laura Trott, education spokesperson for the UK's shadow government, also commented, saying there should be no tolerance for exposing children online to content that would be unacceptable for them to encounter in the real world, calling for stricter measures to protect minors on social media platforms.
The Company Responds
For his part, Chiang rejected holding Director AI responsible for the publication of all clips produced using its tools.
He explained that the company provides AI video-production technology, while users themselves make the editorial decisions about the content and choose which platforms to publish it on, including TikTok.
He added that the company does not own the accounts publishing these clips and does not pay their owners to promote its app.
He noted that some material produced using the company's tools may deal with fictional themes intended for adults, but that responsibility for determining content eligibility and applying "Restricted Mode" filters lies with the platforms hosting it.
TikTok Removes the Clips
A TikTok spokesperson, for their part, affirmed that the platform's community guidelines prohibit content that promotes violence or glorifies sexual assault and exploitation, noting that any clip found to violate these standards is removed.
After the issue came to light, the app removed the clips flagged by the investigation; however, the newspaper said the accounts that had posted them remained active and continued to post other material it considered inappropriate.
