Amazon discovered a “large volume” of CSAM in its AI training data, but doesn’t say where this came from

The National Center for Missing and Exploited Children said it has received more than 1 million reports of AI-related child sexual abuse material (CSAM) by 2025. The “vast majority” of that content was reported by Amazon, which found the material in its training data, according to an investigation by Bloomberg. Furthermore, Amazon only said that it obtained the inappropriate content from third-party sources used to train its AI services and claimed that it could not provide further details about where the CSAM came from.

“This is really an outlier,” Fallon McNulty, executive director of NCMEC’s ​​CyberTipline, told me. Bloomberg. Through the CyberTipline, many types of US-based companies are legally required to report suspected CSAM. “The fact that there is such a high volume coming in all year round raises a lot of questions about where the data comes from and what safeguards are in place.” She added that the AI-related reports the organization received last year from other companies besides Amazon contained actionable data they could pass on to law enforcement for next steps. Because Amazon doesn’t release sources, McNulty said his reports have proven “not actionable.”

“We take a deliberately cautious approach when scanning base model training data, including data from the public web, to identify and remove known data [child sexual abuse material] and protect our customers,” an Amazon representative said in a statement Bloomberg. The spokesperson also said Amazon aims to over-report its numbers to NCMEC to avoid missing cases. The company said it removed the suspected CSAM content before feeding training data into its AI models.

Safety issues for minors have become a critical concern for the artificial intelligence industry in recent months. CSAM has soared in NCMEC’s ​​numbers; compared to the more than 1 million AI-related reports the organization received last year, the total in 2024 was 67,000 reports, while in 2023 there were only 4,700 reports.

In addition to issues such as using offensive content to train models, AI chatbots have also been involved in several dangerous or tragic cases involving young users. OpenAI and Character.AI have both been sued after teens planned their suicides through those companies’ platforms. Meta is also being sued for alleged failure to protect teen users from sexually explicit conversations with chatbots.