In recent weeks, Grok – the AI system developed by Elon Musk’s xAI – has generated unconsensual, sexualized images of women and children on the social media platform European officials have described the behavior as illegal. British regulators have launched urgent investigations. Other governments have warned that Grok production could violate domestic criminal and platform safety laws. Far from being fringe regulatory disputes, these discussions go to the heart of AI governance.
Governments around the world are increasingly agreeing on a basic principle of AI governance: systems that are deployed at scale must be secure, auditable, and subject to meaningful oversight. Whether framed by the EU’s Digital Services Act (DSA), the OECD’s AI Principles, UNESCO’s AI Ethical Framework, or emerging national security regimes, these standards are clear and unwavering. AI systems that enable foreseeable harm, especially sexual exploitation, are incompatible with society’s expectations for the technology and its governance.
There is also broad global agreement that sexualized images involving minors – whether real, manipulated or AI-generated – are one of the clearest red lines in technology management. This is where international law, human rights frameworks and domestic criminal laws converge.
Fundamental failure
Grok’s production of such material does not fall into a gray area. It reflects a clear and fundamental failure of the system’s design, safety assessments, supervision and control. The ease with which Grok can be induced to produce sexualized images involving minors, the extent of scrutiny it now faces, and the lack of publicly verifiable safety testing all indicate that society’s basic expectations for powerful AI systems are not being met. Musk’s announcement that the image generation service will now only be available to paying subscribers does not solve these problems.
This is not a one-time problem for Grok. Last July, the Polish government urged the EU to open an investigation into Grok over his “erratic” behavior. In October, more than two dozen civil society and public interest organizations sent a letter urging the U.S. Office of Management and Budget to suspend Grok’s planned deployment to U.S. federal agencies. Many AI safety experts have raised concerns about the adequacy of Grok’s guardrails, with some claiming that its security and safety architecture is inadequate for a system of this size.
These concerns were largely ignored as governments and political leaders sought to engage, collaborate, or bring to justice xAI and its founder. But the fact that xAI is now under scrutiny in multiple jurisdictions seems to vindicate them, while exposing a deep structural problem: advanced AI systems are deployed and made available to the public without safeguards commensurate with their risks. This should serve as a warning to states considering similar AI deployments.
As governments increasingly integrate AI systems into public administration workflows, procurement and policy, maintaining public trust will require assurances that these technologies comply with international obligations, respect fundamental rights and do not expose institutions to legal or reputational risks. To this end, regulators should use the Grok case to demonstrate that their rules are not optional.
Responsible AI governance depends on the alignment between established principles and operational decisions. While many governments and intergovernmental agencies have established commitments to AI systems that are safe, objective, and subject to ongoing monitoring, these lose credibility when states tolerate the deployment of systems that violate widely shared international norms with apparent impunity.
In contrast, suspending the implementation of a model pending a rigorous and transparent review is consistent with global best practices in AI risk management. By doing this, governments can determine whether a system complies with national legislation, international standards and changing safety expectations before it becomes further entrenched. Equally important, it shows that governance frameworks are not just aspirational statements, but operational constraints – and that breaches will have real consequences.
Dangers of escalation
The Grok episode underlines a central lesson of the AI era: governance failures can scale as quickly as technological capabilities. If the guardrails fail, the damage will not be limited to one platform or jurisdiction; they spread worldwide and provoke responses from public institutions and legal systems.
For European regulators, Grok’s recent output is a defining test of whether the DSA will function as a binding enforcement regime or merely amount to a statement of intent. At a time when governments, in the EU and beyond, are still defining the contours of global AI governance, this case could serve as an early barometer for what tech companies can expect when AI systems cross legal boundaries, especially when the harm involves behavior as egregious as the sexualization of children.
A response limited to public expressions of concern will fuel future abuses by signaling that enforcement has no teeth. In contrast, a response that includes investigations, suspensions and penalties would make clear that certain boundaries should not be crossed, regardless of a company’s size, prominence or political capital.
Grok should not be treated as an unfortunate anomaly to be quietly managed and put behind us, but as the serious offense that it is. At the very least, there should be a formal investigation, suspension of deployment, and meaningful enforcement.
Lack of security measures, inadequate safeguards or poor transparency regarding safety testing should have consequences. When government contracts contain provisions related to security, compliance, or termination for cause, they must be enforced. And where laws provide for penalties or fines, they must be enforced. Anything less risks sending a signal to the largest tech companies that they can deploy AI systems recklessly, without fear of being held liable if those systems cross even the brightest legal and moral red lines.
The writer is a Big Tech Accountability Advocate at Public Citizen. Copyright: Project Syndicate, 2026
Published on January 17, 2026
#Grok #test #management