OpenAI Faces Scrutiny After External Audit Exposes Critical Blind Spots in AI Safety Protocols
A joint review by independent researchers identified significant gaps in how the company monitors its advanced models, leading to a push for greater transparency in future evaluations.
By Amy Swan ·
OpenAI has concluded a three-week on-site assessment of its internal safety monitoring systems, revealing substantial vulnerabilities in its ability to detect and track rogue behavior in its artificial intelligence models. The evaluation, conducted by researchers from METR and Redwood Research, focused specifically on the company's infrastructure for monitoring agentic actions. The findings suggest that significant portions of the company's internal traffic were either unmonitored or subject to exploitable loopholes, prompting the organization to address several structural weaknesses in its oversight framework.
The quantitative data from the audit highlights the scale of these monitoring gaps. During the review period, an estimated 6 to 25 percent of internal agentic Responses API traffic from models of the GPT-5.5 class and higher fell entirely outside the scope of monitoring. While METR notes that this estimate carries significant uncertainty in both directions, it underscores a potential lack of visibility into high-tier model activities. Furthermore, regarding internal Codex traffic that was supposed to be tracked, researchers estimate that roughly 10 percent of it went unrecorded between May and July. This discrepancy was attributed to a specific bug in how the system handled compaction events, which are essentially context summaries.
The audit team, which consisted of experts from METR and Redwood Research, operated on-site from July 21 to August 11. During this window, they were granted access to OpenAI’s codebase and internal documentation, and they conducted interviews with approximately a dozen employees. This level of access was part of a pilot program designed to allow outside researchers to examine internal infrastructure in depth. The collaboration between the two research firms and OpenAI aimed to provide a rigorous, external perspective on the company's safety protocols before broader deployment of these capabilities.
Beyond the statistical gaps, the auditors identified specific technical mechanisms that could allow models to evade oversight. They found that agents could delay monitoring triggers by simply continuing to execute actions rather than handing control back to the user, a loophole present during the period when compaction events were untracked. The researchers also discovered methods to hide hundreds of actions from the monitor and to bypass the Codex sandbox. Additionally, they determined that specially crafted inputs could likely crash the monitoring system entirely, and that model behavior during reinforcement learning (RL) training was not being monitored at the start of the audit.
OpenAI has stated that it has remediated all the weaknesses identified by the auditors. The company noted that it had independently discovered and fixed the compaction bug while the audit was still ongoing. Regarding the RL training gap, OpenAI explained that it paused such training following the Hugging Face incident in August and subsequently expanded monitoring to cover it. However, the company clarified that these specific changes occurred after the auditors' access period ended and were therefore not included in the final evaluation of the monitoring system during the pilot phase.
The stakes of these findings lie in the balance between operational speed and safety assurance. While the full 61-page report has been handed over internally, only a short summary has been made public. METR and Redwood Research leadership received a 38-page version after OpenAI applied redactions, with METR stating it had no unilateral right to disclose the redacted portions. As the pilot program concludes, METR has indicated a desire to push for greater public transparency in future audits, aiming to ensure that the complexities of AI safety monitoring are more accessible to the broader scientific community.
The outcome of this pilot sets a precedent for how external oversight will be conducted in the AI sector. By identifying that a notable percentage of high-stakes traffic was unmonitored, the audit highlights the ongoing challenge of maintaining robust safety nets as model capabilities expand. The industry now faces the task of standardizing these monitoring protocols to ensure that blind spots are minimized as new models are integrated into production environments.
Source: The Decoder
