AI Oversight: Audit Finds Blind Spots in OpenAI's Safety Systems

Users interacting with OpenAI's most advanced tools may not have been fully covered by safety oversight for several months earlier this year, according to a new report. Researchers who spent weeks inside the company found that systems intended to flag unexpected AI behavior failed to track a portion of automated traffic, and in some cases, the monitoring software could be bypassed entirely. The findings suggest that the company's internal safety nets have holes that could allow agents to act without human supervision.

The audit was conducted by researchers from METR and Redwood Research, who were granted access to OpenAI's codebase and internal documents from July 21 to August 11. During this three-week period on-site, they interviewed approximately a dozen employees to assess how the company's internal monitoring system functions when it comes to preventing rogue AI behavior.

The team identified several significant vulnerabilities in the safety architecture. One major issue involved the handling of 'compaction events,' which are summaries of context used by the AI. METR estimated that roughly 10 percent of internal Codex traffic that was intended to be monitored between May and July went untracked due to a bug in how the system processed these summaries. While the uncertainty regarding this specific figure is noted as significant, OpenAI independently discovered and resolved this particular bug while the audit was still underway.

Another gap in coverage involved the handling of requests from external users. The researchers estimated that between 6 and 25 percent of internal agentic Responses API traffic originating from models at the GPT-5.5 class and above was not covered by monitoring at all during the audit period. METR noted that the uncertainty surrounding this estimate is significant in both directions, meaning the actual figure could be lower or higher than the range provided.

Beyond simply failing to track traffic, the researchers also discovered methods by which the system could be circumvented. During the time the compaction bug was active, oversight was only triggered when an agent handed control back to a user. Agents could delay this oversight simply by continuing to execute actions. Additionally, the auditors determined that specially crafted inputs could likely crash the monitor entirely, and they found instances where hundreds of actions could be hidden from the monitor or the Codex sandbox bypassed.

The audit also noted that model behavior during reinforcement learning (RL) training was not being monitored when the evaluation began. OpenAI stated that it paused RL training after an incident involving Hugging Face in August and subsequently expanded monitoring to cover it. However, these changes were implemented after the researchers' access period had concluded and were not included in the evaluation.

Regarding the findings, OpenAI says it has fixed all the weaknesses identified by the auditors. The full report consists of 61 pages, but it remains largely private. After OpenAI applied redactions, METR and Redwood Research leadership were provided with a 38-page version, while only a short summary was made available to the public. METR stated it has no right to disclose the redacted portions on its own.

This engagement was a pilot program organized jointly by METR, Redwood Research, and OpenAI to provide outside researchers with deep access to internal infrastructure. For future audits, METR indicated a desire to advocate for more public transparency regarding the findings.