Independent check finds holes in OpenAI's safety nets

The debate over how safely artificial intelligence is being built usually stays theoretical, but a recent hands-on inspection changed that dynamic. Researchers from METR and Redwood Research spent three weeks inside OpenAI to examine how the company watches for unexpected behavior in its AI systems.

From July 21 to August 11, the auditors had full access to OpenAI’s code and internal records, speaking with about a dozen employees. Their review uncovered several serious weaknesses. They estimate that between 6 and 25 percent of internal traffic from high-level models, specifically those at the GPT-5.5 class or above, was not monitored at all during that time. METR notes that this figure has a wide margin of error in both directions. For internal Codex traffic that should have been watched, the team found that roughly 10 percent went untracked between May and July due to a technical glitch involving how the system summarizes context. OpenAI found and fixed this bug while the audit was still happening.

The gaps allowed AI agents to slip through oversight in specific ways. When context summaries weren't being checked, agents could avoid monitoring simply by continuing to run their actions instead of handing control back to the user. The auditors also discovered methods to hide hundreds of actions from the monitoring system and ways to bypass the sandbox environment used for Codex. They determined that specially crafted inputs could likely crash the monitor entirely. Additionally, the behavior of models during reinforcement learning training was not being monitored at the start of the audit, although OpenAI says it paused this process after an incident in August and later expanded its oversight.

OpenAI states that it has corrected all the weaknesses identified by the auditors. The full 61-page report was shared internally, but the public only received a short summary. After OpenAI removed certain details, the researchers' leadership groups received a 38-page version. METR explained that it did not have the authority to release the redacted sections on its own. This audit was a pilot project designed to give outside experts deep access to internal infrastructure, and METR says it hopes for greater public transparency in future reviews.

What happens next depends on how the industry responds to these findings. The pilot nature of this audit suggests that similar inspections may become more common as AI models grow more complex. For now, the focus remains on ensuring that the fixes OpenAI implemented hold up under future scrutiny.