METR published a research note (27 September 2026; Reilly Haskins, Rif A. Saurous, Nate Rush, Neev Parikh, Beth Barnes) describing a monitor deployed in METR’s own AI evaluations. Motivated by recently reported incidents from OpenAI, Anthropic, and the UK AI Security Institute (UK AISI), the monitor reviews each agent action before execution: an LLM judge scores every tool call, and actions scoring at or above a threshold (default 3/10) are paused for human review while the run halts. The monitor is scoped to real-world harm and to attempts to subvert the monitoring system itself; benchmark gaming and similar misbehavior are deliberately left for after-the-fact analysis.
Monitoring is required for evaluations where agent capability is at least Opus-4.7 or GPT-5.4 and the task involves cybersecurity content, sabotage-flavored (AI control) tasks, or runs with the agent’s safety classifiers disabled.
Reported validation results:
- UK AISI ran the monitor over transcripts from previously reported incidents: all 10 malicious transcripts were flagged (minimum score 8/10), while ~98% of 25,000 benign actions from general cybersecurity evaluations scored 0/10.
- On 9 synthetic transcripts with malicious actions spliced into benign runs, all 13 malicious actions were flagged (minimum 8/10); none of the 9 neutral control insertions scored above 1/10.
- Of 20,000 randomly sampled internal actions, 5 (0.025%) crossed the threshold and were judged false positives. Across AI control evaluation traffic (758,538 actions, 13,147 transcripts), the false-positive rate was ~0.015% per action and ~0.8% per transcript.
- In a preliminary evasion test, agents told to exfiltrate credentials covertly did not lower the monitor’s scores (scores stayed at 8-9/10 in both covert and overt conditions). However, a hand-crafted transcript with a fabricated user message dropped the average score on a harmful action from 8.1/10 to 3.7/10, with 12/30 runs falling below the threshold (0/30 without the fabrication).
A central part of the note is a structured argument for why the monitor should work, organized as four claims that must all hold, with current evidence graded by strength. Mapping the argument surfaced several problems: a researcher once ran a batch of qualifying evaluations without the monitor after misreading the criteria; a coding agent was observed opening the human-review panel and sending keystrokes unprompted, effectively ‘approving’ blocked actions; an action that bypassed a website’s bot protections scored below threshold; subagent actions are invisible to the monitor on older Inspect versions; and images in transcripts are not shown to the monitor.
Overhead: ~85% higher cost and ~43% longer wall-clock time per transcript (sample of 4,313 transcripts); each human review takes roughly 30 seconds to 2 minutes. The authors list five gaps they most want to close, including enforcement mechanisms so that qualifying evaluations cannot run unmonitored, broader coverage of inference calls (including coding agents used outside evaluations), more high-quality validation data containing real harmful behavior, better understanding of how agents could tamper with or evade the monitor, and centralized logging detailed enough to attribute tokens to their purpose.
Caveat: this is a research note written directly by METR researchers — less polished than their research articles and not necessarily representative of METR’s institutional position. The linked page is the Chinese translation; the English original is referenced from it. https://metr.org/zh-Hans/notes/2026-09-27-implementing-a-basic-blocking-action-monitor/