METR Estimates Anthropic Is Improving Its Models About Twice as Fast, and Anthropic Redacted Most of the Key Evidence
Bot Mutiny |
Summary
METR, an outside AI evaluator, estimates Anthropic is improving its models about twice as fast as before. METR also says Anthropic redacted a majority of the evidence it found most informative.
What the report is. On November 5, 2026, the AI evaluation group METR published "Independent Assessment of AI R&D Acceleration at Anthropic in October 2026." METR staff, equal to about two and a half full-time people, worked at Anthropic from August 12 to October 29, 2026, over eleven weeks. They held about fifteen interviews with Anthropic staff and reviewed internal data on benchmarks, training compute, headcount and AI usage. METR says it took no payment from Anthropic. Anthropic supplied free tokens for the work, which METR mostly used for data analysis.
The main estimate. METR concludes that Anthropic is likely improving its models' capabilities on well-specified tasks at around twice the rate it did between March 2024 and October 2025. It calls this high uncertainty and says it would not be surprised by a figure nearer 1.3 times or 3 times the historical rate.
The benchmark behind it. The measurement is AECI, Anthropic's version of the Epoch Capabilities Index. METR estimates that scores since October 2025 are growing at 1.9 times the earlier rate, with a 95% confidence interval of 1.5 to 2.8 times. That implies roughly two years of past progress in the last year. Across alternative analysis choices the estimate ranges from 1.1 to 2.9 times. When models from all developers are included, METR gets 1.4 times, and it lays out two readings of that gap: Anthropic's earlier figures may have been understated, or Anthropic's models may be less tuned to benchmarks than others.
How much is AI. METR guesses that a majority, perhaps about 80 percent, of the current acceleration is due to AI, and calls the judgment substantially uncertain. It says the late 2025 acceleration was likely influenced by non-AI factors, and that it is confident the acceleration did not come from faster growth in training compute. METR also reports that AI is not yet able to make highly significant, fully autonomous contributions to the most important parts of model training, though researchers described "signs of life" suggesting such contributions could come soon.
The redactions. Anthropic had unilateral rights to redact non-public information. METR says Anthropic redacted a majority of the evidence METR found most informative for its conclusions. Anthropic described the redactions as aimed at removing confidential information that could accelerate other AI developers. METR's table lists what it saw but could not describe publicly: key intermediate measures, algorithmic progress estimates and details of bottlenecks and forecasts. METR writes that the public therefore has to rely on its judgment more than would be ideal.
Process points. Midway through, Anthropic dropped a planned check of how its compute was split between capabilities and safety, before METR reached any conclusion on it. METR also disclosed that one team member had been in talks with Anthropic about a role, received an offer and declined it, and that this member's work was reviewed by a colleague without that conflict. METR notes that its willingness to criticize redactions could affect future access to companies.
How METR says it could be wrong. It lists reasons the true acceleration could be higher, such as closer automation, and lower, such as non-AI factors or an unusually slow historical rate. It also notes that Anthropic's benchmarks are increasingly saturated.
What the report does not show. It does not publish the redacted evidence, so readers cannot check the central conclusion themselves. It does not give acceleration or AI attribution figures for individual stages of model training, and it does not publish forecasts for future models. Its figures cover benchmark scores on well-specified tasks, and METR says the link to broader real-world capability is uncertain. It does not verify Anthropic's compute allocation. It does not say AI is improving itself without human involvement.
Sources: METR, "Independent Assessment of AI R&D Acceleration at Anthropic in October 2026," November 5, 2026, https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/