New math benchmark reveals AI models confidently solve problems that have no solution

The benchmark's core design insight is that unsolvable problems aren't a stress test of raw math ability but a probe of metacognition: does the model know what it doesn't know? Most evaluations never ask that question, so the industry has been optimizing for answer quality while leaving answer-refusal capacity almost entirely unmeasured.

This is largely disconnected from recent activity in our archive, as we have no prior coverage to anchor it to. It does, however, belong to a growing body of work questioning whether capability benchmarks and reliability benchmarks are measuring the same thing. The distinction matters because a model that scores well on solvable problems but fabricates solutions to broken ones is not simply 'less capable,' it is miscalibrated in a way that standard leaderboard rankings obscure entirely. That miscalibration is precisely what makes deployment in legal, financial, or scientific contexts risky.

Watch whether any major lab responds to SOOHAK by publishing refusal-rate data alongside accuracy scores in their next model card. If that becomes a reporting norm within the next two release cycles, the benchmark will have done real work; if labs stay silent, the gap it exposes will persist quietly.