Benchmarks are built to die – they’re built to get saturated and be replaced by newer, harder variants. During its lifetime, a benchmark serves several purposes: it both gives AI developers a metric for R&D progress and conveys model capabilities to the general public.

These days, many benchmarks die young, for good and bad. On the one hand, AI capabilities are advancing quickly – by and large, I think this is good. On the other hand, producing new high-quality benchmarks is costly. This is problematic: all benchmarks are flawed, so the best indicator of frontier model capabilities might be taking an average over multiple good-enough benchmarks, as done in the ECI. The more unsaturated benchmarks, the better.

Another strategy, suggested by Greg Burham on Epoch After Hours, is applying natural human benchmarks to AIs. Yes, such benchmarks exist – in fact, there are plenty of them, as I’ll describe soon.

The idea is simple. Take your favourite human benchmark, have an AI ‘compete’ under the same conditions as humans, and apply the exact same grading scheme to human- and AI-generated outputs. For fair comparison, judges need to be oblivious to what’s AI-generated.

Human benchmarks #

One interesting class of natural human benchmarks are competitions, the typical examples being board game tournaments and science olympiads. Any merit-based competition for humans can serve as a benchmark – just have the AI submit a competition entry, and check whether the AI would rank in the top 99:th percentile.

Other examples of natural human benchmarks – perhaps more relevant for current models, which crush humans in board games and STEM olympiads – are best paper awards, forecasting tournaments, auction results or P&L for traders. Harder benchmark: can an AI write a Pulitzer winner? Each profession has its own ranking mechanism for humans, and if concealing AI authorship is technically doable, this mechanism can be used for AIs.

In domains where AIs still perform poorly, you can also do Turing tests. Could an AI pass for a remote worker? This is much easier than winning a prize (passing for a clever human), and is also a helpful tracker of AI progress.

Other considerations #

There are several advantages with applying human benchmarks to AI.

Most notably, most human competitions have carefully elaborated grading rubrics, with entries being double-judged by experts. By contrast, AI benchmarks usually rely on cheap, automated grading, making benchmark results less reliable.

Another advantage of using human benchmarks is that their results are easily interpretable. A professional knows how much it takes to get their licence, or to win the biggest award in their domain. While a flaw with AI benchmarks is that they don’t translate well into productivity gains – what does it even mean for Claude Opus 4.8 getting 47.1% on GSO-bench? – human benchmarks provide an unambiguous lower bound: if an AI outperforms a remote worker, you can cut costs by the human salary minus the model subscription.

There are disadvantages too, of course. You can only run human benchmarks when the human benchmarking occurs, and many competitions only run annually. Moreover, tricking human experts into judging AI output as if it were human might be ethically dubious; applying human benchmarks to AIs will require careful experiment design, so it doesn’t come off as an offensive prank. Furthermore, marking entries in human competitions is generally labor-intensive.

Some ideas #

I’ll end by listing a few experiments I’d love to see carried out responsibly. Not all these are novel ideas; I’ve included them because I’m curious about follow-up work.

• Researcher Turing test. This was an idea I actually suggested implementing back in March. Have AIs write ML conference papers and check whether humans can discriminate between human- and AI-written papers. This year, we’ve seen plenty of attempts at creating autonomous research agents¹, so a researcher Turing test would definitely be timely.

• Trader KPIs. Ask an AI to perform the job of a human trader, check who earns the most. There has already been some work investigating whether AIs can beat the market, both serious (here, here and here) and less serious (Google ‘make money with Claude’). I’d be particularly interested in seeing work comparing the performance of the latest models, like Claude Fable and GPT-5.6 Sol, with human baselines.
• Startup pitch. Have an AI deliver a pitch. Concretely, the AI system needs to come up with a convincing idea, put together a slide deck and give a killer presentation (video camera off, I guess). Would the AI make it into any startup incubators?

Finally, I’ll mention one Turing test AI labs could run right now.

• Sports commentator Turing test. Can an AI system perform the job of a sports commentator? This requires quickly parsing multimodal input, commenting on game developments and, not the least, coming up with non-corny jokes.

As stressed in this essay, sports commentary is challenging. Seeing an AI on the level of a legendary sports commentator would definitely shorten my ASI timelines.

This post was inspired by discussions with Terry Jingchen Zhang and Viktor Lundqvist.