Pass@k
Did any of k attempts succeed?
A metric from code generation: the probability that at least one of k sampled solutions is correct. Measures capability with retries rather than single-shot accuracy.
Report it alongside pass@1. A model with high pass@10 and low pass@1 is a generator you must select from, not an oracle you can trust blindly.
Two cautions. It measures whether any attempt succeeded, not whether the system is reliable, so it flatters models that are intermittently right. And it is not the same as the chance of k correct in a row, which is the number that matters when a pipeline needs every step to hold.
Where pass@k is the honest metric
An agent generates a fix and can run the tests to choose among candidates.
With selection available, capability is the right thing to measure, and pass@k reflects it. Where a human receives the first answer with no filtering, report single-shot accuracy instead — the number is lower and much more meaningful.
- paperEvaluating large language models trained on code ↗
HumanEval, and the formal pass@k estimator: the chance that at least one of k samples passes the tests. Not the same as passing k times in a row.
- blogDemystifying evals for AI agents ↗
Gives working definitions for harness, task, trial, grader, transcript and outcome, and argues you must evaluate the model and harness together and judge final environment state, not just the last message.
Benchmark
A shared public test used to compare models.
Evaluation · Evals
How you know the system is actually working.
Sampling
How the model picks one token out of the distribution.
Automated Check
A machine verdict the agent cannot talk its way past.