Agent Benchmark

A benchmark designed for agents rather than chat models: instead of asking questions with known answers, it gives the model a real environment, a terminal, a repository, a browser, and scores whether it completes tasks. Terminal-Bench, ProgramBench, and DeepSWE are the current examples: they measure multi-step work like running commands, editing programs, and finishing automation jobs. They are why recent model updates report their biggest gains on agent tasks rather than on question answering.

Agent benchmarks are harder to game and harder to run than Q&A benchmarks. A task can pass by accident, fail on environment quirks, or take minutes of tool calls, so results depend on the harness and the agent scaffold as much as on the model. Two models compared on different scaffolds are not really compared at all, which is why the harness name belongs next to every score you quote.

The October 2026 DeepSeek V4.1-Flash-1017 update is a good illustration of what these benchmarks reveal: vendor numbers showed Terminal-Bench 4.0 jumping from 31.2 to 42.3 and ProgramBench from 20.3 to 29.1 against the September checkpoint, while question-answering scores moved far less. That pattern, big agent-task gains and small Q&A gains, is the signature of post-training aimed at tool use.

Comparing agent benchmark scores across different harnesses or scaffolds. The scaffold is part of the measurement, and a score without it is not comparable.

• reportTerminal-Bench 4.0 Benchmark ↗Artificial Analysis · 2026

  Independent evaluation of Terminal-Bench 4.0: 66 terminal tasks run three times, scored on end-to-end completion against hidden tests rather than on answering questions.

• reportDeepSeek Just Released V4.1-Flash-1017, Scores 77.0 on DeepSWE, Opens Search API ↗MarkTechPost · 2026

  Release details for the 1017 checkpoint, including the vendor-reported 77.0 on DeepSWE v1.1 and the MIT-licensed weights on Hugging Face. Benchmark figures are the vendor’s, not independently reproduced here.