AgentBench

An agent is a model plus a harness: the loop that picks tools, feeds results back, and decides what stays in context. That changes the serving problem. A single prompt is one pass in and one stream of tokens out. An agent is dozens of calls against a context that grows with every tool result, each one re-sending most of what came before.

Which makes inference a memory problem before a compute one. The model's working state for a conversation sits in the KV cache, that cache grows with context length and with how many agents run at once, and generation speed is capped by how fast memory can be read. So the levers that decide what an agent costs are cache hit rate on the stretch of context that repeats, whether those cached tokens are ones the model actually uses, and batching that holds capacity for an agent gone quiet during a tool call instead of making it queue on return.

Almost nobody measures them. Agents are moving into enterprise workflows where token spend scales with turns rather than with tasks completed, which is why a growing number of startups are moving off hosted endpoints onto their own stacks with smaller fine-tuned models. The moment they do, they own the memory and scheduling problem, and the question they cannot answer is which model architecture and which serving configuration actually set what a task costs them.

AgentBench is a research study built for that question. It treats model architecture and serving configuration as the two variables that set cost per task, and measures both under real agent traffic rather than single-prompt benchmarks. The harness and task set stay fixed while the model varies, so differences land in serving behavior rather than in scaffolding. Dense against mixture of experts, different attention designs, quantization levels, specialized against general models: each changes how much memory the cache consumes and how fast tokens come back.

The instrumentation is the product. Six measurements, each chosen because it answers a decision someone has to make.

Wall clock stays separate rather than folded into a cost figure, since collapsing time into dollars hides the tradeoff being studied.

The harness is custom. It keeps SWE-agent's container setup and the file-editing and shell interface the agent works through, and replaces the orchestration and measurement layer so every call to the model is visible. Its first version produced clean numbers on a workload that was not real: the agent never solved a task, and a failing agent still puts load on the GPU, just not load that resembles an agent doing its job. Task solvability now gets verified before any GPU time is spent.

Phase 1 is complete. Roughly 70 runs over hosted APIs, stratified by difficulty with deliberate overlap, comparing a frontier model against an open-weight model at both ends of the difficulty range. Phase 2 replays the same tasks with the same frozen harness on open-weight models we host ourselves, served through SGLang, where the memory and scheduling behavior is finally visible. Phase 3 holds one model constant and builds a workload-aware serving layer across the two main open-source serving engines, vLLM and SGLang, with a channel that tells the harness what is already cached, so batching and memory can be planned against the shape of the task instead of guessed at.

The unit that matters is cost per resolved task, not cost per token, and the two can point in opposite directions. A cheaper model that needs three times the turns and carries a longer context can take up more of a GPU than the expensive one it replaced.

Phase 1 established accuracy, cost and serving baselines and confirmed the shape worth studying: the expensive model's advantage concentrates on hard tasks and thins out on routine ones. For a team choosing a serving stack, that is the whole question. Not which model is best, but which model class is sufficient for the traffic they actually run, and how many concurrent agents that choice fits on hardware they already pay for.

The study is framed as characterization rather than optimization. It reports what agent traffic does to a GPU and leaves the fix to whoever reads the traces. Target venue is MLSys, with the trace set published as a reusable artifact.