Cohere gave a GPT-5.6 coding agent 16 cheap helpers, and they only paid off as planners

The job was rebuilding a program nobody would hand you the source for. On ProgramBench, an agent gets a black-box executable and its documentation, then has to write a program that behaves the same way. The 200 tasks run from CMatrix and ripgrep up to FFmpeg and the PHP interpreter. When ProgramBench’s own authors tested nine models, “none fully resolve any task.”

Six researchers at Cohere Labs and Cohere used it to ask a budget question in their paper on arXiv: if one agent isn’t enough, where should the next dollar go? Their lead agent gets a delegate_agent tool, and each worker it spawns sees only its own subtask. Then they swapped models, stages and head counts one at a time.

Teams won, at a price. They beat a solo agent but cost three to seven times as much, the paper says.

Then came the cheap seats. GPT-5.6 Luna costs $0.20 per million input tokens to Sol’s $4.00, so the team kept a Sol lead and filled every worker slot with Luna. Pointed at planning, the bargain helpers worked. Pointed at writing code, “Luna workers reduce the team’s reward below that of the Sol solo agent while increasing its cost,” the authors write. They call that substitution “ruinous.”

Scaled up, the gap got silly. A Sol lead with 16 Luna planners came within 1.4 points of an all-Sol five-agent team at 70% lower cost. A Sol lead with cheap planners “recovers 47.5% of the reward gain of an all-strong team for only 13.9% of its additional cost,” per the full paper.

Steal this

When you split an agent job across models, give the cheap ones the scouting (run the binary, read the docs, list the edge cases) and keep your best model on the keyboard. Kind of like hiring a cheap surveyor to measure the lot and keeping your best carpenter on the framing.
Before you add a fifth implementer, check whether a fourth helped. In this paper, implementers and verifiers stopped improving after four workers, while planners kept adding reward all the way to 16. This is like adding a fifth cook to a kitchen with four burners.
If your harness keeps delegating after the first file is written, count those calls against what they bought. Here, 830 extra delegation calls bought nothing, kind of like a meeting that keeps going after the decision got made.
Spend the model budget on the lead. A Terra lead with Luna workers still trailed a Sol agent working alone by 10.3%. Kind of like a road trip: cheap snacks are fine, but the driver should know the way.

The timing tells you where the value was. With planning-only instructions, the lead fired all 674 of its delegation calls “in a single burst before any code is written.” The heavy plan-implement-verify setup made 794 calls before the first edit, then kept going, “adding an additional 830 calls that yield no additional reward.” Planning alone matched its reward at half the cost.

The researchers looked for the usual suspects behind the implementer plateau and didn’t find them. Implementers almost never wrote the same file, 0.1% of their files. Each planner ran the reference program about seven times no matter how big the team got, so more planners meant more evidence for the lead. More implementers just chopped one plan into smaller pieces, and the Luna workers’ share of source edits climbed from 54% to 85% while the lead’s final check stayed at five or six runs of the reference.

(Also buried in the appendix: GPT-5.6 solo agents wrote their rebuild in Python for 86% to 92% of tasks, whatever the original program was written in.)

Why it matters

The bill. Every agent framework now offers to spin up a crowd. This paper puts a number on the crowd: three to seven times the cost for a team, and a way to get within 1.4 points of the all-Sol team at 30% of its price if the cheap models do the reading and leave the writing alone.

More workers, same score. Raising the worker cap from five to twenty changed reward by 1.6 points or less either way and raised cost per task 5% to 17% in seven of ten settings.

Chatter doesn’t help either. When every agent got a messaging tool, 84% of messages went to the lead, workers barely talked to each other, and reward didn’t rise. The channel “mostly carries status reports to a lead that already receives each worker’s outcome.”

It flips with the task. On Commit0-Lite, which ships the repo’s file layout and function stubs, implementation was the best place for workers and planning was never clearly best. The paper’s read: when the skeleton is already the plan, there’s little left to discover.

The pattern points somewhere uncomfortable for anyone selling agent swarms by the seat. If the lead “determines the ceiling that a team can reach, while the workers determine how cheaply it is approached,” the money moves to one very good model at the top and a crowd of cheap readers around it, and the cheap coders mostly create integration work.

There are asterisks. The 16-worker teams ran once each, and the authors call their best configurations “empirical operating points rather than guaranteed optima.” Commit0’s flip means nobody gets to copy this recipe without checking what their own task already hands the agent.

The teams here topped out at 16 workers, and the paper closes by asking how teams of “hundreds, thousands, or even millions of agents” should be run. Its authors also note they didn’t test any of this for safety, and that errors “can propagate through delegation.” Somebody will have to audit that crowd before it gets bigger.

More Software