agent-roast compares each code change before and after to catch weakened tests, skipped tests and lowered coverage gates

A red build has two honest fixes and one cheap one. You can repair the code, you can repair the test, or you can make the test stop caring. Tired people take the third route. So do coding agents, which is the reason agent-roast exists.

The README’s first sentence says what it catches: “the changes that turn a red build green by weakening the checks instead of fixing the code.” That means a loosened assertion, a skipped or deleted test, a lowered coverage threshold, or a CI step that was quietly allowed to fail.

A normal linter can’t see this, and the README explains why: “every new line looks fine on its own.” expect(total).toBe(0) is a valid line. It only looks wrong next to the expect(total).toBe(42) it replaced. So agent-roast compares the before and after of each change, not just the after.

It also does this without a model. The README is blunt about it: “It is deterministic and offline: no model, no tokens, no network.” A tool that polices a machine’s work by asking another machine to grade it would be a joke with a monthly bill. This one is a rule engine, so the same diff gets the same verdict every time.

When a check got weaker, what happens next is deliberately polite: “If a check got weaker, the agent is sent back once.” If it tries again, the README says, “you get a warning rather than an endless loop.” One bounce, then it hands the decision to you. And when the tool can’t tell, it stays out of the way: “Anything agent-roast cannot check lets the agent finish: it never breaks a session.”

There are 11 rules. Five block by default: a weakened assertion, a weakened guard, a skipped test, a focused test (the one that makes everything else silently not run) and a trivial assertion. The other six, which include removed tests and swallowed errors, only warn.

Setup is short. npx agent-roast hook install adds it to the agent hooks, there is a GitHub Action at Nari-Ab/agent-roast@v1 for pull requests, and --advisory makes it warn without ever sending an agent back. To see it work before touching anything, npx agent-roast demo runs “a scripted example in a temporary repository, nothing of yours is read,” and the README adds that “the scenario is scripted; the review is not.”

The most interesting part is the author’s own number. The README replayed the rules over 12,645 commits that touch tests or CI config in 166 public repositories. “Blocking rules fired on 0.3% of AI-signed commits” and “on 0.3% of the others.” Its conclusion: “There is no sign that AI-signed commits weaken checks more often.”

Same rate for the bots and for the rest of us. The README’s summary sentence says as much: “Weakened checks are rare, mostly deliberate, and not specific to AI.” The tool is aimed at agents, and the data says people have been grading their own homework just as often.

The author also admits the cost. On repositories the rules weren’t tuned against, “between a quarter and two thirds of its blocks were questions with a harmless answer.” Some of the blocks are false alarms, which is why the one-bounce design matters.

Expect before-and-after checks to become a standard CI step. The reasoning is simple: an agent that is rewarded for a green build will find the shortest path to one, and a diff comparison is the cheapest way to see which path it took. It needs no model, so it costs nothing per run and cannot be talked out of its verdict.

What I would watch is the false-alarm rate. A tool that cries wolf gets uninstalled, and the README’s own number says wolf-crying is real.

Five rules block. Six only warn.