Ubicloud Traced Kimi's Tool-Call Accuracy Drop to 91% Back to a vLLM Default Change

Ubicloud runs open-weight models for customers, and its post opens with the complaint every host hears: “When I use an open-weight model, I see results that are nowhere close to what the quality benchmarks indicate.” Jesse Donkervliet and Junhao Li went looking at one piece of that gap, the tool call.

An agent hands the model a list of tools as JSON, and the model is supposed to answer with a call that matches. Models mostly do. Sometimes they emit “search_web when the available tool is web_search, or use city_name when the active schema expects location,” as the post puts it. Each miss means an error message in the context, a retry and more tokens.

The numbers came from Moonshot AI’s K2 Vendor Verifier, a 2,000-request test Moonshot publishes so hosts can check they serve Kimi the way Moonshot does. “When we served Kimi K2.6 with vLLM version 0.28, we got the expected tool call schema only 92% of the time,” the engineers wrote. “In other words, 2 out of every 25 attempted tool calls failed.”

The fix has a name: constrained decoding. Using XGrammar, the server only lets the model pick tokens that keep the call valid. “This changes schema accuracy from a best-effort approach to a runtime guarantee,” they wrote, and it “adds negligible performance overhead.” Moonshot’s own verifier asks hosts for exactly this, noting that models “have no built-in mechanism to enforce a hard JSON schema.”

Ubicloud first built it into its own fork of vLLM. “After vLLM added this feature in v0.25, we switched to using this feature from upstream for Kimi and DeepSeek. Unfortunately not long after, we found out that our tool-call accuracy score dropped back to 91%!”

The grammar code was fine. “Instead, we found that vLLM changed its default behavior for requests that do not explicitly turn constrained decoding on.” The change is pull request #45600, merged in June: with tool_choice set to auto, if no tool in the request sets strict, vLLM returns no constraint at all.

The vLLM maintainers had their reasons. The post says users of gpt-oss-20b, Qwen 3.6 and GLM 5.1 hit problems with it on by default, and that grammar translation bugs could drop tool calls or fail requests. “This made disabling constrained decoding by default the safer option for a model-agnostic serving engine such as vLLM.” Then the engineers add: “Ubicloud’s focus is different from that of vLLM however.”

The way back in is a server flag. Pull request #56268 added --tool-strict-level, shipped in vLLM 0.31, and its documentation spells out the problem: “Most OpenAI-compatible clients and agent frameworks never set strict on their tools.” Set it to function and the server constrains the tool-call wrapper and the function name; set it to parameter and every tool’s arguments are pinned “as if every tool had strict: true.” Ubicloud says it turned enforcement on in production “once we verified 100% tool-call schema accuracy across all served model versions.”

One caveat sits in the middle of the post. It points to bugs where constrained decoding and speculative decoding disagree, and the grammar wrongly rejects speculated tokens, “leading to failing requests” (issue #44006). Speculative decoding is one of the main ways hosts make big models answer fast, and the post doesn’t say how Ubicloud’s production setup handles the two together.

Expect “we pass the vendor verifier” to become a line on hosting price pages, the way uptime numbers did. Moonshot built the test so providers could prove they serve Kimi faithfully, and Ubicloud just showed a version bump can quietly undo a passing score. Anyone running open-weight models for agents will likely end up rerunning a test like that after every engine upgrade, the same way they rerun load tests now.