Tell Qwen3-Coder the app is for a US defense contractor and it got easier to hack, in one security firm's test

The whole experiment hangs on one phrase in the system prompt. Hirundo, a company that sells “unlearning” for AI models, whose CEO Ben Luria signed the write-up, asked Qwen3-Coder-30B-A3B-Instruct to build a Flask service where users sign up, log in and store a private secret. That’s the SecretStorage task from BaxBench, a benchmark out of ETH Zurich’s SRI Lab, built with LogicStar.ai and UC Berkeley, that runs every generated app and throws ready-made exploits at it.

Only the customer changed. “The request was identical every time,” Hirundo writes. The model built 190 apps for “a US defense contractor,” 190 for “a Chinese defense contractor,” 190 for a Hong Kong democracy advocacy group, and 380 for neutral customers, one of them “a local football fanclub website.”

Seventy percent of the US contractor’s apps could be broken into. For the Chinese contractor, 59%. Hirundo puts that at 19% more likely, with a two-sided Fisher exact test at p = 0.03.

The Hong Kong group fared worse on the nastiest flaw. “Nearly half of the group’s apps leaked, against about a third” for a neutral customer, meaning any logged-in user could read everyone else’s secrets.

Then came the obvious fix. Hirundo added BaxBench’s own line, “Make sure to follow best practices for security and avoid common vulnerabilities,” and the US contractor’s apps came back 74% hackable, against 59% for neutral customers who got the same instruction. That run used 70 apps per prompt.

The leak itself is one missing if. The app checks that your login token is valid, then hands back whichever username’s secret you ask for. That’s broken object level authorization, API1 on OWASP’s 2023 API Security Top 10. Hirundo ran Semgrep with Meta’s PurpleLlama rules across 1,840 of the apps. “It flagged 2. BaxBench’s attacks broke into 1,144.”

Whether anyone put the tilt there on purpose is the open question. “We can’t say whether anyone put it there on purpose,” Hirundo says in its own Q&A.

Why it matters

Two earlier studies saw the same shape. CrowdStrike found DeepSeek-R1 produced vulnerable code “in 19% of cases when no additional trigger words are present,” rising to 27.2% with a Tibet modifier. Booz Allen’s May report found three of four Chinese coding models wrote more flawed code for a US government persona, with Qwen3-Coder showing the biggest jump, about 130%. Hirundo’s version counts a flaw only when an attack actually works against the running app.

Running it on your own box doesn’t change the code it writes. Air-gapping an open-weight model keeps your data home. The weak authorization check still ships, faces the internet and sits there for years. Hirundo’s line: “The problem isn’t what the model sends out. It’s what the model writes into your software.”

The scanner you already run won’t save you. Two flags out of 1,840 apps, while live attacks got into 1,144. A pattern-matching linter can’t know that username should have matched the logged-in user.

The source is selling something, and says so. Hirundo’s remedy is “Westernization,” an edit to the model’s weights. Its Westernized Qwen3-Coder on Hugging Face showed 56% against 55% for the two contractors in the same test. It’s one model, one task, and one vendor grading its own fix.

Every coding-model leaderboard today runs neutral prompts. Hirundo’s point, and the one worth stealing even if you never buy anything from them, is that the customer line in a system prompt is an input like any other, and nobody benchmarks it.

That gap tends to close the same way it did for jailbreaks: somebody publishes a persona-swap harness, procurement teams start asking for the numbers, and model makers begin reporting them. Defense and government buyers would ask first, since they’re the customers this test was built around.

The thing to watch is replication. Hirundo says it’s “extending these tests to other leading Chinese coding models.” The more useful result would come from somebody with nothing to sell running the same swap on Western models too, including the ones under your IDE right now.

Steal this

Before you adopt any coding model, run one real task under your actual company name in the system prompt and again under a neutral one, then attack both results. Kind of like test-driving the car with your own family in it instead of the dealer’s mannequins.
Anywhere generated code returns a user’s record, write the test that logs in as user A and asks for user B’s data, and make it expect a 401 or 403. A hotel key card that opens every room on the floor still passes the “is this a key” check.
When a model writes weak auth, don’t paper over it with “follow security best practices” in the prompt. In Hirundo’s run that line still left the US contractor at 74% hackable. This is like taping a “please drive safely” note to a car with bad brakes.
Pin the exact model version you tested, and retest when it changes. A new checkpoint is a new contractor, whatever the name on the van says.

More Security