A Systems Academic Called Marc Brooker's Garbage-Collection Death Spiral a Fantasy. He Animated Real etcd Traces Stuck in It.

Felt-tip sketch of Chris, hands in his hoodie pocket, watching a ball circle a looped track while a lever beside it waits to be pulled. — Illustration: LockerGnome

“When I wrote Garbage Collection and Metastability, a widely-known systems academic told me that it was a fantasy,” writes Marc Brooker in his new post. He’d seen it happen in the wild “in ways I couldn’t share,” and he admits the 2024 post leaned on hand waving. “So it’s nice to have a real world example, picking on etcd again.”

The example is a short animation. etcd is “ticking along nicely at 500 reads/second” until the load jumps to 1,000. More reads mean more time in garbage collection, which means less CPU left over for reads, which means requests pile up and stay in memory longer, which means more garbage collection. “This is a self-reinforcing loop, and the system doesn’t recover without intercession,” Brooker writes. Take the extra load away and etcd stays sick until somebody steps in and breaks the loop by hand.

That’s the definition of the thing. The 2021 HotOS paper that named the class, by Nathan Bronson and colleagues, describes metastable failures as ones where “a trigger causes the system to enter a bad state that persists even when the trigger is removed,” as Brooker quoted it in his 2021 write-up. He compared it to the oldest fix in tech: turning things off and on again.

The other three animations come from the same rig. Postgres 16, running in a real MicroVM, handles 20 queries a second easily while its whole working set sits in page cache. After a spike, the running queries (12MB each) have eaten that memory, so everything reads off disk and runs slower. A write spike on etcd makes overloaded peers “look like they’re unavailable, which triggers a storm of application-level resends.” And the one that opens the post asks whether a Raft cluster can lose writes when a single member’s storage “lies about fsync().” It can.

Brooker built the animations with Kiro and KiroCrew, “mostly with Opus 5.5,” and says the results “came from a real test environment, which captures the traces that back the animations.” His post is the whole source for the four traces; it’s short, and it’s mostly video. He thinks a language without garbage collection could fall into the same hole through allocator work, but hasn’t been able to reproduce it yet.

Why it matters

Production-proven isn’t proof. Brooker’s point about the etcd write storm is that “etcd is widely-used quality software. Operational experience doesn’t seem to effectively shake out these kinds of issues.” Years of uptime only tell you which spikes it hasn’t seen.

This is how big outages happen. A 2022 OSDI study of 22 metastable failures across 11 organizations found that “at least 4 out of 15 major outages in the last decade at Amazon Web Services were caused by metastable failures.” The trigger is usually ordinary: a burst of traffic, a deploy, a short blip.

Postgres shows it doesn’t need a cluster. No consensus, no network partitions, just a cache that held the working set until a spike pushed it out. Brooker likes that example “because it shows that even simple, non-distributed systems have these behaviors.”

The cure for GC loops is a cap. In the 2024 post, Brooker writes that a system that limits concurrency “isn’t likely to” fall into the GC cycle. A system that only limits arrival rate can still get sent “into a mode where it spins around this loop until it collapses.”

Brooker already has Stability Sim for the abstract version, where retries and queues “make metastability really easy to trigger.” What these animations add is a trace from software you’re probably running. Expect more of this: an LLM can draw the picture in an afternoon now, and the hard part has moved to the test rig that captures something true. The open question is the one Brooker left himself, whether a runtime with no garbage collector at all can be pushed into the same spiral, and what it would take to catch it on video.

Steal this

When your service limits requests per second but not requests in flight, add a concurrency cap before the next traffic spike. A rate limit lets slow requests stack up in memory; a cap on in-flight work stops the GC loop from starting. This is like a restaurant that seats people at the door’s pace even after the kitchen backs up.
When a load test ends, don’t stop at “it survived the spike.” Drop the load back to normal and watch whether throughput comes back on its own. If it doesn’t, you’ve found a metastable state, and you’ll want the trace. Kind of like checking that a fuse resets, not just that it blew.
If you run etcd or Postgres, write down what an operator actually does to break the loop (shed load, restart members, flush clients) and test it once on purpose, before the night you need it.

Search

A Systems Academic Called Marc Brooker's Garbage-Collection Death Spiral a Fantasy. He Animated Real etcd Traces Stuck in It.

More Infrastructure