Cloudflare stopped rereading whole DNS zones for one setting and cut peak memory 85%
Ordered queue acknowledgments made a single unfinished job everybody else’s problem, even when workers had completed later changes.
Changing one DNS setting could send Cloudflare back through millions of records. On the largest zones, a full rebuild couldn’t finish, and its retries held up unrelated changes sharing a Kafka partition.
The old path could avoid writing unchanged output while still paying to read and compare the records. A small write count concealed a lot of work. Cloudflare’s engineering post describes a planner that chooses the “smallest build we can prove is safe” and a scheduler that combines queued work for a zone. The figures here are Cloudflare’s production rollout measurements.
The queue had a particularly unpleasant dependency. Kafka tracks a committed position for each ordered partition. Finishing later work does not permit that position to skip an earlier unfinished message. A worker can be busy doing useful things while the durable progress marker remains stuck behind the largest job.
That distinction explains why adding workers alone would not settle the problem. The old expensive build still has to finish before the partition can advance safely. A restart can then replay later messages whose completion existed only in memory. Cloudflare explicitly accepts repeated work rather than risk losing a customer’s change. The interesting optimization is reducing how long that oldest outstanding job occupies the front of the line.
The scheduler also separates the work requested from the messages waiting for acknowledgment. Combining several updates for one zone can simplify the build while preserving the responsibility to finish every attached message. Throwing away a redundant rebuild request must not accidentally throw away the evidence that the corresponding customer event was handled.
For a team facing a similar symptom, the useful first trace would follow one update from receipt through database visibility, scheduling, output and acknowledgment. That distinguishes time spent doing a build from time spent waiting to be eligible for one. A graph of worker CPU alone would leave much of that path unexplained.
The rollout’s mirror environment supplies another practical detail. Reading the same events and database state while writing somewhere outside customer traffic makes an output comparison possible without asking customers to discover a missing record. A smaller work plan earns trust by producing the expected result across those boundaries.
The remaining operational question belongs to the events that cannot describe their own scope. Any future improvement there needs enough information to select a narrower build without guessing which records a setting might affect.
Cloudflare checked targeted and full builds against the same database state and compared owned output keys byte for byte. That matters because zones with the same name can share metadata: getting the boundary wrong could leave the DNS edge with stale data.
Cloudflare writes that “full builds still exist.” The next useful question for any similar pipeline is which events still need them, and whether those events can explain exactly what changed.