Bruce Ediger Built a Million-Page Fake Website as a Trap and Caught One Coordinated Crawler Reading It Rank by Rank From 7,690 Contabo IP Addresses

“I’m interested in getting a few crawlers stuck in a maze, and wasting a lot of their time because they’re poorly designed,” Ediger wrote on his blog, Information Camouflage, in an earlier post about the experiment. He opened the junkyard on August 1, 2026 and pulled the plug on September 30, “when I lost patience with the experiment.”

The maze is a PHP program. Page N links to pages 2N+1 and 2N+2, and back to (N-1)/2, starting from /node_0.html. That numbering is breadth first, so page numbers line up with tree levels, which he calls ranks. He admits the tree came out lopsided, 20 full levels and four nodes on a 21st, because “I didn’t correctly calculate what node number to stop generating parent and child links.”

Over the two months his earlier post counted 15,440,078 requests from 4,248,158 unique IPs. Contabo, “a cloud provider, renting virtual private servers, and dedicated servers,” sent 6,278,685 of them from about 8,616 addresses.

His new post, “Web Page Crawler Revealed,” lines those Contabo requests up against the page numbers. Six full passes show up, each about a million requests, each at roughly four a second. Out of 262,802 logged seconds, 260,308 held two to five requests. “Over 4 series of requests (separated by time), 7690 IP address make requests,” he writes, and 2,964 of them took part in all four.

The tree gave the crawler away. Ediger wrote a checker for pages requested before their parent, which a link-follower can’t know about. Series B had five, the only repeat requests in the whole pass, and he suspects “they’re manual checks.” Every other request followed a link the crawler had already seen, and each pass started the same way: ranks one through five, then rank 6 only up to page 46, with page 47 left for about halfway through.

Then there’s the counting. “The vast majority of the IP addresses make a multiple of 101 requests,” he writes, which “looks like the programmer wanted a worker to make a nice round number of requests, 100 would be nice, but made an off-by-1 error in the loop.”

He tried to reproduce the pattern with a Go simulation of the usual queue-and-workers crawler design that Matthias Endler describes. A plain queue “looks nothing like the Contabo Crawler’s concurrent rank node number requests,” and a stack of queues came out “fuzzier.” Who runs it, and why it read the same million fake pages six times, he doesn’t say.

Six reads of a million worthless pages say something about where crawling is going. Scrapers that pay for thousands of rented addresses can keep that up for weeks, and the money is going into hiding where the requests come from, not into deciding whether a page is worth fetching. Ediger’s own estimate is that “A single VPS could probably keep up four requests per second,” so the thousands of addresses look like disguise more than horsepower.

That points at what site owners will lean on next. Blocking by address gets harder as the pool grows, while behaviour (the order, the rate, the round-number batches) stays stubbornly the same from pass to pass. Expect more traps like this one, with structured URLs that make a crawler’s traversal readable in an ordinary access log.