A hidden link bans disobedient crawlers; its maker reports daily traffic falling from 216GB to 25GB
A site owner reports a large traffic drop after using a forbidden link as a bot trap, with blunt blocking as the tradeoff.
A link tells a crawler not to follow it. The crawler follows it anyway. Terence Eden’s response is to turn that visit into a blocking rule, giving the disobedient request a rather immediate consequence. The website has put a rake behind the velvet rope.
The write-up describes a hidden link whose path is disallowed in robots.txt. Requests to the trap result in IP addresses being added to an .htaccess blocking list. That combines two different signals: the crawler ignored the published instruction, and it fetched something an ordinary visitor was unlikely to seek out.
Eden reports blocking hundreds of addresses during the experiment and a substantial reduction in transferred data. His figures include a peak of 216 GB per day before the change and roughly 25 GB afterward. Those numbers describe the site owner’s observation, rather than a controlled comparison that isolates the trap from every other change in traffic.
The appeal is how little interpretation the mechanism needs. It does not try to decide whether a request’s prose sounds human. It sets a condition and reacts when that condition is met. A server can check a path request directly, which is a smaller question than identifying the intent behind all automated browsing.
The bluntness comes with it. An IP address is a network identifier, not a permanent identity for one person or one program. Shared connections and address reassignment can make a block affect someone other than the original visitor. A useful preview service or crawler can also follow a link in a way the site owner did not anticipate.
Deployment details can change the meaning of the signal. A server behind a proxy must know which address information it can trust. A client-supplied forwarding header is not automatically evidence of the original requester. Blindly copying a blocking script without understanding that boundary can turn a simple trap into a way to add the wrong address.
There is an operational cost to the list as well. Growing per-address rules need inspection and a way to recover from mistakes. If a legitimate visitor reports a problem, the owner needs enough evidence to find the rule responsible. The simplicity of adding a ban should not hide the separate work of explaining and reversing one.
For a personal site, the account is an interesting example of using a deliberately narrow trap against unwanted scraping. The next useful step for another operator is to inspect their own logs and hosting setup before adopting the code. A reported bandwidth drop can justify investigating the idea without establishing that every visitor caught by it deserves a permanent exclusion.
That forbidden link keeps receipts.