A false smoke alarm shut off a trading datacenter's cooling and nearly set off its sprinklers

The most important feature of a datacenter fire system is that it fails safe. At Jane Street’s New York machine room, one design decision made the safe option the dangerous one.

Michael Bacarella, who was on the firm’s systems team, tells the story on his blog. Jane Street kept its datacenter near the trading floor, in a skyscraper at 1 New York Plaza, so that someone could always run in and pull the plugs on runaway trading systems. The room had about 10 rows of server cabinets, 10 cabinets deep. His own apartment was a 200-square-foot studio, nine minutes on foot from his desk, which he boasted about to the team. It ended up mattering.

One evening his phone rang. It was the Hong Kong office, calling about a New York problem: the datacenter’s smoke alarm had gone off. Contractors working in a prep room next door had kicked up dust.

Here is the trap. The sprinklers over the machine room weren’t kept full of water; a separate system detected smoke and only then pressurized the pipes. Each sprinkler head then has a heat-sensitive plug that opens when the air gets hot enough, so the logic is that a smoke alarm plus heat means a real fire. But the same smoke alarm was also “wired to shut down the CRAC units(!),” the room’s air conditioners, because circulating air would feed a real fire. Dust triggered the first step and the cooling shutdown, and a room of computers with no cooling starts climbing toward the heat that opens the sprinklers. He puts the trigger at 150°F.

“As fast as I possibly could,” he writes, he ran back, rode the elevator 33 floors, swiped through two keycard doors and stepped into “what felt like a sauna,” and pulled power cables out “row by row, cabinet by cabinet.” Then he watched the monitor: “141 F. 142. 143. 143.5. 143.6. Finally it stopped.”

With the servers unplugged, the climb ended 6.4 degrees short of the sprinkler plugs. Major New York markets were already closed, so the downtime was small, and the datacenter was running again the next morning. The post-mortems were, he says, “closer to engineering students in a college dorm arguing over what went wrong.” The outcome: the sprinklers stayed, procedures and surveillance tightened, and the servers stayed in the office.

He also fixed the part that was luck. Hong Kong had caught the problem only because someone saw server after server flagging a rising temperature. No single server was set to alert on a little heat, because that happens all the time. So Bacarella wrote an OCaml monitor that, every minute, asks every server in each location for its onboard temperature, averages them, and alerts the systems team if the average is elevated. A dedicated room sensor or a CRAC alarm could have worked too, he notes, but he didn’t trust their telemetry as much as the fleet itself.

The firm kept its servers in the office, right next to the trading floor, until Hurricane Sandy “smacked us upside the head,” in Bacarella’s words, which he calls a story for another day. Automation, as this one shows, protects you from the failures its designers imagined and not a dusty contractor.

Pull the cords, then ask why.