The Month in Cloud Bills: Google's London backup generators never started, and attackers hunting AI keys hit app hosts Zeabur and Railway and the coding tool OpenCode

Chris stands beside a silent backup generator holding a long paper bill that unrolls to the floor, while a stranger walks off with a ring of keys — Illustration: LockerGnome

Google Cloud London had a backup for the generator controller, and the backup failed too

At 22 Pacific on October 19, which was 06 on a Tuesday morning in London, the power grid feeding the building that holds Google Cloud’s europe-west2-b zone dropped out. The building did what it was built to do and moved the load onto UPS batteries while the diesel generators started up.

The generators never took over. Google’s report says the programmable logic controllers that run that handoff, “both primary and backup PLCs,” “failed to initiate the logical automatic start/transfer operations of the generators.” When the batteries ran out at 23 Pacific, the zone’s servers, “network switching fabric,” and storage arrays lost power together.

Utility power came back “within 1 hour,” Google says. The outage ran six hours and ten minutes anyway, because nothing comes back from a cold stop in one step: engineers reset and tested the generator controllers, restarted core infrastructure, then let traffic back into the zone a little at a time.

The two London zones that kept their power got hurt too. Everybody’s failover landed on europe-west2-a and europe-west2-c at the same moment, and Google lists BigQuery, Looker, Compute Engine, Kubernetes Engine, Regional Persistent Disk and Cloud SQL among the products hit by “sudden compute capacity exhaustion from surging demand.” Kind of like every car on a closed motorway taking the same exit.

Outside the data center, the Press Association reported trouble at Lloyds, HSBC, Halifax, Royal Mail, DPD and Rightmove, and Revolut customers having problems with card payments, transfers, cash withdrawals and logins. “We can confirm that all our customers’ money and data are safe,” Revolut said. The courts service’s cloud video platform went down, and at the Russell Brand trial, Mr Justice Bennathan told the court there were “problems with Google.” Google’s own line: “We apologise to all who are affected by the disruption.”

What six hours dark buys you under Google’s SLA

Google’s Compute Engine SLA promises a single VM, outside the memory-optimized family, 99.9% uptime a month. October has 44,640 minutes. A VM that was unreachable for all 370 minutes of this outage ran 99.17% for the month, which lands in the 95-to-99.9% band: 10% of that instance’s monthly bill, “credited to Customer’s future monthly bills.” (Not refunded. Credited.)

The next band, 25%, starts below 95%. That takes 2,232 minutes dark, about 37 hours. The full month back takes about 74.

Then the paperwork. You “must notify Google technical support within 60 days” and “provide Google with log files showing Downtime Periods,” or you “forfeit” the credit. If you did the responsible thing and spread across zones, the SLA only counts Downtime when “all applicable running instances” lose connectivity, so an a-and-c setup that stayed reachable while it crawled has nothing to claim. The SLA calls the credit the customer’s “sole and exclusive remedy.”

Zeabur kept long-lived admin keys in plaintext, then announced it was closing

Zeabur, a platform that deploys and hosts apps, published its incident on October 19. An attacker used leaked internal credentials on October 14 and 15, and the root cause, in Zeabur’s words, was that “long-lived admin credentials were stored in plaintext.” What got exposed: 3,218,965 environment-variable records across 321,810 accounts. Environment variables are where apps keep their passwords and API keys, and Zeabur says “the attacker was mainly looking for API keys they could use directly, especially keys for AI services.”

The same post says “Zeabur has announced that it will shut down on December 31, 2026, which is an operational decision.” (Noted.)

Three days later came the claims notice. Losses from the breach go in a separate support ticket by November 30, backed by “third-party platform invoices or usage records,” meaning the AI provider’s bill for the calls you didn’t make. Refunds are “capped at the total amount you actually paid in cash.” Customer data is deleted December 31, and payouts are due by January 10, 2027.

OpenCode’s console got hit by a “distillation attack” paid for with users’ own AI accounts

OpenCode is an AI coding tool; its hosted Console is a separate service. TechCrunch’s Aditya Mehta reported on October 19 that attackers got in through a legacy system that was being decommissioned, and used customers’ AI provider keys for what OpenCode called “a large-scale distillation attack that impacted 1.4% of OpenCode Console users.” Distillation means pumping a big model full of prompts and keeping its answers to train a cheaper copy, which takes an enormous number of paid calls, preferably on someone else’s account.

Victims found out from their providers. “I saw my Open AI, Anthropic, and Google AI spend alerts all fire off before going to bed,” one developer told TechCrunch. Charges ran to hundreds of dollars per account, a few past $1,000. Anthropic and Google blocked the exposed keys, OpenAI refunded, and OpenCode said refunds were underway.

OpenCode’s Dax Raad drew the line for users of the free tool: “Note this is the OpenCode Console, not OpenCode itself, those keys never come to us.” And when the team pointed AI models at the attack code to figure out what it did, Raad said, the models stopped working because the malicious code tripped their safety filters. “[T]hat was a bit frustrating.”

Railway paused every deployment while it worked out whose keys were read

Railway, another app host, posted its disclosure on October 21. It “detected anomalous activity involving systems that store customer environment variables,” and “an unauthorized third party may have accessed encrypted environment variables.” Railway calls it “part of a series of attacks recently targeting platform companies,” aimed at getting into AI inference providers.

Its response was the cloud version of closing the store to count the keys. “We have notified these upstream providers to revoke all keys proactively,” Railway wrote, and “All deployments are paused while we investigate.” Follow-up posts on October 23 covered resolving login issues and recovering failing deploys.

And the one that billed Gemini to the victim’s card

We covered this one on October 23: attackers bought Google search ads for “gcp,” phished admin logins, linked their own Google Cloud projects to the victims’ billing accounts and ran Gemini on them. Wiz put the damage at tens to hundreds of thousands of dollars within 24 to 48 hours.

Why it matters

The first alarm was a spend alert. OpenCode’s users learned about the breach from their AI providers’ billing emails, not from OpenCode. Zeabur’s attacker was after AI keys, and Railway says its attackers were too. A stolen AI key pays out the moment it’s used, so the meter runs before the security team hears about it.

One platform’s secrets store is a lot of people’s money. Zeabur’s 3.2 million environment-variable records sat behind admin credentials kept in plaintext. Every customer who pasted an AI key into that dashboard handed it to the same lock. The platforms that hold keys for thousands of apps have become the target because one break-in pays for thousands of accounts.

Redundancy fails in the same place twice. Google had a primary and a backup controller, and both faulted. The zones that kept power ran out of room because everyone’s failover pointed at them. Spreading across zones kept a lot of people up, slowly, and also took them out of the SLA’s definition of down.

An outage credit is a coupon. Ten percent of one month’s instance bill, applied to future bills, claimed by you, proven by your logs, within 60 days. None of it covers what the outage cost the business running on top.

Steal this

If an AI provider key sits in a hosting platform’s environment variables, give that app its own key in its own project at the provider, with the provider’s spend limit set near what the app really spends, where the provider offers one. Kind of like giving the teenager a prepaid card instead of your Amex.
When a platform you use discloses a breach of stored secrets, go to each provider and rotate every key you kept there that day, even if the platform says it already asked providers to revoke them. You can check a key you rotated yourself. This is like changing the locks after the landlord says he’s talked to the locksmith.
If you ran a single VM in europe-west2-b on October 20, export the logs that show when it went unreachable and came back, and file with Google technical support now. The SLA gives you 60 days and no credit without logs. Kind of like a warranty claim where the store won’t look up your receipt for you.
If you used Zeabur, file the loss ticket with your AI provider’s invoices before November 30, and pull your data out before December 31, when Zeabur deletes it. This is like clearing out the storage unit before the auction date.

Platform breaches that end in somebody else’s AI bill look set to become a regular entry here. Railway’s line about telling upstream providers to revoke keys is the sentence to watch for in the next disclosure, and the AI providers are now the ones blocking keys and handing out refunds after breaches they didn’t have.

Google’s report is still preliminary. Why a backup controller failed the same way as the primary is the question its final report, “with preventative actions,” has to answer, and it hasn’t been posted yet.

Search