Google says they deleted a $135 billion cloud customer account by accident.

In early May, Google Cloud had one of its worst fears come true: an embarrassing mistake erased all of a customer’s data and accounts. The unfortunate target was the Australian pension fund UniSuper, which is in charge of over 600,000 people and over $135 billion in assets. The pension fund couldn’t do anything for two weeks while it tried to get back to normal using backups from outside sources.

UniSuper quickly lost access to all of its data and services stored on Google Cloud, including backups, on May 2. This was the start of the problem. Soon after, the two companies said in a joint statement that the loss was caused by an “inadvertent misconfiguration,” but they didn’t say much else. When UniSuper went back online on May 15, it had to go through a full repair process.

Google laid out exactly what went wrong this week. When using an internal tool to set up UniSuper’s private cloud services, someone at the company forgot to fill in a field. That mistake, which didn’t seem like a big deal, caused UniSuper’s account to be marked to be deleted automatically after a certain amount of time.

Google has made a TL;DR version of the story:

Google workers accidentally set up the GCVE service incorrectly when they first set up a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool. This happened because a field was left blank. Because of this, the customer’s GCVE Private Cloud was set to a fixed term and deleted automatically at the end of that time, which was not what was meant to happen. Both the incident trigger and the behavior of the system that comes after it have been fixed so that this doesn’t happen again.

After the error, Google says the “customer and Google teams worked 24×7 over several days to retrieve the customer’s GCVE Private Cloud, repair the network and security arrangements, restore its applications, and retrieve data to restore full operations.”

Google also said that there was no “customer notification” because the deletion was done by accident using Google’s tools. UniSuper must have been shocked by the whole thing.

However, as Ars Technica points out, there was mixed news about whether or not UniSuper’s files kept in Google Cloud Storage were deleted. At first, UniSuper said it had to use backups from other companies because its Google files were lost. On the other hand, Google’s blog says that the cloud copies were not damaged and were “instrumental” in the restoration.

Google has promised a lot of “remediation” steps to make sure this doesn’t happen again, which is a good thing. They got rid of the internal tool that was giving them trouble and moved the functions to interfaces that customers could control. They also cleaned up their records and made sure that none of their other Google Cloud accounts were set up incorrectly to be deleted.

The company said again that strong deletion protections are in place, such as soft deletes, advance alerts, and checks that a person approved the deletion.

It’s scary for millions of cloud customers, but Google has stressed that this was an “isolated incident” that only affected one customer. They say there aren’t any problems with the system that could cause other Google Cloud users’ files to disappear on their own.