Lightning strike at one of Google’s European data center caused permanent loss of user data. The lost data is negligible compared to the entire amount if user data and amounts to 0.000001% of the total Google Data. The incident occurred on August 13, 2015 when Google Compute Engine started showing errors to a part of its data. The errors were persistent in disks in the europe-west1-b zone. The issue started after four lightening thunder struck consecutively near the Data center. The affected disks suddenly returned I/O errors to their attached GCE instances. It also returned errors for management operations such as snapshot creation. A total of 5% data were risked out of which engineers were able to recover most of them. Few data were completely unrecoverable and they account to 0.00001%. Unfortunately the loss is permanent.
Google Data Center
From 09:19 PDT on Thursday 13 August 2015, to Monday 17 August 2015, some Standard Persistent Disks in the europe-west1-b zone began to return sporadic I/O errors to their connected GCE instances. In total, approximately 5% of the Standard Persistent Disks in the zone experienced at least one I/O read or write failure during the course of the incident. Some management operations on the affected disks also failed, such as disk snapshot creation.
Google however wrote in its report that they apologise to all those affected users whose data have been lost. The company also added that even though the accident due to natural calamity was an exceptional case, still the company is carrying out thorough analysis of the situation. They have found that there are several factors that have contributed in the mishap and they are trying to mitigate them and enhance the system to maximise reliability of GCE’s whole storage layer.
Read the entire report here
