Disaster is Out There. Is Your Data Center Ready?

Keeping a data center up and running at optimal efficiency is a difficult challenge under the best of circumstances. Power demands, cooling requirements and security considerations are ongoing concerns for data center providers. However, looming in the back of their minds is a much larger potential problem, one that could pose an even greater threat to their clients’ valuable data and assets: natural disasters.

Whether the disaster comes in the form of a hurricane, a flood or something else completely unexpected, data centers must be prepared to handle whatever misfortune Mother Nature throws at them in order to protect customer data and continue to deliver vital services that keep businesses running.

They aren’t called “disasters” for nothing
Losing access to data during or after a natural disaster can be devastating for a business. Far more than simply an inconvenience, a disruption in data and services can have a cascading effect that can easily doom a company. Roughly 60 percent of companies that suffer data loss due to a disaster of some kind go out of business within six months. It’s not just data loss that they need to worry about either; for companies that rely on data centers for cloud services, losing access to critical applications due to disaster-related downtime means losing out on customer opportunities and suffering damage to their brand. Even a short period of disruption can inflict a mortal wound on a small company operating on thin margins.

Data centers have responded by assembling comprehensive disaster plans that inform both current and potential customers of how they plan to protect their data and maintain service uptime in the event of a natural disaster. While these precautionary measures are vital, data centers also need to know how they will respond in the case of a disaster that overwhelms even the best-laid plans.

Plan ahead
Modern data centers are actually quite well-prepared to deal with most types of natural disasters. They perform comprehensive risk assessments to identify likely risks and the potential consequences that would result from each of them. Step-by-step disaster response checklists outline the key actions data center personnel need to take to respond to these situations. A well-managed data center makes these disaster plans readily available and conducts regular drills to ensure everyone knows what to do should the worst come to pass. Considering that about 22 percent of data center outages are caused by human error, these drills are incredibly valuable to maintaining readiness for emergency situations.

Since every second of server downtime equates to financial losses for clients, well-run data centers have extensive network and power redundancies in place to make sure they maintain service uptime in the event of a natural disaster. Regular testing of backup power systems is essential for ensuring that data and critical operations remain online and secure when primary systems fail. As with other readiness drills, data center personnel need to know what actions to take when redundancy systems don’t operate as designed. They may be called upon to make critical manual interventions to keep systems running, so they must have some level of training in this process.

Think beyond the data center
While most data centers are able to withstand quite a bit of damage, recent hurricanes have exposed the unpleasant fact that the local infrastructure around these facilities is far more vulnerable than the data centers themselves. When Hurricane Sandy struck New York City, many backup generators failed because they ran out of fuel and could not be replenished due to flooded city streets that prevented fuel trucks from entering the city. Data centers in the Houston area famously remained online after Hurricane Harvey dumped record-setting rain on the region, but the flooding was so extensive that no one could access the facilities by road.

It’s also advisable for data center providers with multiple facilities to back up data in locations that would be far from the one affected by a natural disaster. Many customers in the New York area held their collective breath during Hurricane Sandy because most of their data centers backed up their data at nearby New Jersey facilities. While the backup systems fortunately held firm, the situation revealed that many providers were taking an unnecessary risk by keeping data backed up in centers so close to the original source. Data center providers with facilities spread across a wide geographic area are better able to ensure that a large-scale natural disaster will not be able to take out all of their services.

Disaster is unpredictable
Natural disasters can hit hard and fast. Here are just a few examples of natural disasters that can leave data centers reeling:

• Lightning: They say lightning doesn’t strike the same place twice, but in 2015 one of Google’s European data centers was struck by lightning not just twice but four times, causing errors in 5 percent of the disks responsible for Google Compute Engine (GCE) instances. Although the company restored many of the drives, an estimated 0.000001% of data stored in the data center was irrecoverable. While that might not sound like much, try telling that to the customers who were affected by it.
• Hurricanes: According to National Geographic, 2017 was the most expensive hurricane season in U.S. history, costing roughly $200 billion. With their lethal combination of high winds, storm surge and heavy rains, hurricanes are one of the most dangerous natural disasters data centers must contend with. The sudden flooding resulting from Hurricane Sandy in 2012 put many data centers at risk in New York, a risk that was made worse by the fact that most of the backup facilities were located in neighboring New Jersey. Although outages were limited, the situation exposed the danger of locating redundancies in the same geographic region where they could potentially be knocked out by the same weather event.
• Tornadoes: A devastating 2011 tornado ripped through several hospital buildings in Joplin, Missouri, one of which was a data center. While none of the data lost was mission-critical, that was only because most of the information stored there had been transferred to a new offsite data center just a few weeks earlier. Hospital officials noted that if the tornado had hit just a month earlier, the data loss would have been catastrophic and would have rendered the hospital completely inoperable.
• Flooding: Severe flooding in Leeds, UK caused a Vodafone data center to temporarily lose power during Christmas of 2015. While data loss was negligible, the power outage disrupted mobile phone service temporarily. Vodafone has a bit of history with flooding, having suffered one of the most infamous data center disasters when its Istanbul data center was devastated by flooding in 2009.
• Earthquakes: So far, data centers have lucked out when it comes to earthquakes. Modern architectural standards and additional precautions such as special enclosures and rollers for server racks have gone a long way toward protecting data centers from earthquakes, even in high-risk areas. Perhaps the most noteworthy example of this was the way Japan’s data centers escaped the devastating Tōhoku earthquake of 2011 unscathed.
• The Unexpected: Disaster planning is all about expecting the unexpected. Take, for instance, the squirrel that knocked Yahoo’s Santa Clara data center offline for several hours in 2010, or the truck that drove into a transformer responsible for feeding power to a Backspace data center in 2007.

With the threat of natural disasters potentially growing in coming years, data centers must do everything in their power to protect their facilities and their customers’ precious data. Cloud services and networks of data centers spread across various regions make it easier to back up critical assets, and modern construction techniques can protect equipment from physical harm, but data centers still rely on factors outside of their control to stay up and running. By assuming the worst and formulating a plan for dealing with every eventuality, they can take the first steps toward true disaster readiness.