Survival of the Prepared
Addressing Disaster Recover: It's all in the plan

It happens. Hurricanes, floods, earthquakes, and fires—the so-called “Acts of God”—that shatter lives and devastate businesses. Their man-made cousins—fire, power failures or worse yet, terrorism—are no less destructive. Together these unanticipated events pose risks to business as usual. Sometimes the financial fallout persists for years. Some companies are never quite the same while others simply go out of business. However, a doomsday outcome can be avoided in many cases.

In today’s digital economy, Disaster Recovery (DR) is often synonymous with data recovery. Companies depend on information to operate. If a critical application goes down, business screeches to a halt and revenue inevitably suffers. Companies that are complacent about DR planning take a huge risk: 50% of businesses that lose their data due to disasters go out of business within two years¹,while 93% of such businesses go under within five years².

And yet as recently as last fall—after Hurricane Katrina had devastated New Orleans and much of the Gulf Coast—many businesses still had their heads in the sand. An AT&T-commissioned survey found that one-third of 1200 companies surveyed had no disaster recovery plan. Furthermore, nearly a quarter of the companies surveyed had not updated their plans in the preceding 12 months³.

In a recent article about organizational resiliency, Harvard Business Review⁴ Senior Editor Diane L. Coutu opines: “[Resilient businesses] train [themselves] how to survive before the fact.” Coutu points to Morgan Stanley’s “hard-nosed realism” in disaster planning as a perfect example. Morgan Stanley, the biggest tenant of the Trade Center Towers, had begun preparing for a major terrorist attack after the 1993 Trade Center bombing. Management invested in three backup centers as part of a larger corporate recovery plan. The day after the September 11th attacks, the company was serving customers.

Firms like Morgan Stanley recover, even thrive, because they prepare. A month before Katrina pounded the Gulf Coast, Alfa Insurance invested in self-contained mobile recovery units equipped with power and voice and data communication systems. After the hurricane hit, adjusters were immediately able to conduct business with policyholders in Mississippi and other areas⁵. Without the mobile units, Alfa would have lost sales and their customers would have been left in the dark.

Adams and Reese, LLP was also ahead of the storm. The New Orleans law firm had already instituted scheduled system backups and had moved backup tapes to an off-site location long before Katrina hit. When Katrina did hit, their email system went down for only 15 minutes, thanks to the third-party emergency mail service the firm engaged following the events of September 11th⁶.

No template plan

Disaster Recovery plans are by definition heterogeneous. Two companies in the same industry—diverse in size, operations, and systems—require two different recovery strategies. Industry classification also dictates which systems should be fixed first and how quickly they must come back online. An online florist relies on e-mail to verify orders and shipping, while an office supply company needs its customer database and phone system to stay in business.

In general, priorities in restoring a business follow an order similar to the following: 1-People 2-Power 3-Hardware 4-Software 5-Data. In other words, employees—at least those who are critical to core functions—should be brought back “on line” before hardware, software, and data. What good is equipment and data when no one is able to work with them?

There is no one-size-fits-all DR plan—different disasters require different recovery plans. However, there are some basic steps every company should follow to ensure they are prepared. The first step is identifying the most likely natural disaster scenarios; this will dictate how preparations will evolve. California doesn’t have tornados but much of the state sits on active fault lines, making earthquakes a likely potential disaster. Twisters are commonplace in the mid-western United States, while seasonal hurricanes batter the Atlantic and Gulf Coasts. Note: Companies operating overseas should perform independent assessments of the mostly likely regional threats.

The next step is categorizing business functions and systems in order of importance. Which are critical to revenue? Which are important but can wait a week or two to be restored? If a function isn’t mission-critical, is it required for normal operations, or can you conduct business without it?

Next, define a minimally acceptable timeframe for recovery for each system and an acceptable amount of data loss for your organization. This is a critical step as it will have a significant impact on budget, preparation timelines and pre-placement of people and equipment. The metrics for downtime and data loss will vary by customer type, size and industry. Some companies (e.g. banks and other financial firms) require zero downtime and loss of data; others (e.g. some manufacturing firms) are less impacted if systems go down for a while and can survive some data loss.

Bear in mind that the more business functions placed in the mission-critical category, the more costly the data recovery—so plan judiciously. If you’re considering outsourcing data recovery, include vendors and fees in your DR budget; also factor in costs over several years—different DR approaches have different cost structures over one, two or three years. Your plan should also reflect any "last-resort" recovery vendor costs in the event of a disaster impacting both primary and backup systems.

Pre-placement of backup systems

Consider your hardware needs as well. Restoration of a mission-critical function such as product sales may require that you pre-place laptops with charged batteries, customer records and user names and passwords. Stock necessary backup hardware (e.g. servers, laptops, power generators, cell phones—even flashlights) in secure, off-site storage areas so that your recovery team can get to the backup systems if the power goes out for extended periods of time.

Above all, document the DR plan thoroughly and share it internally. Give every individual involved in the recovery operation a hard copy. Make sure they’ve read it and understand their roles. Conduct regularly scheduled drills with the DR team, including their designated back-up personnel, to ensure that the plan can be executed in a real-world scenario.

The DR plan must also be a dynamic document. As business conditions change, update the plan—a network upgrade, a new data center overseas or closing a business site could change your recovery strategy.

While no DR plan provides ironclad protection, your company will be far less vulnerable to an extended business disruption from a disaster with a plan than without one.

If you already have a plan in place, dust it off, check it for best practices and run it through the paces. Your company’s life could be at stake.

^1.Faulkner Information Services
^2.U.S. Bureau of Labor
^3.CNET News.com, September 2, 2005. “Recovering data in the post-Katrina Gulf Coast.”
^4.Harvard Business Review, May 2002; “How Resilience Works.”
^5.CIO Insight, September 20, 2005; “Katrina: The Ultimate Testing Group for Disaster Recovery.”
^6.E-week, September 2, 2005. “Disaster Recovery Plans Fight Chaos."