Full-Scale Exercise: How to Conduct a Disaster Recovery Test
Hollywood loves to detonate large-scale disasters on the silver screen. Buildings crumble, cities flood and thousands flee for their lives. It’s a proven recipe for movie success—mayhem crafted to keep the audience in thrall.
FAST FACT: While 80% of IT organizations have a disaster recovery plan, only 40% test it regularly, according to InformationWeek’s 2013 State of Storage survey.
DID YOU KNOW: The top areas of concern among DR planners, according to a 2012 survey by the Business Continuity Institute, are: unplanned IT and telecom outages (74%); data breaches (68%); cyber attacks (65%); and disruptive weather events (59%).
For an enterprise IT executive, however, the prospect of even a tiny disaster is far from entertaining. Any hiccup in the data stream could set a company back—in time, money and reputation. While every technology manager worth his or her salt has a disaster recovery (DR) plan, not every manager tests those plans to ensure recovery efforts work flawlessly. In fact, only half of IT organizations with a disaster recovery plan conduct regular testing, according to InformationWeek’s 2013 State of Storage survey. The best-laid recovery plans count for nothing if they don’t deliver when needed.
Here are key best practices to consider for testing—and fine-tuning—your DR playbook.
Best Practice #1: Focus on human factors.
Sure, in terms of ensuring data availability, you’re dealing with technology like tape backup systems. But you need to be certain your people will swing into action smoothly to recover from a catastrophic outage. Central to any DR plan is determining who does what and establishing a sequence of tasks during emergencies.
Your plan should also counter circumstances that might waylay your staff. For example, if a storm prevents your two top data center managers from reaching your business location, backup personnel must already be designated to run point. Balancing these contingencies is an essential part of regular disaster recovery testing. When you run a drill, you may discover that employees have changed roles or even left the company since the DR plan was last updated. You don’t want to learn about the West Coast office’s reorg with an earthquake, as they say, “earthquake’s already in progress.”
Best Practice #2: Retain the element of surprise.
Enterprises typically maintain two kinds of DR test plans: regularly scheduled rehearsals and unannounced drills. Quarterly walk-throughs help keep everyone on the same page, using checklist items such as verifying failover mechanisms for applications and data systems. That’s a given. But unscheduled recovery simulations may knock those individuals off the same page—which will make your plan stronger. Obviously, you’ll want to carry out any surprise test carefully—typically outside of regular business hours, so the test doesn’t cause business downtime.
Best Practice #3: Learn from your testing snafus.
Did someone (or something) drop the ball during a run-through? That, too, is great news: It’s better to discover what doesn’t work before floodwaters are lapping at your doors.
As part of the postmortem audit, analyze each phase of the testing cycle to find areas in need of aid. Consider whether additional training will benefit the disaster recovery team and their backup counterparts—and if so, exactly what type of training. Clearly, nobody has time for a crash course in database administration during a disaster, but you can bolster basic skills ahead of time.
Best Practice #4: Bring outside partners into the game.
Your company depends on a slew of service providers and vendors every day. So try to include them in your disaster-recovery tests. Do they actually have the resources they promised when you signed the contract two years ago—ready to roll? If they can’t keep their guarantees or meet your recovery-time objectives during testing, it might be prudent to switch.
The ultimate goal of your DR testing routine is to ensure that your organization doesn’t have a starring role in the next ripped-from-the-headlines disaster flick. With diligent, methodical testing, you can be the hero who saves the day—in a world, as they say in movie trailers, of chaos and uncertainty.
Iron Mountain Suggests: Find a Workout Partner
You’re more likely to stay physically fit with a personal trainer guiding your exercise regimen. Likewise, an enterprise benefits from teaming with a trusted third-party partner when testing its disaster recovery strategies. Iron Mountain’s range of disaster recovery services can help you:
- Review threats to your business and analyze the potential impact on operations
- Test recovery by transporting backup tapes to your recovery locations as part of a simulated disaster recovery exercise
- Evaluate DR test results to address gaps in the recovery process and determine why each failure occurred
Do you have questions about records and information management? Read additional Knowledge Center stories on this subject, or contact Iron Mountain’s Information Management team. You’ll be connected with a knowledgeable product and services specialist who can address your specific challenges.
Holy 2033: Tape Travels Back to the Future
Why Tape Rules: Using Tape Backup in the Cloud
Remote Controls: Shaking Off the Branch Office Blues