Wednesday, June 6, 2012

Business Owner vs Business IT vs IT Supplier - disaster recovery fun#2

Few weeks ago, I shared a story about the impact of not having Disaster Recovery. And I mentioned that there will be a sequel to it. So, here it goes.... the saga shall continue....

Business Owner guy: <* Addressing VIP business users *> It is very unfortunate that we were badly hit by the quake. As it was an act of God, we have to accept the loses etc etc.... . Nevertheless, we have come out with a great disaster recovery plan to ensure business continuity in the event of disaster. This time, we WILL be ready to face it!

Information Security lad: <* This guy is surely a great politician.... I wonder how he saved his ass, survived and resurrected from that gigantic mess... Last I heard, he even got a promotion*>  

5 months later......

Business Owner guy: <* shouting over the phone *> What's happening? Why can't my customer access the application?

Business IT bloke:  The whole IT Supplier's data center is currently down. It has something to do with the earthquake that happened 10 minutes ago. Seems like the data center was badly hit by the quake. I managed to get hold of the service manager and he has arranged an emergency meeting in 15 minutes to update us on the situation.

After 15mins, in the emergency meeting...

Business Owner guy: IT Supplier chap! Why my application has not been recovered yet? According to our SLA, my application should have been recovered 1 hour ago! I paid a lot of money for this DR stuffs and what takes you so long?

IT Supplier chap: We doing our best here sir. We followed the recovery steps and procedures but it didn't work. These are the procedures your Business IT colleague gave us. It seems like the procedures were missing some other steps. We are still figuring it out.....

Business IT bloke: That does not seem right! We were thorough when we did that procedures and we subsequently did perform a walk through test as well.

Information Security lad: Sorry to interrupt guys. Did you guys perform a live test?

IT Supplier chap: Not really... we supposed to do one and we did propose to do the live testing but we couldn't get agreement from Business IT and Business Owner.... something related to time and budget...

Business IT bloke: Mr. Business owner... we did receive that proposal from IT Supplier... but we couldn't get agreement from you to proceed. You mentioned about lack of time and budget etc...

Business Owner guy: <* start sweating....*> I thought we covered all that during the walk through test..

Information Security lad: Not really sir, for a disaster recovery to work, you not only need to have a good disaster recovery plan, but most importantly you MUST also TEST the plan to ensure that it works. Not only virtually test the plan, but test the plan via a live testing as well. The live testing will ensure that the plan actually works.

Moral of the story?
1. You have to look at the disaster recovery process from end to end. Not just creating procedures and hope it works. Live testing is a very crucial component within the whole Disaster Recovery process. It ensures that the plan did work. Don't miss it. 

Ain't security fun? ;) 
Acknowledgement - photo taken from

No comments:

Post a Comment