Wednesday, July 4, 2012

Information Security screw-up #1 - security vs uptime

Well, after reading through all of my previous stories, some of you may have had the feeling that I'm telling these stories to show that Information Security professional is always right. No, not really... we do make some mistakes... like everyone, we learn from experiences as well.

This is a story about how one Information Security lad screw-up, when "he was young" :) 

Once upon a time, there was this one "young" Information Security lad who just joined a quite successful .com company. As the new Information Security Manager, he felt like he was the town sheriff and everyone got to listen to him when it comes to security matters. 

It was a weekend, a nice weather weekend indeed, when he received a call from his company. Apparently, there was a virus outbreak in his company. He was called back to office immediately... 

Young Information Security lad: After some investigations, I found the source of the outbreak. It's coming from this server called E-pay. You shall take down this server immediately and have that malware cleaned right away!

Server Admin folk: That would not be a good idea. We can't just shutdown a server like that. I don't think the application owner would be happy with that.. we need to...

Young Information Security lad: <* interrupting *> This is a serious security issue! We must stop this before it spreads around! Shutdown that server immediately!

Server Admin folk: Errr.......

Young Information Security lad: Damm! Give me that keyboard! <* snatched the keyboard and clicked the shutdown button *>

Server Admin folk: What the heck???? <* stunned and speechless *> 

Guess what happened next? No reward though.... 15 minutes later, the young lad got called into the CEO office... 

Young Information Security lad: <* The CEO must be thinking to personally thank me for my fast reaction in containing the security incident :) *> 

Mr.CEO: WHAT WERE YOU THINKING! You can't just shutdown a server like that! This is not your old 5 people company! Do you know what kind of server is that?

Young Information Security lad: <* shocked and surprised by the unexpected reaction *> I don't know.... but it doesn't matter because it was a security incident and we need to react fast to contain it.

Mr. CEO: Damm! No! You just can't do that! Did you how much money the company just lost? That downtime of 15 minutes cost us 15K! 15K in 15 minutes! That is 1K per minute! Even I don't earn that much! The server you shutdown is our main payment server. This is a very peak business period for us, people are busy doing online shopping!

Young Information Security lad: <* start to realize why he screwed-up *> Sorry.. next time I would be more careful...

Mr. CEO: WHAT? Still got next time?

Young Information Security lad: Sorry sir.. I meant it won't happen again...

Mr. CEO: This is your first and last warning. If you screw-up again, you can pack your bag and go. Get out of my office. I don't want to see your face now!

Young Information Security lad: <* almost pissed in his pant *... One hard lesson learnt...>

Moral of the story?
1. Information Security incident needs to be addressed, but not without considering the business process  and impact first. Keeping critical service/business running is always of higher priority.
2. During a security incident, we, as an Information Security professional should work closely with the business to find a balance solution between containing the incident and keeping the service running. It's not easy but doable.
3. Information Security Manager cannot be playing the role of town sheriff but should act as a consultant instead. 

Ain't security fun? ;)

Acknowledgement - picture taken from

No comments:

Post a Comment