Tuesday, April 24, 2007

Did I do that?

Last Friday I left work and for some reason I was very happy. People normally complain of a lack of or a not so friendly facial expression but as I left work I was smiling. It didn't last long though cause my boss called me when I was about 5 minutes away from home asking me about some problems he was having with a system, I told him I didn't touch it but as I continued on to home I realized I deleted some disks(that were supposed to be unused) from a RAID array we were retiring about 10 minutes before I left work. I called him back at and told him and we tracked back those disks to some convoluted old configuration and realized the shit has hit the fan. He was on standby but he told me his DSL was giving problems so he asked me to fix it. Since it was caused by me I had no problems doing to work to resolve it.

As I investigated how deep my problems went I found out I corrupted a few hundred gigabytes of table space in our largest database[by largest I mean greater than 10TB]. This is what my backup vendor calls a RPE - resume producing event. I laughed when he talked about it back then but now have great appreciation for them. I worked from home recreating the disks, restoring the data and helping the database admin rebuild her database and by Sunday evening I was in the clear, I had restored 90% of my data and the DBA's restore was well on its way.

With all that behind me I went to a friends house to play some dominoes at 9pm to get my weekend started. It has been a while since I've been up there and that night wasn't any old games night this was the last weekend our friend would be in Jamaica for his break visit. It was like old times only thing now we are no longer pauper students and can afford food, I saw a big KFC bucket there and stuffed my face till 3am. It's a shame we have grown up/apart and a lot of the old crew are no longer in Jamaica, we used to do this all the time when we were in university.. the good old days.

I reached home close to 4am on Monday and I went straight back to my VPN connection to put the finishing touches on my restoration work so people could come to work later and actually have applications to use. I went to work at the corporate building at 8:30am, I spoke to some of the users, explained to them what happened.. nobody noticed a thing! I am good and safe :p. I didn't spend long there and left for my office downtown where I can get some quiet and privacy. After I docked my laptop I went to look for my boss to give him a status update and everything was all good :D In fact things were more than good, when I went back to my desk I checked my email and saw he sent my 6 month appraisal for me to go over and I scored 4.15 out of 5, now I can get my bonus in the middle of the year when everybody else gets a Christmas bonus :(

8 comments:

Dee said...

whew
good job Adrian
you can do that again
j/k

Jhaldir said...

Its incredible just how much power we have in the IT/Technical world isn't it? One simple, careless mistake by me and I could take down an entire corporate Insurance giant. Ho hum.

Mad Bull said...

Doesn't sound like a fun way to spend a weekend. I've been there, man, and it wasn't fun. Glad you got through it all ok.

Adrian said...

@fiyah.. lol I don't think the power to lose one's job is an incredible power to have :p i'll the power to look into the future any day over this heh.

Anonymous said...

differently still adrian.. your a true geek. your weekend involved work.. and you say.. great weekend.

a nuh nothing still its the story of my life. in job fields like this work is always with you cuase you can connect from ne where you are and do what you need to do.. last night i got an email stating an error that just showed up in an application i built (a scholarship repository) apparently when i did the last update the indexes that were being collected got thrown out.. (i replaced 2 tables in the db which were being indiex for a full text search query i made). i fixed it right away before giong to bed...

Nadya Dee said...

so about the bonus now... sharing is caring :)

Crankyputz said...

congrats on the bonus and wonder work

Anonymous said...

Well, I'm not in IT in the general form of the term, but I work on very large and very expensive computer systems... I understand these RPE's of which you speak...

Glad you not only avoided one, but also scored well on your appraisal... Enjoy spending your bonus.