PDA

View Full Version : Nub Live outage this past weekend


tekeeladude
06-02-2008, 06:33 AM
Hey guys, just wanted to give everyone a little info about the outage this weekend. The web hosting company had some major problems. We will be looking at options to prevent this from happening in the future.

The following was posted by the web hosting company on Saturday May 31.

This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.

NubLive
06-02-2008, 07:51 AM
Man it totally sucked. Tekeeladude called me Saturday night and asked me if I was doing maintenance. I looked and then called the server company and got few answers at the begining. The kept saying to watch the the following thread that would be updated hourly.

http://forums.theplanet.com/index.php?showtopic=90185

One of our servers was affected by the power outage and it happened the it was the one with CigarLive.com and NubLive.com.

I kept thinking that the power to it would be back on soon so I did not want to move servers. I knew propigation of the DNS could take 3 hours so as long as the power would be up soon it was not worth it to move servers. Too bad I was totally wrong from the get go and that 3 hours would have been better than to keep waiting more than a day.

We have offsite back up for moving if needed but we could not even hit the system they have to change the DNS if we had wanted to.

I really wish I had a list of email addresses to hit everyone up on what was going on. This will definately be a future plan. As for now I heard something about that Tekeeladude wants to run 12 T1 lines to my house and house a back up server in his house in Nebraska.

I am really sorry for all of this guys. I did smoke some cigars while I waited. Only good think that came out of it.

CSMAnderson
06-02-2008, 09:08 AM
Well, I guess Murphy has found the NUB BB. :-)

skipper
06-02-2008, 09:43 AM
glad all is back to normal.

When I found I was unable to reach this site, then cigar live too, I realized it must be a provider problem.

shroom91
06-03-2008, 01:58 AM
agree with skipper...i was lost i had nothing to do at work no web site to visit..lol

matt257
06-03-2008, 02:45 PM
agree with skipper...i was lost i had nothing to do at work no web site to visit..lol

I was lost too. It was my day off and no CL and NL. It was a bad day :( lol

Glad its all up and working now. Lets hope it doesnt happen again. Thanks for the update guys