Page 1 of 2 12 LastLast
Results 1 to 25 of 40
  1. #1
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    The Planet - Houston Datacenter Down Due to Fire
    For anyone hosted on The Planet's Houston Datacenter (many former EV1Servers / RackShack customers), they're down. (All three of my servers are there.)

    They had a transformer catch on fire in their electrical room. They had to shut down power to building, evacuate, and have the fire department put out the fire. No servers were damaged, but they're currently evaluating what needs to be done to restore power.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  2. #2
    Best New ABW Member 2007 sfcom's Avatar
    Join Date
    October 9th, 2007
    Location
    Columbus, OH
    Posts
    1,705
    Sorry to hear about that. It is never easy to have downtime, but to know that nobody was hurt and everything is going to be OK is a big plus.

    -sfcom


  3. #3
    ABW Ambassador CathyM's Avatar
    Join Date
    May 30th, 2006
    Location
    Torrance, CA
    Posts
    893
    Michael,

    Thanks for the heads up. I have some sites hosted there as well. I have a back up host I could move the sites to but not sure yet whether to do that or to wait it out.

    I feel for you and hope your sites are back up soon.

    Best,
    Cathy

  4. #4
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    Here's their latest update:
    From The Planet CEO Doug Erwin:

    This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

    We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

    This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.

    We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.

    There is no impact in any of our other five data centers.

    I am sorry that this accident has occurred and I apologize for the impact.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  5. #5
    ABW Ambassador CathyM's Avatar
    Join Date
    May 30th, 2006
    Location
    Torrance, CA
    Posts
    893
    It's going to be a while before they are back up.

    http://forums.theplanet.com/index.php?showtopic=90185

    Edit - Posting at the same time you were Michael.

  6. #6
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    4,423
    It is times like these that make you find out where your servers live... only a couple in that datacenter at the planet, the rest are fine...

    With the fire inside, I wonder how long these will be down?
    EDIT - thanks for the update guys.

  7. #7
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Wow, I didn't hear the full story until I saw this thread. I've been going back and forth with support staff with form responses. My site has been down for nearly 6 hours which is a disaster in my world. I can only hope Google doesn't slap everybody who was affected by this. It's a very scary thought hearing my site won't be back until tomorrow, I really hope they get it up sooner than that.

    In the mean time I need to figure out if I'm going to switch my IP address to another server..

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  8. #8
    Life is Supposed to be Fun! Rexanne's Avatar
    Join Date
    January 18th, 2005
    Location
    Los Angeles
    Posts
    12,360
    Wow guys, this is a big drag. So sorry - I can imagine what you're going through. :-(

    I have no idea where my servers are - afraid to check my sites. LOL

    Hope you're all back up and running before tomorrow!
    Peace,

    Rexanne

    Rexanne.com
    Loving Everyone's Child Creates Magic


  9. #9
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    I doubt it'll be any sooner. They're trucking in parts in from Dallas and I'm sure they have to get an inspection and the go-ahead from the fire department before they power up. As long as it took to get an ETA, I'm sure their ETA will be pretty close.

    This will be a major financial hit for them. Their SLA refunds 3.5% of the monthly hosting fee for the first hour of downtime plus an extra 3.5% (up to 100% total) for each additional 30 minutes. By my calculations, anything over 15 hours of downtime (and it'll be a bit more than that if they're "afternoon" guess is right) will max it out. With 9,000 servers in the datacenter, if they're paying $115 each (and that's probably low), that's over $1 million.

    Of all the times for an outage, it could have been far worse for me. Summer is the slowest time of the year. Saturdays are my slowest day of the week. Most of the outage will be overnight, which is slow. I'll lose more than what I'll get back with the SLA, but the SLA will make a dent in it.

    In the years I've used RackShack/EV1Servers/ThePlanet, they've been quite reliable. This outage will be longer than all of my other outages put together. My previous host had outages longer than this almost every month. No complaints here.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  10. #10
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    4,423
    Same here. For all the years of rock solid performance, I can't hold this one against them.

    Added bonus for me being mr smarty pants on some domains and not using my own name servers but ev1s... which happen to be in that data center. While I am sure they will move them at some point sooner than later, I made the change myself.

    Wish I could just access my site, not the public, could get some spring cleaning done.

    Chet

  11. #11
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Wow, some interesting stats there Michael, thanks for doing all that math. The sad irony here is I *just* switched IP's to this new server. My old server is going offline tonight (which is in an un-affected datacenter). Another server of mine is fine, but the one that's down is my most important server.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  12. #12
    Antisocial Media Expert ProWebAddict's Avatar
    Join Date
    March 25th, 2006
    Location
    Go Daddy
    Posts
    1,109
    It's cases like these where I pat myself on the back for being so paranoid and spreading my sites out among 3 companies.

  13. #13
    ABW Ambassador SunshineTricia's Avatar
    Join Date
    May 7th, 2006
    Location
    Indianapolis
    Posts
    896
    We have our forum and a couple of blog sites hosted there. Thankfully our main sites are hosted elsewhere. We're going to go ahead and move our forum to another server this morning using the backup from Friday night. We'll lose Saturdays posts, but it's better than nothing. We may put it in "maintenance mode" so no one can post but everyone can see all of the old posts. I don't know. We'll be anxiously watching the updates as the morning goes on.
    --Tricia Meyer-- I love being the exception to the rule.

    Tricia Meyer | Helping Moms Connect | Wine Club Reviews and Ratings | Hunger Games Fan

  14. #14
    Beachy Bill's Avatar
    Join Date
    November 20th, 2005
    Posts
    8,266
    Quote Originally Posted by ProWebAddict
    It's cases like these where I pat myself on the back for being so paranoid and spreading my sites out among 3 companies.
    I'm with you on that, PWA. I've got sites hosted in New Jersey, Virginia and Texas (thankfully, not at The Planet today).

    My thoughts and best wishes go out to my fellow ABWers who may be "down" at The Planet. This is a frustrating and expensive time. The Planet does have a fine reputation but they got hit with an unforeseen circumstance. You never know what can happen next - or to whom.

    Right now I am doing a private alpha test for a new service - and the server is located in Virginia. When we get into the public beta we plan to have redundant servers in two locations.
    Bill / Marketing Blog @ 12PM - Current project: Resurrecting my "baby" at South Baltimore..
    Cute Personal Checks and Business Checks
    If you are too busy to laugh you are too busy.

  15. #15
    Moderator
    Join Date
    April 6th, 2006
    Posts
    2,689
    I just noticed that Statcounter has been affected.. service is down..

    For anyone impacted, I hope your site comes back soon..

    These unforeseen outages are frustrating, but from my own experiences, traffic shouldn't be affected - it won't be long enough for google to even notice, and regular visitors will just try again later

  16. #16
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    If anybody wants to move a hosting account temporarily during the outage let me know ASAP. I'll set up a site for you on a server of mine.
    ---
    This response was masterly crafted via the fingers of Patrick Allmond who believe you should StopDoingNothing starting today.
    ---
    Focus Consulting is where I roll | Follow @patrickallmond on Twitter
    Search Engine Marketing | Search Engine Optimization | Social Media | Online Video

  17. #17
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Still nothing

    I'm trying to load their forum for updates but it's not loading properly for me. I sure hope they get things running soon!!

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  18. #18
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    12:24 PM central update from their forum
    ---
    To keep you up-to-date, here is the latest information about the outage in our H1 data center.

    We expect to be able to provide initial power to parts of the H1 data center beginning at 5:00 p.m. CDT. At that time, we will begin testing and validating network and power systems, turning on air-conditioning systems and monitoring environmental conditions. We expect this testing to last approximately four hours.

    Following this testing, we will begin to power-on customer servers in phases. These are approximate times, and as we know more, we will keep you apprised of the situation.

    We will update you again around 2:30 p.m. this afternoon.



    -----
    Urvish Vashi
    Director, Product Management
    The Planet
    ---
    This response was masterly crafted via the fingers of Patrick Allmond who believe you should StopDoingNothing starting today.
    ---
    Focus Consulting is where I roll | Follow @patrickallmond on Twitter
    Search Engine Marketing | Search Engine Optimization | Social Media | Online Video

  19. #19
    Moderator
    Join Date
    April 6th, 2006
    Posts
    2,689
    From the forum, as of 5:02pm:

    We continue to pursue our plans to provide initial power to our H1 data center this evening. It will take several hours to assure power can be safely restored to the facility. Based on how the initial work goes, we will have more information to provide you in the upcoming hours. We will post another update by 7:30 pm tonight.

    In the meantime, we have further rerouted both old and new IP addresses for our name servers previously housed in H1. This means that the servers can start resolving IP addresses on both their former and new addresses, and this will alleviate the issues we have been seeing with propagation delay from this address change.

    As always, please continue to monitor this thread or contact our support teams if you have any questions.

    ------
    Urvish Vashi
    Director, Product Management
    The Planet

  20. #20
    Advocate mellie's Avatar
    Join Date
    January 18th, 2005
    Location
    Here
    Posts
    1,925
    How horrible. Hope your sites are all back up soon!
    Melanie
    President - Affiliate Advocacy 2008 ShareaSale Performance Industry Advocate Award, 2009 Affiliate Summit Pinnacle Award - Affiliate Advocate
    Affiliate Advocacy
    NYAffiliateVoice Seery Writing

  21. #21
    Moderator
    Join Date
    April 6th, 2006
    Posts
    2,689
    Statcounter is back online & tracking visitors... which means The Planet has restored service..

    Hope everyone is back online now!

  22. #22
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    Won't be that fast. They will slowly start turning things back on and checking them. Even with the servers back on I'd bet they'd keep the outside network away for awhile. But that is a good sign.
    ---
    This response was masterly crafted via the fingers of Patrick Allmond who believe you should StopDoingNothing starting today.
    ---
    Focus Consulting is where I roll | Follow @patrickallmond on Twitter
    Search Engine Marketing | Search Engine Optimization | Social Media | Online Video

  23. #23
    Moderator
    Join Date
    April 6th, 2006
    Posts
    2,689
    I meant that the statcounter.com website is actually back up now.. and tracking new visits. I'm seeing this from the outside..

    This means general connectivity has been restored (for some sites anyway)...

    Pulled this from their site:

    http://forums.theplanet.com/index.ph...ic=90185&st=20

    "There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running.

    We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we're expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production.

    Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers. The news is not as good, and we were not as lucky. The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary. We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system. During the upcoming week, we will be working with those customers to resolve issues.

    We know this may not be a satisfactory solution for you and your business but at this time, it is the best we can do.

    We understand that you will be due service credits based on our Service Level Agreement. We will proactively begin providing those following the restoration of service, which is our number priority, so please bear with us until this has been completed."

    Tough times, but they've handled the communication side very well..

  24. #24
    I like traffic lights
    Join Date
    January 18th, 2005
    Location
    Southern hemisphere - away from Fukushima
    Posts
    2,936
    I suggest statcounter.com is now running from their Dallas datacenter.

  25. #25
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    All three of my servers are back online. I'm working through a few small issues, but my sites are up.

    If anyone sent me (or anyone affected) any email over the past couple days, it probably bounced. If it was important, please resend.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

+ Reply to Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Wake up the CJ Datacenter Guys
    By sam_park in forum Commission Junction - CJ
    Replies: 5
    Last Post: November 24th, 2007, 12:40 PM
  2. no report for Planet DVD and Planet CDROM
    By wangchuk in forum ShareASale - SAS
    Replies: 1
    Last Post: June 21st, 2005, 12:23 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •