Friday, December 21, 2007

Okay data center

We have a web hosting company in the San Francisco bay area. We used XO communications because their sales person was professional and provided a very good deal at the time. This was nearly 4 years ago.We were treated very nicely. The data center folks were definitely courteous, professional and provided the service as expected. We had initial routing issues when setting up but it was resolved within the same day. We had a committed connection of 10 Mbps with a 42U cabinet with 20 AMP power circuit. Everything ran as expected and we were happy for the last 3.5 years. However, all is well, does not end well. In the last 6 months, we have had to fight XO on nearly everything. XO initially quoted us $$$$ for the colo fees. This contract should not have been open-ended and the prices should have remained the same. However, due to their lack of understanding power consumption, they misquoted us and we were forced to pay $$$ in addition to our colo fees. This happened in July. So, we complied and began to pay the new fees, despite the fact that we had agreed to a lower fee upon contract agreement. I asked them where I signed the dotted line (me being in charge of this) that would allow them to increase the fees without our agreement. They could not answer; in addition, I did not sign anything to that degree. I left that one go...SeptemberWe requested remote hands reboot of a server with the label SERVER-L23. The customer service rep on the phone evidently heard us incorrectly and did not confirm the server hostname. This caused the data center technician to not reboot the server for 2 hours. While we should have been on top of this, we remained oblivious to the fact that they hadn't rebooted the server, thus it remained down for duration of 2 hours. When we confronted the technician and asked her (Monica) to explain why she did not follow-up, she replied with: "I want to let you know that I couldn't find the server with that label, and I'm the only one in the data center right now"Okay, so I can understand why she couldn't find the server hostname despite the fact that: A. ALL servers are labeled clearly with a label maker on the front and backB. That was the ONLY server name that had the letter "L" and number "23"But what I don't understand is why in the world, an entire data center is using one technician, and one that's not very competent...I asked their manager of the data center to explain why this happened. Oh wait, their data center manager in Fremont, ca is actually in Irvine, Ca. Unaware of what really happens in Fremont (200 miles away). I asked her for a credit on the downtime, but that fell on heavy ears.OctoberWe requested a power cycle of a host named SERVER-WVPS10. The customer service rep on the phone recorded SERVER-WETS10 and didn't confirm on the phone. I can understand that mistake. The data center technician took 1 hour to get to the server, and then rebooted the wrong server, named SERVER-W10. Now, I can understand if you're busy and can't get to the server seeing how they only have one person in the entire data center. What I don't understand is that there is no server named SERVER-WETS10 so why in the world would this person reboot SERVER-W10, when it wasn't even the right one to begin with. To that end, he rebooted a database server that caused replication to break, the cluster to break and the database locks to be defunct, all leading to data loss. Again, this is all despite the fact that: A. ALL servers are labeled clearly with a label maker on the front and backB. That was the ONLY server name that had the letter "WVPS"But I guess the number 10 caused the idiot to get confused...When I asked their manager why this happened, she just ignored my emails and calls. I asked for a credit on this incident and nothing happened. NovemberTheir entire data center goes dark for a whole hour. Our servers fail; we lose uptime for an entire hour and not to mention that we had to run a series of diags to fix broken server software, etc. When we asked them for credit, we were told that "the network was down for a minute". Umm... I just spoke to the Seagate admin regarding this and they got a credit for this. So what gives XO?DecemberTheir entire datacenter losses network and all cooling function. This causes the temp to rise close to 100 degrees. We had to shutdown all machine really quickly. A whole 2 hours later and we were back up again. 2 of our core virtual servers were defunct after boot up and it took us nearly 10 hours to recover, answer emails, phone calls and issues refunds for the outage. We also lost close to 10 well paying customers in the duration.They finally admitted to their fault and gave us credit for this one. Suffice to say, we have terminated our contract with them and are in the middle of moving away. They're currently trying to get us to pay $$$$ for the past invoices but we refuse on the counts of incidents above. So, if you want to spare yourself from the headache and agony of dealing with this sort of issue, please don't sign-up with XO.On a final note, when we spoke with our sales person, he mentioned that we were a profitable customer. Yet despite our numerous efforts to get them to realize that they were going to lose a “profitable” customer, they chose to ignore our emails, phone calls and complaints. I guess that’s the XO business model.

No comments: