Our (mt) Media Temple fiasco

3 10 2008

Update: Media Temple sent an email apology regarding this. The full names of (mt) customer service representatives are now withheld as requested by Media Temple to protect their privacy. I agreed and removed their complete names.

Update 2: Websites seem to be up now. I got this report from Montastic. Thank you.

Website status: OK

Date: Sat Oct  4 07:26:26 2008
URL:  http://pro.inboundpass.com

~~~~
Yesterday, our site inboundpass.com suddenly went down. What looked like a simple problem solving issue, we’ll-quickly-bring-it-back-up by the Media Temple staff rendered our very busy site inaccessible for almost 5 freaking hours. Media Temple, you screwed us up big time. And also my other websites are still down for the past 13 hours.

So to dispel the rumors, no our site was not hacked. It was a stupid admin bubu.

This is a very bad time to go down. When thousands of people are looking at the site. This season has caused our server to creak under pressure and we’ve seen more than 700% increase in site traffic. Woah! (This is reserved for another blog post.)

So what happened?

  • At September 30, 10:42 PM (PDT) That’s approximately 1:42PM (+0800) Manila Time

REQUEST ID #449907 / STATUS: In Progress
downgrade from MySQL Grid Container

David Quitoriano (Administrative Contact)

Hi (mt), Thank you for the letting us try the MySQL Container for a few days during this spike of traffic.

We are looking at other options and will not not be continuing the MySQL Container Lite.

Please DO NOT upgrade automatically. Thanks.
@
2008.09.30 10:42 PM

So far so good.

  • At October 1, 4:25AM (PDT) (Approx. 7:25PM (+0800) Manila Time) I got relatively prompt response from Mike.In regards to your inquiry,The burst container process is a automated process and can not be manually controlled or edited for one specific client. As long as you are not using a tremendous amount the MySQL container you should not have any problems. Please keep in mind that if you get bursted three times you will be required to purchase a MySQL container.

    If you have any further questions regarding your (mt) Media Temple services, please feel free to contact us at any time.

    Best Regards,

    Mike (Update: Lastname withheld for privacy)
    Customer Support
    Check out the New User Forums: http://kb.mediatemple.net/questions/824/
    (mt) Media Temple
    <v> 877-578-4000
    <f> 310-564-2007
    @
    2008.10.01 04:25 AM

Still good.

  • The next day I was irked to received this email–On Oct 2, 2008 at 3:05 PM, (mt) Media Temple <no-reply@mediatemple.net> wrote:Dear David,Your new MySQL GridContainer Lite (srv# 289172) has been successfully activated and is now available for use with your (gs) Grid-Server (coloredcompany.com). All of your MySQL databases have automatically been moved into your new GridContainer environment. Your databases are now being served exclusively from within your MySQL GridContainer. No changes to your applications are needed to work inside the MySQL GridContainer. Your databases can still be managed the same way as before directly from inside the AccountCenter or using remote tools. In addition new tools which give you greater control and insight into your database have been added to the MySQL Container tab inside your AccountCenter Database Management Controls.

    If you require the assistance of a support agent, please start by opening a request inside the Account Center: https://accountcenter.mediatemple.net/

    Your request will be routed to the appropriate agent or group on our team.

    Best Regards,
    (mt) Media Temple
    Hosting Operations
    @
    2008.10.02 03:36 AM

Very good, just as I thought what would happen.

Now to lend you a piece of my mind. Media Temple was courteous to support us in our spike of traffic during the UAAP & NCAA Championships. They “temporarily” placed us on the MySQL Grid Contrainer Lite which costs monthly at $20. So all in all our monthly hosting at Media Temple would increase to $20 (gs) + $20 (Grid Container) for a total of $40.

Now at that rate why not get a (dv) Dedicated-Virtual $50 or a Slicehost at $38 a month. Gimme time to think it through but don’t pressure sell me.

So what ensued was a string of message exchanges to supposedly cancel this arrangement.

  • I sent a reply, David Quitoriano (Administrative Contact)

Ok then un-automate it.

I received this already. Thanks! (referring to the automated email I received which I copied below for their reference.)

~~~~
On Thu, Oct 2, 2008 at 3:05 PM, (mt) Media Temple <no-reply@mediatemple.net> wrote:

Dear David,

Your new MySQL GridContainer Lite (srv# 289172) has been successfully activated and is now available for use with your (gs) Grid-Server (coloredcompany.com). All of your MySQL databases have automatically been moved into your new GridContainer environment. Your databases are now being served exclusively from within your MySQL GridContainer. No changes to your applications are needed to work inside the MySQL GridContainer. Your databases can still be managed the same

Yada. Yada. Yada.

~~~~

  • Now for the kicker replies:

Grace (Lastname withheld for privacy) (mt) Media Temple Customer Support

There is no way to un-automate the burst container, it cannot be manually controlled, as what the previous technician stated before.

Best Regards,
Grace
@
2008.10.02 04:03 AM

~~~~

David Quitoriano (Administrative Contact)

Ok Grace.

Let me be very clear. We will NOT purchase an additional MySQL GridContainer Lite Container (srv# 289172).

So it’s an automatic process. Nevertheless, we will decline in getting an additional MySQL GridContainer Lite Container.

Cancel Service Order # 289172.

Thank you.
@
2008.10.02 05:04 AM

~~~~

Grace (Lastname withheld for privacy) (mt) Media Temple Customer Support

Order number: 289172 has been canceled per your request.

Best Regards,

Grace

@
2008.10.02 05:16 AM

~~~~

BOOM!

At this point tragically, inboundpass.com went down at a blink of an eye. Cold sweat on my brow. A growling WT*? tear through the silence of the premises.

Hello? Are you mad? Are you our of your mind? I guess, YES!

In frustration some more exchanges:

  • I sent this reply

my website is now DOWN. http://inboundpass.com

thank you very much
@
2008.10.02 05:27 AM

~~~~
Grace (Lastname withheld for privacy) (mt) Media Temple Customer Support

You requested the cancellation, I did it for you.

If you need the service back, I can add it back.

Best Regards,
Grace
@
2008.10.02 05:35 AM

~~~~
David Quitoriano (Administrative Contact)

I cancelled the MySQL GridContainer Lite. If a service will go offline you should have given a heads up. I don’t know the inner workings of (mt).

1. I want http://inboundpass.com up immediately.
2. We would like the default MySQL that’s included with (dv) Hosting. <- I meant (gs) hosting
3. Cancel the MySQL GridContainer Lite – Service Order # 289172

Please.
@
2008.10.02 05:54 AM

~~~~
Drew (Lastname withheld for privacy) (mt) Media Temple Customer Support

The database is now reachable via the phpmyadmin page on your account, so all of your sites should be working.

If you have any further questions regarding your (mt) Media Temple services, please feel free to contact us at any time.

Best Regards,
Drew
@
2008.10.02 09:10 AM

~~~~

Thank you. inboundpass is now up. We were down for almost 5 hours by my estimates. My other sites are still down.

~~~~

David Quitoriano (Administrative Contact)

Hi Drew,

Thank you for bringing up http://www.inboundpass.com. Our main site went down for more than 4 hours.

All my other sites are down as well, please revert the databases :
http://www.air21express.com.ph
http://dev.inboundpass.com
http://h2orepublic.com/index.php

http://livewire.ph
http://pmxaudio.com
http://staluciarealtors.com

(site list trimmed)

Please fix these asap to reverse this fiasco 😦
@
2008.10.02 01:57 PM

As of this writing, I hope they’re resolving the issues. My other sites are still down.

(mt) Media Temple is NOT great!





Catch us at www.inboundpass.com

20 09 2008

Hello UAAP & NCAA sports fan!

We have moved servers to www.inboundpass.com and we haven’t been updating here.

Do remember to add us in your feeds and follow us on Twitter for the latest updates.

So enjoy and let the games begin.