[outages] Dreamhost MySQL outage?

[Martin Hannigan]

The RFO is irrelevant. The damage is done. Maybe there are questions? Most
important is when you got back in business. The rest is an almost total
waste of time. The public stuff will be image oriented. It has to be.
Privately, you all may get more mileage. Contractually? If you know you
know. Many can/should/will get compensation for the damage. I would.
Whether credits or benefits at renewal time. This may be worth exploiting
as a customer. After all, these aren't supposed to go down. Hope it never
happens to (me)! If it does, this is the script.

   - How did all generators fail together?
   - How did all transfer switches fail to shift from utility to generator
   together?
   - How did all the UPS fail all together?
   - Why did the facility come back in pieces?
   - Are you sure you're really an N+ facility and not an N facility? Show
   me. Sign new SLA
   - Why did it take an hour to restore "any" power?
   - How will this not happen again?
   - How do you plan to make me whole?

They know at the end of the day this sucks. And they are embarrassed. They
really do care. Bunch of good people. However, who's leaving customer wise?
That's calculus. Answer? Almost none of you. :)

YMMV

-M<

On Fri, Nov 3, 2023 at 12:45 PM Zach Hanna <chezbut at gmail.com> wrote:

> I was told it was the transfer switches, not the gensets.
>
> But waiting for the RFO
>
> Sent from my iPhone
>
> On Nov 2, 2023, at 5:26 PM, Martin Hannigan via Outages <
> outages at outages.org> wrote:
>
> 
>
> That means a series of parallel gens failed “all together”. The RFO for
> this is going to be uncomfortable.
>
>
> On Thu, Nov 2, 2023 at 09:10 Matt Morgensen via Outages <
> outages at outages.org> wrote:
>
>> We're in Flexential PDX02, I have guys on-site now.
>>
>> Looks like they lost utility, switched to generator, and then generator
>> failed (not clear on scope of Gen failure yet). Some utility power is back,
>> so recovery is in progress for some portion of the site.  I still have ~70
>> racks without power....but things are coming back to life slowly.
>>
>> On Thu, Nov 2, 2023 at 6:01 AM Tom Strickx via Outages <
>> outages at outages.org> wrote:
>>
>>> Flexential PDX02 is entirely down right now. Full power outage.
>>> --
>>> Tom Strickx
>>> Principal Network Engineer
>>> AS13335 - Cloudflare
>>>
>>>
>>> On Thu, Nov 2, 2023 at 12:31 PM Bob Colon via Outages <
>>> outages at outages.org> wrote:
>>>
>>>> All my sites, hosted on one of their VPSes is down hard.    I can't SSH
>>>> in, and it's not responding to pings.   According to their status page,
>>>> their data center on Portland is down hard:
>>>> https://www.dreamhoststatus.com/
>>>>
>>>> On Thu, Nov 2, 2023 at 7:27 AM Max Leonard via Outages <
>>>> outages at outages.org> wrote:
>>>>
>>>>> cloudflare is having issues again (https://www.cloudflarestatus.com/)
>>>>> -- probably unrelated but ::shrug::
>>>>>
>>>>> On Thu, Nov 2, 2023 at 7:16 AM James Lawrie via Outages <
>>>>> outages at outages.org> wrote:
>>>>>
>>>>>> It could be something else, but one of our sites is down because
>>>>>> MySQL endpoint is unresponsive, and attempting to login to contact them
>>>>>> panel.dreamhost.com has the same problem (gives either 503 or times
>>>>>> out).
>>>>>> _______________________________________________
>>>>>> Outages mailing list
>>>>>> Outages at outages.org
>>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>>
>>>>> _______________________________________________
>>>>> Outages mailing list
>>>>> Outages at outages.org
>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20231103/9e284dc4/attachment.htm>