[Outages-discussion] [outages] Dreamhost MySQL outage?

[Peter Beckman]

https://aws.amazon.com/blogs/mt/why-you-should-develop-a-correction-of-error-coe/

On Fri, 3 Nov 2023, Martin Hannigan via Outages wrote:

> The RFO is irrelevant. The damage is done. Maybe there are questions? Most
> important is when you got back in business. The rest is an almost total
> waste of time. The public stuff will be image oriented. It has to be.
> Privately, you all may get more mileage. Contractually? If you know you
> know. Many can/should/will get compensation for the damage. I would.
> Whether credits or benefits at renewal time. This may be worth exploiting
> as a customer. After all, these aren't supposed to go down. Hope it never
> happens to (me)! If it does, this is the script.
>
>   - How did all generators fail together?
>   - How did all transfer switches fail to shift from utility to generator
>   together?
>   - How did all the UPS fail all together?
>   - Why did the facility come back in pieces?
>   - Are you sure you're really an N+ facility and not an N facility? Show
>   me. Sign new SLA
>   - Why did it take an hour to restore "any" power?
>   - How will this not happen again?
>   - How do you plan to make me whole?
>
> They know at the end of the day this sucks. And they are embarrassed. They
> really do care. Bunch of good people. However, who's leaving customer wise?
> That's calculus. Answer? Almost none of you. :)
>
> YMMV
>
> -M<
>
> On Fri, Nov 3, 2023 at 12:45 PM Zach Hanna <chezbut at gmail.com> wrote:
>
>> I was told it was the transfer switches, not the gensets.
>>
>> But waiting for the RFO
>>
>> Sent from my iPhone
>>
>> On Nov 2, 2023, at 5:26 PM, Martin Hannigan via Outages <
>> outages at outages.org> wrote:
>>
>> 
>>
>> That means a series of parallel gens failed “all together”. The RFO for
>> this is going to be uncomfortable.
>>
>>
>> On Thu, Nov 2, 2023 at 09:10 Matt Morgensen via Outages <
>> outages at outages.org> wrote:
>>
>>> We're in Flexential PDX02, I have guys on-site now.
>>>
>>> Looks like they lost utility, switched to generator, and then generator
>>> failed (not clear on scope of Gen failure yet). Some utility power is back,
>>> so recovery is in progress for some portion of the site.  I still have ~70
>>> racks without power....but things are coming back to life slowly.
>>>
>>> On Thu, Nov 2, 2023 at 6:01 AM Tom Strickx via Outages <
>>> outages at outages.org> wrote:
>>>
>>>> Flexential PDX02 is entirely down right now. Full power outage.
>>>> --
>>>> Tom Strickx
>>>> Principal Network Engineer
>>>> AS13335 - Cloudflare
>>>>
>>>>
>>>> On Thu, Nov 2, 2023 at 12:31 PM Bob Colon via Outages <
>>>> outages at outages.org> wrote:
>>>>
>>>>> All my sites, hosted on one of their VPSes is down hard.    I can't SSH
>>>>> in, and it's not responding to pings.   According to their status page,
>>>>> their data center on Portland is down hard:
>>>>> https://www.dreamhoststatus.com/
>>>>>
>>>>> On Thu, Nov 2, 2023 at 7:27 AM Max Leonard via Outages <
>>>>> outages at outages.org> wrote:
>>>>>
>>>>>> cloudflare is having issues again (https://www.cloudflarestatus.com/)
>>>>>> -- probably unrelated but ::shrug::
>>>>>>
>>>>>> On Thu, Nov 2, 2023 at 7:16 AM James Lawrie via Outages <
>>>>>> outages at outages.org> wrote:
>>>>>>
>>>>>>> It could be something else, but one of our sites is down because
>>>>>>> MySQL endpoint is unresponsive, and attempting to login to contact them
>>>>>>> panel.dreamhost.com has the same problem (gives either 503 or times
>>>>>>> out).
>>>>>>> _______________________________________________
>>>>>>> Outages mailing list
>>>>>>> Outages at outages.org
>>>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Outages mailing list
>>>>>> Outages at outages.org
>>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>>
>>>>> _______________________________________________
>>>>> Outages mailing list
>>>>> Outages at outages.org
>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>

---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
beckman at angryox.com                                https://www.angryox.com/
---------------------------------------------------------------------------
-------------- next part --------------
_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages