<div dir="ltr"><div dir="ltr"><div><br></div>The RFO is irrelevant. The damage is done. Maybe there are questions? Most important is when you got back in business. The rest is an almost total waste of time. The public stuff will be image oriented. It has to be. Privately, you all may get more mileage. Contractually? If you know you know. Many can/should/will get compensation for the damage. I would. Whether credits or benefits at renewal time. This may be worth exploiting as a customer. After all, these aren't supposed to go down. Hope it never happens to (me)! If it does, this is the script.<br><ul><li>How did all generators fail together?</li><li>How did all transfer switches fail to shift from utility to generator together?</li><li>How did all the UPS fail all together?</li><li>Why did the facility come back in pieces? <br></li><li>Are you sure you're really an N+ facility and not an N facility? Show me. Sign new SLA<br></li><li>Why did it take an hour to restore "any" power?</li><li>How will this not happen again?</li><li>How do you plan to make me whole?</li></ul>They know at the end of the day this sucks. And they are embarrassed. They really do care. Bunch of good people. However, who's leaving customer wise? That's calculus. Answer? Almost none of you. :)<br></div><div dir="ltr"><br></div><div>YMMV<br></div><div dir="ltr"><div><br></div><div>-M<<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 3, 2023 at 12:45 PM Zach Hanna <<a href="mailto:chezbut@gmail.com">chezbut@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">I was told it was the transfer switches, not the gensets.<div><br></div><div>But waiting for the RFO </div><div><br id="m_1030670476289203203lineBreakAtBeginningOfSignature"><div dir="ltr">Sent from my iPhone</div><div dir="ltr"><br><blockquote type="cite">On Nov 2, 2023, at 5:26 PM, Martin Hannigan via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><div><br></div><div dir="auto">That means a series of parallel gens failed “all together”. The RFO for this is going to be uncomfortable.</div><div dir="auto"> </div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 2, 2023 at 09:10 Matt Morgensen via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">We're in Flexential PDX02, I have guys on-site now. <div><br></div><div>Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in progress for some portion of the site. I still have ~70 racks without power....but things are coming back to life slowly. </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 2, 2023 at 6:01 AM Tom Strickx via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Flexential PDX02 is entirely down right now. Full power outage.<br clear="all"><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><font style="color:rgb(136,136,136)">-- </font></div><div><font style="color:rgb(136,136,136)">Tom Strickx</font></div><div><font style="color:rgb(136,136,136)">Principal Network Engineer</font></div><div><font style="color:rgb(136,136,136)">AS13335 - Cloudflare</font></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 2, 2023 at 12:31 PM Bob Colon via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">All my sites, hosted on one of their VPSes is down hard. I can't SSH in, and it's not responding to pings. According to their status page, their data center on Portland is down hard:<br><a href="https://www.dreamhoststatus.com/" target="_blank">https://www.dreamhoststatus.com/</a><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 2, 2023 at 7:27 AM Max Leonard via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">cloudflare is having issues again (<a href="https://www.cloudflarestatus.com/" target="_blank">https://www.cloudflarestatus.com/</a>) -- probably unrelated but ::shrug::</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 2, 2023 at 7:16 AM James Lawrie via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">It could be something else, but one of our sites is down because MySQL endpoint is unresponsive, and attempting to login to contact them <a href="http://panel.dreamhost.com" rel="noreferrer" target="_blank">panel.dreamhost.com</a> has the same problem (gives either 503 or times out).<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote></div>
</div>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote></div>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote></div>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote></div>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote></div></div>
<span>_______________________________________________</span><br><span>Outages mailing list</span><br><span><a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a></span><br><span><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a></span><br></div></blockquote></div></div></blockquote></div></div>