<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Lucida Console";
panose-1:2 11 6 9 4 5 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">I can only imagine the series of failures that led to that outcome. That’s wild. Resume Generating Event for someone I’d assume.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Lucida Console";color:black">--<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Lucida Console";color:black">Christopher Conley<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Lucida Console";color:black">Systems Administrator |</span><span style="font-family:"Lucida Console""> <span style="color:#FF3D00">Fors Marsh</span><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Lucida Console";color:black">4250 N Fairfax Dr, Suite 520<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Lucida Console";color:black">Arlington, VA 22203</span><span style="font-family:"Lucida Console""><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Outages <outages-bounces@outages.org> <b>On Behalf Of
</b>Matt Morgensen via Outages<br>
<b>Sent:</b> Thursday, November 2, 2023 9:09 AM<br>
<b>To:</b> Outages <outages@outages.org><br>
<b>Subject:</b> [EXTERNAL] Re: [outages] Dreamhost MySQL outage?<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">We're in Flexential PDX02, I have guys on-site now. <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in progress for some portion of the site. I still have ~70 racks without
power....but things are coming back to life slowly. <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Nov 2, 2023 at 6:01 AM Tom Strickx via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Flexential PDX02 is entirely down right now. Full power outage.<br clear="all">
<o:p></o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:#888888">-- </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">Tom Strickx</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">Principal Network Engineer</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">AS13335 - Cloudflare</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Nov 2, 2023 at 12:31 PM Bob Colon via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">All my sites, hosted on one of their VPSes is down hard. I can't SSH in, and it's not responding to pings. According to their status page, their data center on Portland is down hard:<br>
<a href="https://www.dreamhoststatus.com/" target="_blank">https://www.dreamhoststatus.com/</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Nov 2, 2023 at 7:27 AM Max Leonard via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">cloudflare is having issues again (<a href="https://www.cloudflarestatus.com/" target="_blank">https://www.cloudflarestatus.com/</a>) -- probably unrelated but ::shrug::<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Nov 2, 2023 at 7:16 AM James Lawrie via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">It could be something else, but one of our sites is down because MySQL endpoint is unresponsive, and attempting to login to contact them
<a href="http://panel.dreamhost.com" target="_blank">panel.dreamhost.com</a> has the same problem (gives either 503 or times out).<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p>
</blockquote>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal">_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal">_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p>
</blockquote>
</div>
</div>
<p><span style="font-size:7.0pt;line-height:107%;font-family:
"Arial",sans-serif">The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party,
without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.</span></p>
</body>
</html>