<div dir="ltr">Someone didn't have proper protections in place it looks like.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 10, 2020 at 9:20 AM Mike Bolitho via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div>So another BGP highjacking...<br><br><div>- Mike Bolitho</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 10, 2020, 5:40 AM Greg Dickinson via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Found this on Reddit as an initial RFO, so keep in mind the source (although r/sysadmin is usually pretty good):<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif">“</span><span style="font-size:10.5pt;font-family:Arial,sans-serif">A 3rd party network provider was advertising routes which resulted in our WorldWide traffic becoming severely
impeded. This led to IBM Cloud clients being unable to log-in to their accounts, greatly limited internet/DC connectivity and other significant network route related impacts. Network Specialists have made adjustments to route policies to restore network access,
and alleviate the impacts. The overall incident lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed Customer Incident Report/Root Cause Analysis as soon as possible”</span><span style="font-size:11pt;font-family:Calibri,sans-serif"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(157,21,53)">Greg Dickinson<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Network Engineer<u></u><u></u></span></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11pt;line-height:110%;font-family:Calibri,sans-serif;color:rgb(157,21,53)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11pt;line-height:110%;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><img width="155" height="32" style="width: 1.6145in; height: 0.3333in;" id="gmail-m_988000254023592495m_-7055204350911225905Picture_x0020_15" alt="cid:image001.png@01D2DDE3.06E76B70"></span><span style="font-size:11pt;line-height:110%;font-family:Calibri,sans-serif;color:rgb(157,21,53)"><u></u><u></u></span></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11pt;line-height:110%;font-family:Calibri,sans-serif;color:rgb(64,65,64)">234 Goodwin Crest Drive, Suite 500<u></u><u></u></span></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11pt;line-height:110%;font-family:Calibri,sans-serif;color:rgb(64,65,64)">Birmingham, AL 35209<u></u><u></u></span></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11pt;line-height:110%;font-family:Calibri,sans-serif;color:rgb(64,65,64)">Office: 205.917.2407 | Mobile: 205.234.6427<u></u><u></u></span></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11pt;line-height:110%;font-family:Calibri,sans-serif;color:rgb(64,65,64)">Fax: 205.945.0515 |
<u></u><u></u></span></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11pt;line-height:110%;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><a href="mailto:greg.dickinson@bryantbank.com" rel="noreferrer" target="_blank">greg.dickinson@bryantbank.com</a></span><span style="font-size:11pt;line-height:110%;font-family:Calibri,sans-serif;color:rgb(64,65,64)">
| </span><span style="font-size:11pt;line-height:110%;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><a href="http://bryantbank.com/" rel="noreferrer" target="_blank"><span style="color:rgb(64,65,64)">BryantBank.com</span></a></span><span style="font-size:11pt;line-height:110%;font-family:Calibri,sans-serif;color:rgb(64,65,64)"><u></u><u></u></span></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11pt;line-height:110%;font-family:Calibri,sans-serif;color:rgb(64,65,64)">Unbeatable Service. Legendary Results.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><a href="https://facebook.com/bryantbank/" rel="noreferrer" target="_blank"><span style="color:black">Facebook</span></a> |
<a href="https://twitter.com/bryantbank/" rel="noreferrer" target="_blank"><span style="color:black">Twitter</span></a></span><span style="font-size:11pt;color:black"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif"> Outages <<a href="mailto:outages-bounces@outages.org" rel="noreferrer" target="_blank">outages-bounces@outages.org</a>>
<b>On Behalf Of </b>Jason Kuehl via Outages<br>
<b>Sent:</b> Tuesday, June 9, 2020 8:38 PM<br>
<b>To:</b> James Brown <<a href="mailto:jbrown@easypost.com" rel="noreferrer" target="_blank">jbrown@easypost.com</a>><br>
<b>Cc:</b> <a href="mailto:outages@outages.org" rel="noreferrer" target="_blank">outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] IBM Cloud / NetworkLayer Outage<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<table border="0" cellspacing="3" cellpadding="0" align="left" width="100%" style="width:100%">
<tbody>
<tr>
<td style="background:rgb(255,102,0);padding:5.25pt 1.5pt"></td>
<td width="100%" style="width:100%;background:rgb(255,255,213);padding:5.25pt 3.75pt 5.25pt 11.25pt">
<p>
<span style="font-size:10pt">This M</span><span style="font-size:9pt;font-family:"Segoe UI",sans-serif;color:rgb(33,33,33)">essage originates from<strong><span style="font-family:"Segoe UI",sans-serif"> outside</span></strong> Bryant Bank. Please use
<span><u>caution</u></span> when opening this correspondence, attachments or hyperlinks (URLs). If you have questions, please contact IT Support. Thank you.</span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal">I can't wait for this post mortem.<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages <<a href="mailto:outages@outages.org" rel="noreferrer" target="_blank">outages@outages.org</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look normal to me.<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, Jun 9, 2020 at 5:27 PM James Brown <<a href="mailto:jbrown@easypost.com" rel="noreferrer" target="_blank">jbrown@easypost.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">Just wanted to share updates:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I got through to someone at IBM who confirmed the level of chaos over there. It looks like they've started deploying some mitigations because I'm seeing packet loss drop off but latency is now awful for all US locations. It looks like IBM
has failed over all of their peering to go through their European facilities because every return packet from every POP is coming from
<a href="http://secure-web.cisco.com/1ZCENVaw45JMEdCoXCwG0rvb1UUCeCpWh4Gtfftz2u_gZYgdQhoKwrBfewM4QrECWqHUjB3ntgfnOudlN7Txq8mYVsGWJWeXWg-v2cvwS_INwBf66XwGN6PMKDQHpa9-nztcpzjJKAL45c9ehKsPd3RmZsKBti3c78NtK4_5orSmMjjftj2O5ihZtM3DI39J1jzcHEJkQ4qn1RaZRZvtck6dmc6mHvWEK07J6n5lfEiaFtNTzbRMqYg5PRkS5mXxpZdjs13BLwOYthPbwpG2y9sxQhA4vTT6kJ0WTImFe8A_gFn1O_YWB9bnGM_p6egB-/http%3A%2F%2Flon03.networklayer.com" rel="noreferrer" target="_blank">
lon03.networklayer.com</a> or <a href="http://secure-web.cisco.com/11XtT4bESggqtqUrb-8NP8jSqmiZcXpSKONGPf8fLiEIvSGY8GXfJOQvPgNH2fin3Mf1oX1qtIrUEhzncE_p6bafeLZfwQNedalFA2ZzWGHl31bfPAPv2AZR6vyTjFMCcWz92HYy6YIl1zhiQebTrb_76Ea-AEzmUrontPsu-oMbxR1e7nrBlRL1dTA7t3blZO8P3p58WDvMLWgLkm-fwMGANaDJ16Pw8BEmKXJJ-fHjl_4kXPUHTptOJJ2tBu8JYCQDVILZauSUz7kHx_JyJDNQEv4x9MojMPj2lnajAHeRvPjmy7vFPJnybP1MU8vGw/http%3A%2F%2Fams02.networklayer.com" rel="noreferrer" target="_blank">ams02.networklayer.com</a>
(via Telia).<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, Jun 9, 2020 at 4:12 PM James Brown <<a href="mailto:jbrown@easypost.com" rel="noreferrer" target="_blank">jbrown@easypost.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">James Brown<u></u><u></u></p>
<div>
<p class="MsoNormal"><span>Network </span>Engineer<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">James Brown<u></u><u></u></p>
<div>
<p class="MsoNormal">Engineer<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">James Brown<u></u><u></u></p>
<div>
<p class="MsoNormal">Engineer<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" rel="noreferrer" target="_blank">Outages@outages.org</a><br>
<a href="https://secure-web.cisco.com/1ALbUsNtj5hDIBcrx-EkowvDfnZpZsLIXlTU9xpfnU2i2nr2PgCQpVywQsS39qJHljpuu-05y4c4OqVp6vnUXfkwMjdFzdWAx6ufZY6amRzi-W8ZDQuOeeScc9jnHNYi4_XXaHMDw62k5a0GGAPi2e7kdcA7oL1IKNx_qZPZumgIs16f_vwL28g93YV-Kco95rDiKdE0bmSRoXy19iByqDxxST8yHpluFq-50ihMwmJnR5cX7NXCjKltHkLQJmNokhORvV20qvLxA6jaejf_FP1PGqUQYp8tZbyW7-wXmtTDlX2L3Pi07ilf9uVHRvOBhR9NhsuDyhRSpbieu3tNNcQ/https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal">Sincerely,<br>
<br>
Jason W Kuehl<br>
Cell 920-419-8983<br>
<a href="mailto:jason.w.kuehl@gmail.com" rel="noreferrer" target="_blank">jason.w.kuehl@gmail.com</a><u></u><u></u></p>
</div>
</div>
<br><br>NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, print, save, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete copies. Thank you.</div>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" rel="noreferrer" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote></div></div></div>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">Sincerely,<br> <br>Jason W Kuehl<br>Cell 920-419-8983<br><a href="mailto:jason.w.kuehl@gmail.com" target="_blank">jason.w.kuehl@gmail.com</a></div>