<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.style71
{mso-style-name:style71;}
span.gmaildefault
{mso-style-name:gmail_default;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">FWIW: https [:] //www [.] computing.co.uk/news/4016300/ibm-cloud-suffers-global-outage-bringing-customer-websites<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Includes Twitter shots and links.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">IBM Cloud suffered a widespread outage on Tuesday that brought down multiple services hosted on the platform, as well as IBM's own cloud status page.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The problem likely started around 2:30pm Pacific Time (PT) and escalated at around 5:30pm PT. Quickly, it started to impact services such as Cloud Object Storage, App Connect,
Kubernetes Service, Continuous Delivery, Identity and Access Management (IAM), VPN for VPC, and Watson AI cloud services.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">IBM's own status page also failed to load during the outage and returned an internal server error message.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">DownDetector.com, a website that monitors website outages, showed that the outage was impacting IBM Cloud services in different parts of the US, South America, Japan and Australia.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">IBM's cloud-based streaming and file transfer services, which come under the Aspera brand, were also mostly out.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">IBM Aspera, an IBM unit that operates under hybrid cloud business, said that following the first outage, it was alerted to a service disruption affecting all regions.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Aspera further revealed that AoC Managed Storage was suffering "major outage" in Dallas, Amsterdam, Frankfurt, Melbourne, and Toronto.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">"Our engineers are currently investigating the incident and we will provide updates when more information is available," an Aspera advisory stated.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">IBM Cloud Transfer Clusters in Frankfurt, Washington DC, Hong Kong, Chennai, London, Montreal, Mexico, Milan, Oslo, Seoul, San Jose, Sao Paulo, Tokyo, Paris, and Singapore
were also listed as suffering major outages.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">It also emerged that IBM Cloud Monitoring service Sysdig had a three-hour maintenance window earlier in the day at its London facilities. IBM had informed customers that Sysdig
maintenance work would be completed between 3pm to 6pm, and that it would cause only a 30-minute disruption to Cloud Monitoring service.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">IBM Cloud eventually posted an update on Twitter, stating that it was aware of the issues and that the issue was being investigated.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">At present, the issue appears to have been fully resolved. IBM also said in a tweet at around 7:00pm PT that all IBM Cloud services are now working normally.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">- andy</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Outages <outages-bounces@outages.org> on behalf of Greg Dickinson via Outages <outages@outages.org><br>
<b>Reply-To: </b>Greg Dickinson <Greg.Dickinson@bryantbank.com><br>
<b>Date: </b>Wednesday, June 10, 2020 at 8:44 AM<br>
<b>To: </b>"'outages@outages.org'" <outages@outages.org><br>
<b>Subject: </b>Re: [outages] IBM Cloud / NetworkLayer Outage<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Found this on Reddit as an initial RFO, so keep in mind the source (although r/sysadmin is usually pretty good):</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">“</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif">A 3rd party network provider was advertising routes which resulted in our WorldWide traffic becoming severely
impeded. This led to IBM Cloud clients being unable to log-in to their accounts, greatly limited internet/DC connectivity and other significant network route related impacts. Network Specialists have made adjustments to route policies to restore network access,
and alleviate the impacts. The overall incident lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed Customer Incident Report/Root Cause Analysis as soon as possible”</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#9D1535">Greg Dickinson</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Network Engineer</span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11.0pt;line-height:110%;font-family:"Calibri",sans-serif;color:#9D1535"> </span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11.0pt;line-height:110%;font-family:"Calibri",sans-serif;color:#1F497D"><img width="155" height="32" style="width:1.6145in;height:.3333in" id="Picture_x0020_15" src="cid:image001.png@01D63F03.CAA2D890"></span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11.0pt;line-height:110%;font-family:"Calibri",sans-serif;color:#404140">234 Goodwin Crest Drive, Suite 500</span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11.0pt;line-height:110%;font-family:"Calibri",sans-serif;color:#404140">Birmingham, AL 35209</span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11.0pt;line-height:110%;font-family:"Calibri",sans-serif;color:#404140">Office: 205.917.2407 | Mobile: 205.234.6427</span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11.0pt;line-height:110%;font-family:"Calibri",sans-serif;color:#404140">Fax: 205.945.0515 |
</span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11.0pt;line-height:110%;font-family:"Calibri",sans-serif;color:#1F497D"><a href="mailto:greg.dickinson@bryantbank.com">greg.dickinson@bryantbank.com</a></span><span style="font-size:11.0pt;line-height:110%;font-family:"Calibri",sans-serif;color:#404140">
| </span><span style="font-size:11.0pt;line-height:110%;font-family:"Calibri",sans-serif;color:#1F497D"><a href="http://bryantbank.com/"><span style="color:#404140">BryantBank.com</span></a></span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:110%"><span style="font-size:11.0pt;line-height:110%;font-family:"Calibri",sans-serif;color:#404140">Unbeatable Service. Legendary Results.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><a href="https://facebook.com/bryantbank/"><span style="color:black">Facebook</span></a> |
<a href="https://twitter.com/bryantbank/"><span style="color:black">Twitter</span></a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Outages <outages-bounces@outages.org>
<b>On Behalf Of </b>Jason Kuehl via Outages<br>
<b>Sent:</b> Tuesday, June 9, 2020 8:38 PM<br>
<b>To:</b> James Brown <jbrown@easypost.com><br>
<b>Cc:</b> outages@outages.org<br>
<b>Subject:</b> Re: [outages] IBM Cloud / NetworkLayer Outage</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="3" cellpadding="0" align="left" width="100%" style="width:100.0%">
<tbody>
<tr>
<td style="background:#FF6600;padding:5.25pt 1.5pt 5.25pt 1.5pt"></td>
<td width="100%" style="width:100.0%;background:#FFFFD5;padding:5.25pt 3.75pt 5.25pt 11.25pt;WORD-WRAP: break-word">
<p style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<span style="font-size:10.0pt;color:black">This M</span><span style="font-size:9.0pt;font-family:"Segoe UI";color:#212121">essage originates from<strong><span style="font-family:"Segoe UI""> outside</span></strong> Bryant Bank. Please use
<span class="style71"><u>caution</u></span> when opening this correspondence, attachments or hyperlinks (URLs). If you have questions, please contact IT Support. Thank you.</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal"><span style="color:white">I can't wait for this post mortem</span>.<o:p></o:p></p>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look normal to me.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Jun 9, 2020 at 5:27 PM James Brown <<a href="mailto:jbrown@easypost.com" target="_blank">jbrown@easypost.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">Just wanted to share updates:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I got through to someone at IBM who confirmed the level of chaos over there. It looks like they've started deploying some mitigations because I'm seeing packet loss drop off but latency is now awful for all US locations. It looks like IBM
has failed over all of their peering to go through their European facilities because every return packet from every POP is coming from
<a href="http://secure-web.cisco.com/1ZCENVaw45JMEdCoXCwG0rvb1UUCeCpWh4Gtfftz2u_gZYgdQhoKwrBfewM4QrECWqHUjB3ntgfnOudlN7Txq8mYVsGWJWeXWg-v2cvwS_INwBf66XwGN6PMKDQHpa9-nztcpzjJKAL45c9ehKsPd3RmZsKBti3c78NtK4_5orSmMjjftj2O5ihZtM3DI39J1jzcHEJkQ4qn1RaZRZvtck6dmc6mHvWEK07J6n5lfEiaFtNTzbRMqYg5PRkS5mXxpZdjs13BLwOYthPbwpG2y9sxQhA4vTT6kJ0WTImFe8A_gFn1O_YWB9bnGM_p6egB-/http%3A%2F%2Flon03.networklayer.com" target="_blank">
lon03.networklayer.com</a> or <a href="http://secure-web.cisco.com/11XtT4bESggqtqUrb-8NP8jSqmiZcXpSKONGPf8fLiEIvSGY8GXfJOQvPgNH2fin3Mf1oX1qtIrUEhzncE_p6bafeLZfwQNedalFA2ZzWGHl31bfPAPv2AZR6vyTjFMCcWz92HYy6YIl1zhiQebTrb_76Ea-AEzmUrontPsu-oMbxR1e7nrBlRL1dTA7t3blZO8P3p58WDvMLWgLkm-fwMGANaDJ16Pw8BEmKXJJ-fHjl_4kXPUHTptOJJ2tBu8JYCQDVILZauSUz7kHx_JyJDNQEv4x9MojMPj2lnajAHeRvPjmy7vFPJnybP1MU8vGw/http%3A%2F%2Fams02.networklayer.com" target="_blank">ams02.networklayer.com</a>
(via Telia).<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Jun 9, 2020 at 4:12 PM James Brown <<a href="mailto:jbrown@easypost.com" target="_blank">jbrown@easypost.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">James Brown<o:p></o:p></p>
<div>
<p class="MsoNormal"><span class="gmaildefault">Network </span>Engineer<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">James Brown<o:p></o:p></p>
<div>
<p class="MsoNormal">Engineer<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">James Brown<o:p></o:p></p>
<div>
<p class="MsoNormal">Engineer<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://secure-web.cisco.com/1ALbUsNtj5hDIBcrx-EkowvDfnZpZsLIXlTU9xpfnU2i2nr2PgCQpVywQsS39qJHljpuu-05y4c4OqVp6vnUXfkwMjdFzdWAx6ufZY6amRzi-W8ZDQuOeeScc9jnHNYi4_XXaHMDw62k5a0GGAPi2e7kdcA7oL1IKNx_qZPZumgIs16f_vwL28g93YV-Kco95rDiKdE0bmSRoXy19iByqDxxST8yHpluFq-50ihMwmJnR5cX7NXCjKltHkLQJmNokhORvV20qvLxA6jaejf_FP1PGqUQYp8tZbyW7-wXmtTDlX2L3Pi07ilf9uVHRvOBhR9NhsuDyhRSpbieu3tNNcQ/https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal">Sincerely,<br>
<br>
Jason W Kuehl<br>
Cell 920-419-8983<br>
<a href="mailto:jason.w.kuehl@gmail.com" target="_blank">jason.w.kuehl@gmail.com</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<br>
NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized
review, use, print, save, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete copies. Thank you.
<o:p></o:p></span></p>
</div>
</body>
</html>