[outages] IBM Cloud / NetworkLayer Outage

Wed Jun 10 09:15:27 EDT 2020

So another BGP highjacking...

- Mike Bolitho

On Wed, Jun 10, 2020, 5:40 AM Greg Dickinson via Outages <
outages at outages.org> wrote:

> Found this on Reddit as an initial RFO, so keep in mind the source
> (although r/sysadmin is usually pretty good):
>
>
>
> “A 3rd party network provider was advertising routes which resulted in
> our WorldWide traffic becoming severely impeded. This led to IBM Cloud
> clients being unable to log-in to their accounts, greatly limited
> internet/DC connectivity and other significant network route related
> impacts. Network Specialists have made adjustments to route policies to
> restore network access, and alleviate the impacts. The overall incident
> lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed
> Customer Incident Report/Root Cause Analysis as soon as possible”
>
>
>
> Greg Dickinson
>
> Network Engineer
>
>
>
> [image: cid:image001.png at 01D2DDE3.06E76B70]
>
> 234 Goodwin Crest Drive, Suite 500
>
> Birmingham, AL  35209
>
> Office: 205.917.2407  |  Mobile: 205.234.6427
>
> Fax: 205.945.0515  |
>
> greg.dickinson at bryantbank.com  |  BryantBank.com <http://bryantbank.com/>
>
> Unbeatable Service.  Legendary Results.
>
> Facebook <https://facebook.com/bryantbank/>  |  Twitter
> <https://twitter.com/bryantbank/>
>
>
>
> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Jason Kuehl
> via Outages
> *Sent:* Tuesday, June 9, 2020 8:38 PM
> *To:* James Brown <jbrown at easypost.com>
> *Cc:* outages at outages.org
> *Subject:* Re: [outages] IBM Cloud / NetworkLayer Outage
>
>
>
>
>
> This Message originates from* outside* Bryant Bank.   Please use *caution*
> when opening this correspondence, attachments or hyperlinks (URLs).  If you
> have questions, please contact IT Support.  Thank you.
>
> I can't wait for this post mortem.
>
>
>
> On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages <
> outages at outages.org> wrote:
>
> As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look
> normal to me.
>
>
>
> On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbrown at easypost.com> wrote:
>
> Just wanted to share updates:
>
>
>
> I got through to someone at IBM who confirmed the level of chaos over
> there. It looks like they've started deploying some mitigations because I'm
> seeing packet loss drop off but latency is now awful for all US locations.
> It looks like IBM has failed over all of their peering to go through their
> European facilities because every return packet from every POP is coming
> from lon03.networklayer.com
> <http://secure-web.cisco.com/1ZCENVaw45JMEdCoXCwG0rvb1UUCeCpWh4Gtfftz2u_gZYgdQhoKwrBfewM4QrECWqHUjB3ntgfnOudlN7Txq8mYVsGWJWeXWg-v2cvwS_INwBf66XwGN6PMKDQHpa9-nztcpzjJKAL45c9ehKsPd3RmZsKBti3c78NtK4_5orSmMjjftj2O5ihZtM3DI39J1jzcHEJkQ4qn1RaZRZvtck6dmc6mHvWEK07J6n5lfEiaFtNTzbRMqYg5PRkS5mXxpZdjs13BLwOYthPbwpG2y9sxQhA4vTT6kJ0WTImFe8A_gFn1O_YWB9bnGM_p6egB-/http%3A%2F%2Flon03.networklayer.com>
> or ams02.networklayer.com
> <http://secure-web.cisco.com/11XtT4bESggqtqUrb-8NP8jSqmiZcXpSKONGPf8fLiEIvSGY8GXfJOQvPgNH2fin3Mf1oX1qtIrUEhzncE_p6bafeLZfwQNedalFA2ZzWGHl31bfPAPv2AZR6vyTjFMCcWz92HYy6YIl1zhiQebTrb_76Ea-AEzmUrontPsu-oMbxR1e7nrBlRL1dTA7t3blZO8P3p58WDvMLWgLkm-fwMGANaDJ16Pw8BEmKXJJ-fHjl_4kXPUHTptOJJ2tBu8JYCQDVILZauSUz7kHx_JyJDNQEv4x9MojMPj2lnajAHeRvPjmy7vFPJnybP1MU8vGw/http%3A%2F%2Fams02.networklayer.com>
> (via Telia).
>
>
>
> On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown at easypost.com> wrote:
>
> It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global
> outage since about 15:04 Pacific time affecting routing to AWS, Level 3 /
> CenturyLink, and a bunch of other providers. Has anyone been able to get
> more details?
>
>
>
> --
>
> James Brown
>
> Network Engineer
>
>
>
>
> --
>
> James Brown
>
> Engineer
>
>
>
>
> --
>
> James Brown
>
> Engineer
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
> <https://secure-web.cisco.com/1ALbUsNtj5hDIBcrx-EkowvDfnZpZsLIXlTU9xpfnU2i2nr2PgCQpVywQsS39qJHljpuu-05y4c4OqVp6vnUXfkwMjdFzdWAx6ufZY6amRzi-W8ZDQuOeeScc9jnHNYi4_XXaHMDw62k5a0GGAPi2e7kdcA7oL1IKNx_qZPZumgIs16f_vwL28g93YV-Kco95rDiKdE0bmSRoXy19iByqDxxST8yHpluFq-50ihMwmJnR5cX7NXCjKltHkLQJmNokhORvV20qvLxA6jaejf_FP1PGqUQYp8tZbyW7-wXmtTDlX2L3Pi07ilf9uVHRvOBhR9NhsuDyhRSpbieu3tNNcQ/https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages>
>
>
>
>
> --
>
> Sincerely,
>
> Jason W Kuehl
> Cell 920-419-8983
> jason.w.kuehl at gmail.com
>
>
> NOTICE: This electronic mail message and any files transmitted with it are
> intended exclusively for the individual or entity to which it is addressed.
> The message, together with any attachment, may contain confidential and/or
> privileged information. Any unauthorized review, use, print, save, copy,
> disclosure or distribution is strictly prohibited. If you have received
> this message in error, please immediately advise the sender by reply email
> and delete copies.  Thank you.
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20200610/d56c3104/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 5977 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/outages/attachments/20200610/d56c3104/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 5977 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/outages/attachments/20200610/d56c3104/attachment-0001.png>