[outages] IBM Cloud / NetworkLayer Outage

[Jason Kuehl]

Didnt know what was a list.

On Wed, Jun 10, 2020 at 9:45 AM Andy Ringsmuth via Outages <
outages at outages.org> wrote:

> Could we move further discussion to outages-discuss please?
>
> ----
> Andy Ringsmuth
> 5609 Harding Drive
> Lincoln, NE 68521-5831
> (402) 304-0083
> andy at andyring.com
>
> “Better even die free, than to live slaves.” - Frederick Douglas, 1863
>
> > On Jun 10, 2020, at 8:22 AM, Jason Kuehl via Outages <
> outages at outages.org> wrote:
> >
> > Someone didn't have proper protections in place it looks like.
> >
> > On Wed, Jun 10, 2020 at 9:20 AM Mike Bolitho via Outages <
> outages at outages.org> wrote:
> > So another BGP highjacking...
> >
> > - Mike Bolitho
> >
> > On Wed, Jun 10, 2020, 5:40 AM Greg Dickinson via Outages <
> outages at outages.org> wrote:
> > Found this on Reddit as an initial RFO, so keep in mind the source
> (although r/sysadmin is usually pretty good):
> >
> >
> >
> > “A 3rd party network provider was advertising routes which resulted in
> our WorldWide traffic becoming severely impeded. This led to IBM Cloud
> clients being unable to log-in to their accounts, greatly limited
> internet/DC connectivity and other significant network route related
> impacts. Network Specialists have made adjustments to route policies to
> restore network access, and alleviate the impacts. The overall incident
> lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed
> Customer Incident Report/Root Cause Analysis as soon as possible”
> >
> >
> >
> > Greg Dickinson
> >
> > Network Engineer
> >
> >
> >
> >
> >
> > 234 Goodwin Crest Drive, Suite 500
> >
> > Birmingham, AL  35209
> >
> > Office: 205.917.2407  |  Mobile: 205.234.6427
> >
> > Fax: 205.945.0515  |
> >
> > greg.dickinson at bryantbank.com  |  BryantBank.com
> >
> > Unbeatable Service.  Legendary Results.
> >
> > Facebook  |  Twitter
> >
> >
> >
> > From: Outages <outages-bounces at outages.org> On Behalf Of Jason Kuehl
> via Outages
> > Sent: Tuesday, June 9, 2020 8:38 PM
> > To: James Brown <jbrown at easypost.com>
> > Cc: outages at outages.org
> > Subject: Re: [outages] IBM Cloud / NetworkLayer Outage
> >
> >
> >
> >
> >
> > This Message originates from outside Bryant Bank.   Please use caution
> when opening this correspondence, attachments or hyperlinks (URLs).  If you
> have questions, please contact IT Support.  Thank you.
> >
> > I can't wait for this post mortem.
> >
> >
> >
> > On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages <
> outages at outages.org> wrote:
> >
> > As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look
> normal to me.
> >
> >
> >
> > On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbrown at easypost.com> wrote:
> >
> > Just wanted to share updates:
> >
> >
> >
> > I got through to someone at IBM who confirmed the level of chaos over
> there. It looks like they've started deploying some mitigations because I'm
> seeing packet loss drop off but latency is now awful for all US locations.
> It looks like IBM has failed over all of their peering to go through their
> European facilities because every return packet from every POP is coming
> from lon03.networklayer.com or ams02.networklayer.com (via Telia).
> >
> >
> >
> > On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown at easypost.com> wrote:
> >
> > It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global
> outage since about 15:04 Pacific time affecting routing to AWS, Level 3 /
> CenturyLink, and a bunch of other providers. Has anyone been able to get
> more details?
> >
> >
> >
> > --
> >
> > James Brown
> >
> > Network Engineer
> >
> >
> >
> >
> >
> > --
> >
> > James Brown
> >
> > Engineer
> >
> >
> >
> >
> >
> > --
> >
> > James Brown
> >
> > Engineer
> >
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> >
> >
> >
> >
> >
> > --
> >
> > Sincerely,
> >
> > Jason W Kuehl
> > Cell 920-419-8983
> > jason.w.kuehl at gmail.com
> >
> >
> >
> > NOTICE: This electronic mail message and any files transmitted with it
> are intended exclusively for the individual or entity to which it is
> addressed. The message, together with any attachment, may contain
> confidential and/or privileged information. Any unauthorized review, use,
> print, save, copy, disclosure or distribution is strictly prohibited. If
> you have received this message in error, please immediately advise the
> sender by reply email and delete copies.  Thank you.
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> >
> >
> > --
> > Sincerely,
> >
> > Jason W Kuehl
> > Cell 920-419-8983
> > jason.w.kuehl at gmail.com
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>


-- 
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.kuehl at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20200610/657fa3f0/attachment.htm>