[outages] Outages Digest, Vol 147, Issue 10

Sun Aug 30 11:20:50 EDT 2020

This was more a world wide issue for them. I had issues with getting
connectivity back to the US from EU markets as well for services that we
have with them domestically in the US and internationally.

It was first noticed at 6:04 AM EST on our end with a hold timer expire.
Routes stuck/held in LA, Miami and local markets of those.  Numerous
providers were still routing towards level3 and yes they were holding onto
the routes and they were not being updated in remote markets (e.g. Miami)
However, I would see them be updated in Tampa, but not processed throughout
their network.

Dejan Dan Protich

HIVELOCITY | Sr. Network Engineer

From: Outages <outages-bounces at outages.org> On Behalf Of Tino Montemor via
Outages
Sent: Sunday, August 30, 2020 11:06 AM
To: outages at outages.org
Subject: Re: [outages] Outages Digest, Vol 147, Issue 10

Just want to reflect (pun?) what others are saying, Centurylink wasn't
letting go of our routes. 

As of about 2 minutes ago it seems out AS Prepend has finally began to
propagate 

_____

Tino Montemor | Skechers USA, Inc. 

O: 310-406-0132 M: 909-721-6673

  _____  

From: Outages <outages-bounces at outages.org
<mailto:outages-bounces at outages.org> > on behalf of
outages-request at outages.org <mailto:outages-request at outages.org>
<outages-request at outages.org <mailto:outages-request at outages.org> >
Sent: Sunday, August 30, 2020 6:32:35 AM
To: outages at outages.org <mailto:outages at outages.org>  <outages at outages.org
<mailto:outages at outages.org> >
Subject: Outages Digest, Vol 147, Issue 10 

Send Outages mailing list submissions to
        outages at outages.org <mailto:outages at outages.org> 

To subscribe or unsubscribe via the World Wide Web, visit

https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman
_listinfo_outages
<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma
n_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=
Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiS
qjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=>
&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l
2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JP
fcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= 
or, via email, send a message with subject or body 'help' to
        outages-request at outages.org <mailto:outages-request at outages.org> 

You can reach the person managing the list at
        outages-owner at outages.org <mailto:outages-owner at outages.org> 

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Outages digest..."

Today's Topics:

   1. Re: CenturyLink peering issues? (David Hubbard)
   2. Re: 3356 does not WITHDRAW bgp routes (randal k)
   3. Re: 3356 does not WITHDRAW bgp routes (David Hubbard)
   4. Re: CenturyLink peering issues? (Chris Adams)
   5. Re: 3356 does not WITHDRAW bgp routes (randal k)
   6. Re: CenturyLink peering issues? (Stephen Flynn)

----------------------------------------------------------------------

Message: 1
Date: Sun, 30 Aug 2020 12:22:27 +0000
From: David Hubbard <dhubbard at dino.hostasaurus.com
<mailto:dhubbard at dino.hostasaurus.com> >
To: "outages at outages.org <mailto:outages at outages.org> " <outages at outages.org
<mailto:outages at outages.org> >
Subject: Re: [outages] CenturyLink peering issues?
Message-ID:
        <D5BED54B-45C3-4EC8-9DBA-EF1276D52C73 at dino.hostasaurus.com
<mailto:D5BED54B-45C3-4EC8-9DBA-EF1276D52C73 at dino.hostasaurus.com> >
Content-Type: text/plain; charset="utf-8"

Ugh; seeing same thing.  Have had sessions turned down for over two hours
and looking glasses are still showing 3356 propagating the advertisements.

?On 8/30/20, 8:11 AM, "Outages on behalf of Chris Adams via Outages"
<outages-bounces at outages.org on behalf of outages at outages.org
<mailto:outages-bounces at outages.org%20on%20behalf%20of%20outages at outages.org
> > wrote:

    Once upon a time, Stephen Flynn via Outages <outages at outages.org
<mailto:outages at outages.org> > said:
    > Odd part --- I disconnected my Level3 circuit at the ORL-FL facility
so that I could fully failover to my other carrier link.
    > Level3 is still advertising my routes, even though my link and BGP
session is down.

    I can confirm this - I shut down IPv4 BGP with Level3 in Chicago, but
    checking route-views and such, they're still advertising our routes (but
    don't know how to get to us once packets hit their network)..

    AS7007 all over again?
    -- 
    Chris Adams <cma at cmadams.net <mailto:cma at cmadams.net> >
    _______________________________________________
    Outages mailing list
    Outages at outages.org <mailto:Outages at outages.org> 

https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman
_listinfo_outages
<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma
n_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=
Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiS
qjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=>
&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l
2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JP
fcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= 

------------------------------

Message: 2
Date: Sun, 30 Aug 2020 06:33:41 -0600
From: randal k <rkohutek+outages at gmail.com
<mailto:rkohutek+outages at gmail.com> >
To: outages at outages.org <mailto:outages at outages.org> 
Subject: Re: [outages] 3356 does not WITHDRAW bgp routes
Message-ID:
        <CANeLk7RmHnXa=EieXpahpmM966bL8==7qsywR89Yrd0fuZ9tHA at mail.gmail.com
<mailto:CANeLk7RmHnXa=EieXpahpmM966bL8==7qsywR89Yrd0fuZ9tHA at mail.gmail.com>
>
Content-Type: text/plain; charset="UTF-8"

Seeing the same thing - shut a peer, still seeing those routes via
3356 across multiple route-servers & looking glasses.

So, do we disconnect 3356 and suffer the blackhole in hopes that it
will eventually withdraw those routes, or leave it on to prevent
blackholing but suffer massive packet loss to other carriers? Wow.

On Sun, Aug 30, 2020 at 6:28 AM Lukas Tribus via Outages
<outages at outages.org <mailto:outages at outages.org> > wrote:
>
> As previously mentioned by Stephen Flynn, 3356 does not WITHDRAW stale
> bgp routes, can be confirmed with AT&T's route server at (telnet
> route-server.ip.att.net).
>
> Stale routes from 1 hour + are still announced by 3356.
>
>
> This is causing blackholing.
> _______________________________________________
> Outages mailing list
> Outages at outages.org <mailto:Outages at outages.org> 
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman
_listinfo_outages
<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma
n_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=
Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiS
qjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=>
&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l
2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JP
fcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= 

------------------------------

Message: 3
Date: Sun, 30 Aug 2020 12:47:57 +0000
From: David Hubbard <dhubbard at dino.hostasaurus.com
<mailto:dhubbard at dino.hostasaurus.com> >
To: "outages at outages.org <mailto:outages at outages.org> " <outages at outages.org
<mailto:outages at outages.org> >
Subject: Re: [outages] 3356 does not WITHDRAW bgp routes
Message-ID:
        <4BA8D9E3-9A4F-45FB-AFE4-8BBB2E3C6902 at dino.hostasaurus.com
<mailto:4BA8D9E3-9A4F-45FB-AFE4-8BBB2E3C6902 at dino.hostasaurus.com> >
Content-Type: text/plain; charset="utf-8"

I tried bringing two circuits back up hoping to not have blackholing, one
never left idle, the other received <2000 routes, which I know from prior
outages to mean that the entire region (Tampa Bay) has been BGP isolated
from the rest of their network.  Good times...

?On 8/30/20, 8:43 AM, "Outages on behalf of randal k via Outages"
<outages-bounces at outages.org on behalf of outages at outages.org
<mailto:outages-bounces at outages.org%20on%20behalf%20of%20outages at outages.org
> > wrote:

    Seeing the same thing - shut a peer, still seeing those routes via
    3356 across multiple route-servers & looking glasses.

    So, do we disconnect 3356 and suffer the blackhole in hopes that it
    will eventually withdraw those routes, or leave it on to prevent
    blackholing but suffer massive packet loss to other carriers? Wow.

    On Sun, Aug 30, 2020 at 6:28 AM Lukas Tribus via Outages
    <outages at outages.org <mailto:outages at outages.org> > wrote:
    >
    > As previously mentioned by Stephen Flynn, 3356 does not WITHDRAW stale
    > bgp routes, can be confirmed with AT&T's route server at (telnet
    > route-server.ip.att.net).
    >
    > Stale routes from 1 hour + are still announced by 3356.
    >
    >
    > This is causing blackholing.
    > _______________________________________________
    > Outages mailing list
    > Outages at outages.org <mailto:Outages at outages.org> 
    >
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman
_listinfo_outages
<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma
n_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=
Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiS
qjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=>
&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l
2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JP
fcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= 
    _______________________________________________
    Outages mailing list
    Outages at outages.org <mailto:Outages at outages.org> 

https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman
_listinfo_outages
<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma
n_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=
Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiS
qjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=>
&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l
2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JP
fcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= 

------------------------------

Message: 4
Date: Sun, 30 Aug 2020 07:48:14 -0500
From: Chris Adams <cma at cmadams.net <mailto:cma at cmadams.net> >
To: outages at outages.org <mailto:outages at outages.org> 
Subject: Re: [outages] CenturyLink peering issues?
Message-ID: <20200830124814.GB29578 at cmadams.net
<mailto:20200830124814.GB29578 at cmadams.net> >
Content-Type: text/plain; charset=us-ascii

Once upon a time, randal k <rkohutek+outages at gmail.com
<mailto:rkohutek+outages at gmail.com> > said:
> It's weird you say that - I have been attempting to use Level3's well
> known communities to attempt to prepend, no-export etc and have been
> having limited luck ... and those communities even appear in
> route-views, so I know they're being sent!

I brought my session back up, not accepting any routes, and prepending
ours (just in case the change propagated)... it took a while, but
eventually I do see the prepended routes.

But for a provider that said they shut down their link to Level3, I
still see their routes... possibly (slowly) propagating changes but not
actual withdraws?
-- 
Chris Adams <cma at cmadams.net <mailto:cma at cmadams.net> >

------------------------------

Message: 5
Date: Sun, 30 Aug 2020 06:52:16 -0600
From: randal k <rkohutek+outages at gmail.com
<mailto:rkohutek+outages at gmail.com> >
To: outages at outages.org <mailto:outages at outages.org> 
Subject: Re: [outages] 3356 does not WITHDRAW bgp routes
Message-ID:
        <CANeLk7RcTGWkQp4SYJQO=j=c04dTpYSGrXTxfK9ovi_QkgyxZQ at mail.gmail.com
<mailto:CANeLk7RcTGWkQp4SYJQO=j=c04dTpYSGrXTxfK9ovi_QkgyxZQ at mail.gmail.com>
>
Content-Type: text/plain; charset="UTF-8"

Watching closely, after applying ^3356$ on our inbound routes, I can
see that they are bouncing sessions and slowly adding in prefixes -- I
have 17x 3356-originated routes in PA, and 893x in CO. And they have
reset the PA BGP session numerous times.

On Sun, Aug 30, 2020 at 6:33 AM randal k <rkohutek+outages at gmail.com
<mailto:rkohutek+outages at gmail.com> > wrote:
>
> Seeing the same thing - shut a peer, still seeing those routes via
> 3356 across multiple route-servers & looking glasses.
>
> So, do we disconnect 3356 and suffer the blackhole in hopes that it
> will eventually withdraw those routes, or leave it on to prevent
> blackholing but suffer massive packet loss to other carriers? Wow.
>
> On Sun, Aug 30, 2020 at 6:28 AM Lukas Tribus via Outages
> <outages at outages.org <mailto:outages at outages.org> > wrote:
> >
> > As previously mentioned by Stephen Flynn, 3356 does not WITHDRAW stale
> > bgp routes, can be confirmed with AT&T's route server at (telnet
> > route-server.ip.att.net).
> >
> > Stale routes from 1 hour + are still announced by 3356.
> >
> >
> > This is causing blackholing.
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org <mailto:Outages at outages.org> 
> >
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman
_listinfo_outages
<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma
n_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=
Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiS
qjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=>
&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l
2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JP
fcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= 

------------------------------

Message: 6
Date: Sun, 30 Aug 2020 13:32:31 +0000
From: Stephen Flynn <sflynn at staff.atlantic.net
<mailto:sflynn at staff.atlantic.net> >
To: "outages at outages.org <mailto:outages at outages.org> " <outages at outages.org
<mailto:outages at outages.org> >
Subject: Re: [outages] CenturyLink peering issues?
Message-ID:

<BN6PR19MB10269D437047598C160A6724EB500 at BN6PR19MB1026.namprd19.prod.outlook.
com
<mailto:BN6PR19MB10269D437047598C160A6724EB500 at BN6PR19MB1026.namprd19.prod.o
utlook.com> >

Content-Type: text/plain; charset="us-ascii"

Just attempted a prepend (x4) on my Level3 advertisements --- only noticed a
change within the NTT network tables.
Other large carriers received no advertisement changes (Telia, Hurricane
Electric, AT&T)

I then noticed that my BGP session was constantly flapping.  I've now
shutdown my circuit again.  Level3 is still announcing my address space.
Thank you Level3!

Regards,

Stephen Flynn
Atlantic.Net
Direct: (321) 206-1390
sflynn at staff.atlantic.net <mailto:sflynn at staff.atlantic.net> 
www.atlantic.net <http://www.atlantic.net> 

-----Original Message-----
From: Outages <outages-bounces at outages.org
<mailto:outages-bounces at outages.org> > On Behalf Of Chris Adams via Outages
Sent: Sunday, August 30, 2020 8:48 AM
To: outages at outages.org <mailto:outages at outages.org> 
Subject: Re: [outages] CenturyLink peering issues?

CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you recognize the sender and know the
content is safe.

Once upon a time, randal k <rkohutek+outages at gmail.com
<mailto:rkohutek+outages at gmail.com> > said:
> It's weird you say that - I have been attempting to use Level3's well 
> known communities to attempt to prepend, no-export etc and have been 
> having limited luck ... and those communities even appear in 
> route-views, so I know they're being sent!

I brought my session back up, not accepting any routes, and prepending ours
(just in case the change propagated)... it took a while, but eventually I do
see the prepended routes.

But for a provider that said they shut down their link to Level3, I still
see their routes... possibly (slowly) propagating changes but not actual
withdraws?
--
Chris Adams <cma at cmadams.net <mailto:cma at cmadams.net> >
_______________________________________________
Outages mailing list
Outages at outages.org <mailto:Outages at outages.org> 
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman
_listinfo_outages
<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma
n_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=
Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiS
qjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=>
&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l
2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JP
fcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= 

------------------------------

Subject: Digest Footer

_______________________________________________
Outages mailing list
Outages at outages.org <mailto:Outages at outages.org> 
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman
_listinfo_outages
<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma
n_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=
Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiS
qjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=>
&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l
2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JP
fcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= 

------------------------------

End of Outages Digest, Vol 147, Issue 10
****************************************

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20200830/f67fb9a1/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 208 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/outages/attachments/20200830/f67fb9a1/attachment.png>