<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div>
<div>
<div style="direction: ltr;">Just want to reflect (pun?) what others are saying, Centurylink wasn’t letting go of our routes.
</div>
<div><br>
</div>
<div style="direction: ltr;">As of about 2 minutes ago it seems out AS Prepend has finally began to propagate
<span id="ms-outlook-ios-cursor"></span></div>
</div>
<div><br>
</div>
<div class="ms-outlook-ios-signature">
<div style="direction: ltr;">_____</div>
<div style="direction: ltr;">Tino Montemor | Skechers USA, Inc. </div>
<div style="direction: ltr;">O: 310-406-0132 M: 909-721-6673</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Outages <outages-bounces@outages.org> on behalf of outages-request@outages.org <outages-request@outages.org><br>
<b>Sent:</b> Sunday, August 30, 2020 6:32:35 AM<br>
<b>To:</b> outages@outages.org <outages@outages.org><br>
<b>Subject:</b> Outages Digest, Vol 147, Issue 10</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Send Outages mailing list submissions to<br>
outages@outages.org<br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=">
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=</a>
<br>
or, via email, send a message with subject or body 'help' to<br>
outages-request@outages.org<br>
<br>
You can reach the person managing the list at<br>
outages-owner@outages.org<br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of Outages digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: CenturyLink peering issues? (David Hubbard)<br>
2. Re: 3356 does not WITHDRAW bgp routes (randal k)<br>
3. Re: 3356 does not WITHDRAW bgp routes (David Hubbard)<br>
4. Re: CenturyLink peering issues? (Chris Adams)<br>
5. Re: 3356 does not WITHDRAW bgp routes (randal k)<br>
6. Re: CenturyLink peering issues? (Stephen Flynn)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Sun, 30 Aug 2020 12:22:27 +0000<br>
From: David Hubbard <dhubbard@dino.hostasaurus.com><br>
To: "outages@outages.org" <outages@outages.org><br>
Subject: Re: [outages] CenturyLink peering issues?<br>
Message-ID:<br>
<D5BED54B-45C3-4EC8-9DBA-EF1276D52C73@dino.hostasaurus.com><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Ugh; seeing same thing. Have had sessions turned down for over two hours and looking glasses are still showing 3356 propagating the advertisements.<br>
<br>
?On 8/30/20, 8:11 AM, "Outages on behalf of Chris Adams via Outages" <outages-bounces@outages.org on behalf of outages@outages.org> wrote:<br>
<br>
Once upon a time, Stephen Flynn via Outages <outages@outages.org> said:<br>
> Odd part --- I disconnected my Level3 circuit at the ORL-FL facility so that I could fully failover to my other carrier link.<br>
> Level3 is still advertising my routes, even though my link and BGP session is down.<br>
<br>
I can confirm this - I shut down IPv4 BGP with Level3 in Chicago, but<br>
checking route-views and such, they're still advertising our routes (but<br>
don't know how to get to us once packets hit their network)..<br>
<br>
AS7007 all over again?<br>
-- <br>
Chris Adams <cma@cmadams.net><br>
_______________________________________________<br>
Outages mailing list<br>
Outages@outages.org<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=">
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=</a>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Sun, 30 Aug 2020 06:33:41 -0600<br>
From: randal k <rkohutek+outages@gmail.com><br>
To: outages@outages.org<br>
Subject: Re: [outages] 3356 does not WITHDRAW bgp routes<br>
Message-ID:<br>
<CANeLk7RmHnXa=EieXpahpmM966bL8==7qsywR89Yrd0fuZ9tHA@mail.gmail.com><br>
Content-Type: text/plain; charset="UTF-8"<br>
<br>
Seeing the same thing - shut a peer, still seeing those routes via<br>
3356 across multiple route-servers & looking glasses.<br>
<br>
So, do we disconnect 3356 and suffer the blackhole in hopes that it<br>
will eventually withdraw those routes, or leave it on to prevent<br>
blackholing but suffer massive packet loss to other carriers? Wow.<br>
<br>
On Sun, Aug 30, 2020 at 6:28 AM Lukas Tribus via Outages<br>
<outages@outages.org> wrote:<br>
><br>
> As previously mentioned by Stephen Flynn, 3356 does not WITHDRAW stale<br>
> bgp routes, can be confirmed with AT&T's route server at (telnet<br>
> route-server.ip.att.net).<br>
><br>
> Stale routes from 1 hour + are still announced by 3356.<br>
><br>
><br>
> This is causing blackholing.<br>
> _______________________________________________<br>
> Outages mailing list<br>
> Outages@outages.org<br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=">
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=</a>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Sun, 30 Aug 2020 12:47:57 +0000<br>
From: David Hubbard <dhubbard@dino.hostasaurus.com><br>
To: "outages@outages.org" <outages@outages.org><br>
Subject: Re: [outages] 3356 does not WITHDRAW bgp routes<br>
Message-ID:<br>
<4BA8D9E3-9A4F-45FB-AFE4-8BBB2E3C6902@dino.hostasaurus.com><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
I tried bringing two circuits back up hoping to not have blackholing, one never left idle, the other received <2000 routes, which I know from prior outages to mean that the entire region (Tampa Bay) has been BGP isolated from the rest of their network. Good
times...<br>
<br>
?On 8/30/20, 8:43 AM, "Outages on behalf of randal k via Outages" <outages-bounces@outages.org on behalf of outages@outages.org> wrote:<br>
<br>
Seeing the same thing - shut a peer, still seeing those routes via<br>
3356 across multiple route-servers & looking glasses.<br>
<br>
So, do we disconnect 3356 and suffer the blackhole in hopes that it<br>
will eventually withdraw those routes, or leave it on to prevent<br>
blackholing but suffer massive packet loss to other carriers? Wow.<br>
<br>
On Sun, Aug 30, 2020 at 6:28 AM Lukas Tribus via Outages<br>
<outages@outages.org> wrote:<br>
><br>
> As previously mentioned by Stephen Flynn, 3356 does not WITHDRAW stale<br>
> bgp routes, can be confirmed with AT&T's route server at (telnet<br>
> route-server.ip.att.net).<br>
><br>
> Stale routes from 1 hour + are still announced by 3356.<br>
><br>
><br>
> This is causing blackholing.<br>
> _______________________________________________<br>
> Outages mailing list<br>
> Outages@outages.org<br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=">
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=</a>
<br>
_______________________________________________<br>
Outages mailing list<br>
Outages@outages.org<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=">
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=</a>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 4<br>
Date: Sun, 30 Aug 2020 07:48:14 -0500<br>
From: Chris Adams <cma@cmadams.net><br>
To: outages@outages.org<br>
Subject: Re: [outages] CenturyLink peering issues?<br>
Message-ID: <20200830124814.GB29578@cmadams.net><br>
Content-Type: text/plain; charset=us-ascii<br>
<br>
Once upon a time, randal k <rkohutek+outages@gmail.com> said:<br>
> It's weird you say that - I have been attempting to use Level3's well<br>
> known communities to attempt to prepend, no-export etc and have been<br>
> having limited luck ... and those communities even appear in<br>
> route-views, so I know they're being sent!<br>
<br>
I brought my session back up, not accepting any routes, and prepending<br>
ours (just in case the change propagated)... it took a while, but<br>
eventually I do see the prepended routes.<br>
<br>
But for a provider that said they shut down their link to Level3, I<br>
still see their routes... possibly (slowly) propagating changes but not<br>
actual withdraws?<br>
-- <br>
Chris Adams <cma@cmadams.net><br>
<br>
<br>
------------------------------<br>
<br>
Message: 5<br>
Date: Sun, 30 Aug 2020 06:52:16 -0600<br>
From: randal k <rkohutek+outages@gmail.com><br>
To: outages@outages.org<br>
Subject: Re: [outages] 3356 does not WITHDRAW bgp routes<br>
Message-ID:<br>
<CANeLk7RcTGWkQp4SYJQO=j=c04dTpYSGrXTxfK9ovi_QkgyxZQ@mail.gmail.com><br>
Content-Type: text/plain; charset="UTF-8"<br>
<br>
Watching closely, after applying ^3356$ on our inbound routes, I can<br>
see that they are bouncing sessions and slowly adding in prefixes -- I<br>
have 17x 3356-originated routes in PA, and 893x in CO. And they have<br>
reset the PA BGP session numerous times.<br>
<br>
On Sun, Aug 30, 2020 at 6:33 AM randal k <rkohutek+outages@gmail.com> wrote:<br>
><br>
> Seeing the same thing - shut a peer, still seeing those routes via<br>
> 3356 across multiple route-servers & looking glasses.<br>
><br>
> So, do we disconnect 3356 and suffer the blackhole in hopes that it<br>
> will eventually withdraw those routes, or leave it on to prevent<br>
> blackholing but suffer massive packet loss to other carriers? Wow.<br>
><br>
> On Sun, Aug 30, 2020 at 6:28 AM Lukas Tribus via Outages<br>
> <outages@outages.org> wrote:<br>
> ><br>
> > As previously mentioned by Stephen Flynn, 3356 does not WITHDRAW stale<br>
> > bgp routes, can be confirmed with AT&T's route server at (telnet<br>
> > route-server.ip.att.net).<br>
> ><br>
> > Stale routes from 1 hour + are still announced by 3356.<br>
> ><br>
> ><br>
> > This is causing blackholing.<br>
> > _______________________________________________<br>
> > Outages mailing list<br>
> > Outages@outages.org<br>
> > <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=">
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=</a>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 6<br>
Date: Sun, 30 Aug 2020 13:32:31 +0000<br>
From: Stephen Flynn <sflynn@staff.atlantic.net><br>
To: "outages@outages.org" <outages@outages.org><br>
Subject: Re: [outages] CenturyLink peering issues?<br>
Message-ID:<br>
<BN6PR19MB10269D437047598C160A6724EB500@BN6PR19MB1026.namprd19.prod.outlook.com><br>
<br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Just attempted a prepend (x4) on my Level3 advertisements --- only noticed a change within the NTT network tables.<br>
Other large carriers received no advertisement changes (Telia, Hurricane Electric, AT&T)<br>
<br>
I then noticed that my BGP session was constantly flapping. I've now shutdown my circuit again. Level3 is still announcing my address space.<br>
Thank you Level3!<br>
<br>
<br>
Regards,<br>
<br>
Stephen Flynn<br>
Atlantic.Net<br>
Direct: (321) 206-1390<br>
sflynn@staff.atlantic.net<br>
<a href="http://www.atlantic.net">www.atlantic.net</a><br>
<br>
<br>
-----Original Message-----<br>
From: Outages <outages-bounces@outages.org> On Behalf Of Chris Adams via Outages<br>
Sent: Sunday, August 30, 2020 8:48 AM<br>
To: outages@outages.org<br>
Subject: Re: [outages] CenturyLink peering issues?<br>
<br>
<br>
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.<br>
<br>
<br>
Once upon a time, randal k <rkohutek+outages@gmail.com> said:<br>
> It's weird you say that - I have been attempting to use Level3's well <br>
> known communities to attempt to prepend, no-export etc and have been <br>
> having limited luck ... and those communities even appear in <br>
> route-views, so I know they're being sent!<br>
<br>
I brought my session back up, not accepting any routes, and prepending ours (just in case the change propagated)... it took a while, but eventually I do see the prepended routes.<br>
<br>
But for a provider that said they shut down their link to Level3, I still see their routes... possibly (slowly) propagating changes but not actual withdraws?<br>
--<br>
Chris Adams <cma@cmadams.net><br>
_______________________________________________<br>
Outages mailing list<br>
Outages@outages.org<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=">https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=</a>
<br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
Outages mailing list<br>
Outages@outages.org<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=">https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e=</a>
<br>
<br>
<br>
------------------------------<br>
<br>
End of Outages Digest, Vol 147, Issue 10<br>
****************************************<br>
</div>
</span></font></div>
</body>
</html>