[outages] Centurylink nationwide outage

[Andree Toonk]

While going over some of our BGP data I noticed significant instability
for AS5668 CenturyTel Internet Holdings, Inc.
See for example:
http://portal.bgpmon.net/data/AS5668-outages-may7-2013.png

Clearly visible are the routes flapping (few minute outages) between
08:06 UTC and 15:34.
Below an example for 98.125.0.0/16 (CenturyTel Service Group, LLC)

down for 3 minutes	 2013-05-07 08:06 - 2013-05-07 08:09
down for 3 minutes	 2013-05-07 08:25 - 2013-05-07 08:28
down for 3 minutes	 2013-05-07 08:53 - 2013-05-07 08:56
down for 4 minutes	 2013-05-07 09:04 - 2013-05-07 09:08
down for 4 minutes	 2013-05-07 09:20 - 2013-05-07 09:24
down for 3 minutes	 2013-05-07 10:05 - 2013-05-07 10:08
down for 5 minutes	 2013-05-07 10:11 - 2013-05-07 10:16
down for 3 minutes	 2013-05-07 10:56 - 2013-05-07 10:59
down for 3 minutes	 2013-05-07 12:14 - 2013-05-07 12:17
down for 3 minutes	 2013-05-07 12:34 - 2013-05-07 12:37
down for 4 minutes	 2013-05-07 13:02 - 2013-05-07 13:06
down for 3 minutes	 2013-05-07 13:12 - 2013-05-07 13:15
down for 3 minutes	 2013-05-07 14:11 - 2013-05-07 14:14
down for 9 minutes	 2013-05-07 14:24 - 2013-05-07 14:33
down for 3 minutes	 2013-05-07 14:45 - 2013-05-07 14:48
down for 3 minutes	 2013-05-07 14:55 - 2013-05-07 14:58
down for 3 minutes	 2013-05-07 15:24 - 2013-05-07 15:27
down for 3 minutes	 2013-05-07 15:31 - 2013-05-07 15:34

Cheers,
 Andree


.-- My secret spy satellite informs me that at 2013-05-07 9:33 AM  Frank
Bulk wrote:
> Sounds like it was scheduled maintenance gone bad:
> 	"Perrine said he spoke with CenturyLink at 6 a.m. 
> 	where they advised him the company was doing scheduled
> 	maintenance. At 7:35 a.m. they told him something
> 	had gone wrong during the maintenance and it was
> 	affecting customers in 13 states."
> http://www.tallahassee.com/article/20130507/NEWS/130507010/Centurylink-outag
> e-affecting-local-internet-provider?nclick_check=1
> 
> I bite my tongue in regards to Network Tallahasee's singlehomedness and
> Perrine's comment on his questioning of the timing of the update. 
> 
> I encourage all my competitors to stay single homed and to do their
> maintenance Sunday morning, while their vendors are at their lowest staffing
> levels.
> 
> Frank
> 
> -----Original Message-----
> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of Frank Bulk
> Sent: Tuesday, May 07, 2013 8:29 AM
> To: 'Marco Prechel'; outages at outages.org
> Subject: Re: [outages] Centurylink nationwide outage
> 
> I first saw timeouts with www.qwest.com over IPv6 at 3:08 am Central -- I
> was wondering why, now I know. =)
> 
> Frank
> 
> -----Original Message-----
> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of Marco
> Prechel
> Sent: Tuesday, May 07, 2013 7:33 AM
> To: outages at outages.org
> Subject: [outages] Centurylink nationwide outage
> 
> Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around
> 0400 EDT. 
> ANS/ABS states it's a nationwide routing issue. No ETR.
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
> 
> 
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
> 
> 
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>