[outages] Comcast - Seattle (maybe other areas)
Ujjval Karihaloo
ujjval at simplesignal.com
Tue Oct 21 20:21:10 EDT 2014
Fiber cut was restored at 604 am GMT per Level3
Ujjval K
> On Oct 21, 2014, at 5:16 PM, Ian Mock via Outages <outages at outages.org> wrote:
>
> Is there an update on Comcast/Level 3? Haven't seen anything since yesterday. Still having issues with links in Delaware and Arizona.
>
> Thanks,
>
> Ian Mock
>
> -----Original Message-----
> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of outages-request at outages.org
> Sent: Monday, October 20, 2014 3:30 PM
> To: outages at outages.org
> Subject: Outages Digest, Vol 77, Issue 27
>
> Send Outages mailing list submissions to
> outages at outages.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://puck.nether.net/mailman/listinfo/outages
> or, via email, send a message with subject or body 'help' to
> outages-request at outages.org
>
> You can reach the person managing the list at
> outages-owner at outages.org
>
> When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..."
>
>
> Today's Topics:
>
> 1. Comcast - Seattle (maybe other areas) (Michael Loftis)
> 2. Re: Comcast - Seattle (maybe other areas) (John Neiberger)
> 3. Re: Comcast - Seattle (maybe other areas) (Michael Loftis)
> 4. Re: Comcast - Seattle (maybe other areas) (Michael Loftis)
> 5. Re: Comcast - Seattle (maybe other areas) (John Neiberger)
> 6. Re: Comcast - Seattle (maybe other areas) (Andrew Losey)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 20 Oct 2014 12:57:39 -0700
> From: Michael Loftis <mloftis at wgops.com>
> To: outages <outages at outages.org>
> Subject: [outages] Comcast - Seattle (maybe other areas)
> Message-ID:
> <CAHDg04swoLYLZY_KJ4P9zNo4O4BkkariQAfVhdF8iMeLW-12xQ at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> Having serious issues getting outside of Comcast's network starting around 1250 Pacific time from Seattle area.
>
>
> --
>
> "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds."
> -- Samuel Butler
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 20 Oct 2014 14:12:45 -0600
> From: John Neiberger <jneiberger at gmail.com>
> To: Michael Loftis <mloftis at wgops.com>
> Cc: outages <outages at outages.org>
> Subject: Re: [outages] Comcast - Seattle (maybe other areas)
> Message-ID:
> <CAOQYbjrVdzCWLP8ga+ii1EeCP55NOyQZr26o_1G-hgpQt9g3ow at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Michael,
>
> Can you share some traces?
>
> Thanks,
> John
>
>> On Mon, Oct 20, 2014 at 1:57 PM, Michael Loftis via Outages < outages at outages.org> wrote:
>>
>> Having serious issues getting outside of Comcast's network starting
>> around 1250 Pacific time from Seattle area.
>>
>>
>> --
>>
>> "Genius might be described as a supreme capacity for getting its
>> possessors into trouble of all kinds."
>> -- Samuel Butler
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://puck.nether.net/pipermail/outages/attachments/20141020/b4396d40/attachment-0001.html>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 20 Oct 2014 13:18:39 -0700
> From: Michael Loftis <mloftis at wgops.com>
> To: John Neiberger <jneiberger at gmail.com>
> Cc: outages <outages at outages.org>
> Subject: Re: [outages] Comcast - Seattle (maybe other areas)
> Message-ID:
> <CAHDg04tX5JLRLYKgqCK+JC7KySyji8m9oSZRpCVt2VpA-Bm8jg at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> Its like alost like this netblock (24.16.0.0/13) I'm in isn't getting announced to transit or is getting filtered at transit - just paid peerings...
>
> loneos:~ mloftis$ traceroute www.mysql.com traceroute to mysql-adc.oracle.com (137.254.60.6), 64 hops max, 52 byte packets
> 1 v6fw (10.70.0.1) 3.065 ms 0.798 ms 1.448 ms
> 2 96.120.101.1 (96.120.101.1) 12.289 ms 9.274 ms 9.744 ms
> 3 te-0-1-0-0-sur02.bellevue.wa.seattle.comcast.net (68.86.113.145)
> 11.498 ms 8.708 ms 14.447 ms
> 4 be-40-ar01.burien.wa.seattle.comcast.net (69.139.164.2) 12.184 ms
> be-1-sur03.bellevue.wa.seattle.comcast.net (68.86.96.142) 9.187 ms
> be-40-ar01.burien.wa.seattle.comcast.net (69.139.164.2) 11.595 ms
> 5 he-0-15-0-1-ar01.seattle.wa.seattle.comcast.net (68.85.240.94) 12.756 ms
> be-40-ar01.seattle.wa.seattle.comcast.net (69.139.164.6) 12.613 ms 21.782 ms
> 6 he-1-3-0-0-10-cr01.seattle.wa.ibone.comcast.net (68.86.93.165)
> 12.035 ms 12.944 ms 11.011 ms
> 7 * * *
>
> loneos:~ mloftis$ traceroute www.he.net
> traceroute to he.net (216.218.186.2), 64 hops max, 52 byte packets
> 1 v6fw (10.70.0.1) 1.205 ms 1.218 ms 1.103 ms
> 2 96.120.101.1 (96.120.101.1) 8.987 ms 8.738 ms 9.187 ms
> 3 te-0-1-0-0-sur02.bellevue.wa.seattle.comcast.net (68.86.113.145)
> 9.956 ms 9.791 ms 9.429 ms
> 4 be-40-ar01.burien.wa.seattle.comcast.net (69.139.164.2) 13.325 ms
> be-1-sur03.bellevue.wa.seattle.comcast.net (68.86.96.142) 16.625 ms 16.423 ms
> 5 be-40-ar01.seattle.wa.seattle.comcast.net (69.139.164.6) 10.978 ms 11.470 ms
> he-0-15-0-1-ar01.seattle.wa.seattle.comcast.net (68.85.240.94) 11.714 ms
> 6 he-1-3-0-0-10-cr01.seattle.wa.ibone.comcast.net (68.86.93.165)
> 14.815 ms 12.986 ms 11.557 ms
> 7 * * *
>
> loneos:~ mloftis$ traceroute www.google.com
> traceroute: Warning: www.google.com has multiple addresses; using 74.125.20.99 traceroute to www.google.com (74.125.20.99), 64 hops max, 52 byte packets
> 1 v6fw (10.70.0.1) 1.390 ms 1.103 ms 1.116 ms
> 2 96.120.101.1 (96.120.101.1) 9.317 ms 8.923 ms 10.132 ms
> 3 te-0-2-0-0-sur03.bellevue.wa.seattle.comcast.net (68.86.113.153)
> 9.267 ms 11.765 ms 10.088 ms
> 4 be-40-ar01.seattle.wa.seattle.comcast.net (69.139.164.6) 10.993 ms 23.054 ms 11.369 ms
> 5 he-1-3-0-0-10-cr01.seattle.wa.ibone.comcast.net (68.86.93.165)
> 15.023 ms 13.000 ms 14.430 ms
> 6 be-12-pe03.56marietta.ga.ibone.comcast.net (68.86.82.234) 10.968 ms 11.789 ms 14.294 ms
> 7 * * *
>
> loneos:~ mloftis$ traceroute 8.8.8.8
> traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
> 1 v6fw (10.70.0.1) 1.382 ms 1.200 ms 1.118 ms
> 2 96.120.101.1 (96.120.101.1) 7.683 ms 7.869 ms 9.136 ms
> 3 te-0-1-0-1-sur02.bellevue.wa.seattle.comcast.net (68.86.113.141)
> 9.527 ms 9.973 ms 9.871 ms
> 4 be-40-ar01.burien.wa.seattle.comcast.net (69.139.164.2) 12.974 ms
> be-1-sur03.bellevue.wa.seattle.comcast.net (68.86.96.142) 12.171 ms 11.826 ms
> 5 be-40-ar01.seattle.wa.seattle.comcast.net (69.139.164.6) 12.322 ms 10.624 ms 26.744 ms
> 6 he-1-3-0-0-10-cr01.seattle.wa.ibone.comcast.net (68.86.93.165)
> 17.635 ms 13.682 ms 12.770 ms
> 7 he-0-12-0-1-pe04.seattle.wa.ibone.comcast.net (68.86.82.230)
> 11.546 ms 13.296 ms 12.413 ms
> 8 * * *
>
> loneos:~ mloftis$ traceroute netflix.com traceroute to netflix.com (69.53.236.17), 64 hops max, 52 byte packets
> 1 v6fw (10.70.0.1) 1.420 ms 0.927 ms 0.981 ms
> 2 96.120.101.1 (96.120.101.1) 11.685 ms 15.119 ms 8.369 ms
> 3 te-0-1-0-1-sur02.bellevue.wa.seattle.comcast.net (68.86.113.141)
> 24.195 ms 9.636 ms 9.928 ms
> 4 be-40-ar01.burien.wa.seattle.comcast.net (69.139.164.2) 13.081 ms
> be-1-sur03.bellevue.wa.seattle.comcast.net (68.86.96.142) 10.842 ms
> be-40-ar01.burien.wa.seattle.comcast.net (69.139.164.2) 13.250 ms
> 5 be-40-ar01.seattle.wa.seattle.comcast.net (69.139.164.6) 11.050 ms 11.298 ms 11.361 ms
> 6 he-1-3-0-0-10-cr01.seattle.wa.ibone.comcast.net (68.86.93.165)
> 14.352 ms 34.253 ms 13.392 ms
> 7 he-0-13-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.85.162)
> 30.279 ms 31.711 ms 35.863 ms
> 8 he-0-10-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.214)
> 29.173 ms 31.250 ms 29.302 ms
> 9 * * *
>
>
>
>
>> On Mon, Oct 20, 2014 at 1:12 PM, John Neiberger <jneiberger at gmail.com> wrote:
>> Michael,
>>
>> Can you share some traces?
>>
>> Thanks,
>> John
>>
>> On Mon, Oct 20, 2014 at 1:57 PM, Michael Loftis via Outages
>> <outages at outages.org> wrote:
>>>
>>> Having serious issues getting outside of Comcast's network starting
>>> around 1250 Pacific time from Seattle area.
>>>
>>>
>>> --
>>>
>>> "Genius might be described as a supreme capacity for getting its
>>> possessors into trouble of all kinds."
>>> -- Samuel Butler
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>
>
>
> --
>
> "Genius might be described as a supreme capacity for getting its possessors
> into trouble of all kinds."
> -- Samuel Butler
>
>
> ------------------------------
>
> Message: 4
> Date: Mon, 20 Oct 2014 13:24:35 -0700
> From: Michael Loftis <mloftis at wgops.com>
> To: John Neiberger <jneiberger at gmail.com>
> Cc: outages <outages at outages.org>
> Subject: Re: [outages] Comcast - Seattle (maybe other areas)
> Message-ID:
> <CAHDg04viZqp=BptpVpOAJfjvjmup=Zh3TD4Sov0Wz5JuDOwS3g at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> It seems to have now mysteriously gone away/cleared up. Not at all
> sure what was going on there. I'd written it off initially as an
> issue with my laptop (OSX since a while back seems to need a reboot
> every few weeks if you've Parallels installed and use it - something
> inside the network stack goes nuts)
>
>> On Mon, Oct 20, 2014 at 1:12 PM, John Neiberger <jneiberger at gmail.com> wrote:
>> Michael,
>>
>> Can you share some traces?
>>
>> Thanks,
>> John
>>
>> On Mon, Oct 20, 2014 at 1:57 PM, Michael Loftis via Outages
>> <outages at outages.org> wrote:
>>>
>>> Having serious issues getting outside of Comcast's network starting
>>> around 1250 Pacific time from Seattle area.
>>>
>>>
>>> --
>>>
>>> "Genius might be described as a supreme capacity for getting its
>>> possessors
>>> into trouble of all kinds."
>>> -- Samuel Butler
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>
>
>
> --
>
> "Genius might be described as a supreme capacity for getting its possessors
> into trouble of all kinds."
> -- Samuel Butler
>
>
> ------------------------------
>
> Message: 5
> Date: Mon, 20 Oct 2014 14:25:20 -0600
> From: John Neiberger <jneiberger at gmail.com>
> To: Michael Loftis <mloftis at wgops.com>
> Cc: outages <outages at outages.org>
> Subject: Re: [outages] Comcast - Seattle (maybe other areas)
> Message-ID:
> <CAOQYbjp9SxvkpHDJzkoXFKdOv5hU5DUccvZx0gE6nHw4eOBiyA at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Okay, good to hear! That's very odd. Let me know if it happens again and
> you need someone to check into it.
>
> Thanks,
> John
>
>> On Mon, Oct 20, 2014 at 2:24 PM, Michael Loftis <mloftis at wgops.com> wrote:
>>
>> It seems to have now mysteriously gone away/cleared up. Not at all
>> sure what was going on there. I'd written it off initially as an
>> issue with my laptop (OSX since a while back seems to need a reboot
>> every few weeks if you've Parallels installed and use it - something
>> inside the network stack goes nuts)
>>
>> On Mon, Oct 20, 2014 at 1:12 PM, John Neiberger <jneiberger at gmail.com>
>> wrote:
>>> Michael,
>>>
>>> Can you share some traces?
>>>
>>> Thanks,
>>> John
>>>
>>> On Mon, Oct 20, 2014 at 1:57 PM, Michael Loftis via Outages
>>> <outages at outages.org> wrote:
>>>>
>>>> Having serious issues getting outside of Comcast's network starting
>>>> around 1250 Pacific time from Seattle area.
>>>>
>>>>
>>>> --
>>>>
>>>> "Genius might be described as a supreme capacity for getting its
>>>> possessors
>>>> into trouble of all kinds."
>>>> -- Samuel Butler
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>>
>> --
>>
>> "Genius might be described as a supreme capacity for getting its possessors
>> into trouble of all kinds."
>> -- Samuel Butler
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://puck.nether.net/pipermail/outages/attachments/20141020/5fffb78c/attachment-0001.html>
>
> ------------------------------
>
> Message: 6
> Date: Mon, 20 Oct 2014 16:29:21 -0400
> From: Andrew Losey <andrew at addthis.com>
> To: John Neiberger <jneiberger at gmail.com>
> Cc: outages <outages at outages.org>
> Subject: Re: [outages] Comcast - Seattle (maybe other areas)
> Message-ID: <15E83CA5-888D-47B1-9D5F-F93CC07316F3 at addthis.com>
> Content-Type: text/plain; charset="windows-1252"
>
> Level3?s experiencing a massive back hoe outage. Reports started in Seattle, but now is being reported across the West Coast - updates from there still open ticket:
>
> Reverse cron
> 10/20/14 7:10:56 PM GMT The IP NOC continues to work with the Transport NOC to develop additional services routes so the impacted IP traffic can be migrated to relieve some overutilization. Simultaneously, the Transport NOC has advised that Field Services have arrived onsite and are now assessing the damage while working develop a repair strategy for this case. This is no ETTR for this case at this time.
> 10/20/14 6:29:31 PM GMT The IP NOC has specified that while working to add the necessary bandwidth to mitigate to overutilization causing the IP and Voice service impact, they are also working to re-route services, when possible, to expedite service restoral. The troubleshooting process for this case is ongoing and the IP NOC is unable to provide an ETTR for this case at this time.
> 10/20/14 6:07:50 PM GMT The IPNOC and Transport NOCs efforts to increase the bandwidth between the over utilized markets is being done to expedite service restoral for the impacted markets. The Transport NOC is also working to repair the damaged cable in El Centro, CA. At this time, Field services has isolated the root cause of the break to a construction project and outside plant engineers have been dispatched to the damage site with an ETA of 18:30 GMT. A splice crew has also been dispatched to the failure site with an ETA of 20:00 GMT.
> 10/20/14 5:56:31 PM GMT The IP NOC has advised that the fiber damage was impacting their connectivity to the impacted equipment in the various markets. The IP NOC has resolved the connectivity issue and continues to work with the Transport NOC to increase the bandwidth in the impacted markets.
> 10/20/14 4:56:40 PM GMT The IP NOC has advised that this issue is now impacting multiple markets across the Western United States the event case has been adjusted to reflect the change. The IP NOC has determined that a fiber cut in El Centro, CA has caused multiple route failures between various these western. Specifically, the fiber damage impacted multiple backbones and has caused overutilization between these markets. The IP NOC has engaged the Transport NOC help mitigate the Voice and IP service failure for this case.
> 10/20/14 4:04:09 PM GMT Multiple routing protocol failures in Seattle, WA are impacting IP and voice services. The IP NOC has engaged Senior Level Management to help troubleshoot this issue. The root cause investigation for this case is ongoing at this time.
>
>
>
>
>> On Oct 20, 2014, at 4:12 PM, John Neiberger via Outages <outages at outages.org> wrote:
>>
>> Michael,
>>
>> Can you share some traces?
>>
>> Thanks,
>> John
>>
>> On Mon, Oct 20, 2014 at 1:57 PM, Michael Loftis via Outages <outages at outages.org> wrote:
>> Having serious issues getting outside of Comcast's network starting
>> around 1250 Pacific time from Seattle area.
>>
>>
>> --
>>
>> "Genius might be described as a supreme capacity for getting its possessors
>> into trouble of all kinds."
>> -- Samuel Butler
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://puck.nether.net/pipermail/outages/attachments/20141020/698ea68e/attachment.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> ------------------------------
>
> End of Outages Digest, Vol 77, Issue 27
> ***************************************
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
More information about the Outages
mailing list