[outages] Comcast - Seattle (maybe other areas)

[Andrew Losey]

Level3’s experiencing a massive back hoe outage. Reports started in Seattle, but now is being reported across the West Coast - updates from there still open ticket: 

Reverse cron
10/20/14 7:10:56 PM GMT	 The IP NOC continues to work with the Transport NOC to develop additional services routes so the impacted IP traffic can be migrated to relieve some overutilization. Simultaneously, the Transport NOC has advised that Field Services have arrived onsite and are now assessing the damage while working develop a repair strategy for this case. This is no ETTR for this case at this time.
10/20/14 6:29:31 PM GMT	 The IP NOC has specified that while working to add the necessary bandwidth to mitigate to overutilization causing the IP and Voice service impact, they are also working to re-route services, when possible, to expedite service restoral. The troubleshooting process for this case is ongoing and the IP NOC is unable to provide an ETTR for this case at this time.
10/20/14 6:07:50 PM GMT	 The IPNOC and Transport NOCs efforts to increase the bandwidth between the over utilized markets is being done to expedite service restoral for the impacted markets. The Transport NOC is also working to repair the damaged cable in El Centro, CA. At this time, Field services has isolated the root cause of the break to a construction project and outside plant engineers have been dispatched to the damage site with an ETA of 18:30 GMT. A splice crew has also been dispatched to the failure site with an ETA of 20:00 GMT.
10/20/14 5:56:31 PM GMT	 The IP NOC has advised that the fiber damage was impacting their connectivity to the impacted equipment in the various markets. The IP NOC has resolved the connectivity issue and continues to work with the Transport NOC to increase the bandwidth in the impacted markets.
10/20/14 4:56:40 PM GMT	 The IP NOC has advised that this issue is now impacting multiple markets across the Western United States the event case has been adjusted to reflect the change. The IP NOC has determined that a fiber cut in El Centro, CA has caused multiple route failures between various these western. Specifically, the fiber damage impacted multiple backbones and has caused overutilization between these markets. The IP NOC has engaged the Transport NOC help mitigate the Voice and IP service failure for this case.
10/20/14 4:04:09 PM GMT	 Multiple routing protocol failures in Seattle, WA are impacting IP and voice services. The IP NOC has engaged Senior Level Management to help troubleshoot this issue. The root cause investigation for this case is ongoing at this time.




On Oct 20, 2014, at 4:12 PM, John Neiberger via Outages <outages at outages.org> wrote:

> Michael,
> 
> Can you share some traces?
> 
> Thanks,
> John
> 
> On Mon, Oct 20, 2014 at 1:57 PM, Michael Loftis via Outages <outages at outages.org> wrote:
> Having serious issues getting outside of Comcast's network starting
> around 1250 Pacific time from Seattle area.
> 
> 
> --
> 
> "Genius might be described as a supreme capacity for getting its possessors
> into trouble of all kinds."
> -- Samuel Butler
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
> 
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20141020/698ea68e/attachment.htm>