[outages] Level3 10G DWDM Ckt Detroit to Chicago

Wed Jul 30 17:39:57 EDT 2014

Oddly enough, the ckt came up a few hours ago - 
I'm wondering if they re-routed us.  No notes in 
the ticket (we're getting the updates now... they had a bad email address).

At 04:53 PM 30/07/2014, Luke Rockwell wrote:
>7/30/14 8:27:19 PM GMT Repair work is ongoing at 
>the site. Updates will be provided as they are received.
>7/30/14 7:32:21 PM GMTField Services continue 
>excavation efforts and have estimated splicing 
>will begin in approximately one hour.
>7/30/14 7:10:12 PM GMT Repair efforts are 
>ongoing at the site. No estimated restoral time is currently available.
>7/30/14 6:18:25 PM GMTField Services report 
>excavation efforts are in progress to access the 
>damaged cable. The splice crew is onsite to 
>continue with repair following excavation.
>7/30/14 5:58:09 PM GMT Repair crews continue en 
>route to the site to proceed with restoration efforts.
>7/30/14 5:18:26 PM GMTA 108-count fiber cut has 
>been confirmed to have been caused by 
>construction near the railway. All necessary 
>repair crews are en route with no ETA available at this time.
>7/30/14 4:22:00 PM GMT Field Services report 
>that there is construction in the area that has 
>likely caused the fiber cut. An excavation crew 
>has been engaged to uncover the suspected 
>damaged portion of the span. No ETA has been provided for the excavation crew.
>7/30/14 4:03:40 PM GMT Field Services continue 
>investigation in conjunction with railway 
>workers to further isolate the suspected fiber cut.
>7/30/14 3:41:00 PM GMTField Services report that 
>the fault location is in close proximity to a 
>railway. As such, Field Services is coordinating 
>with railway workers to ensure safe access to 
>continue investigation. Splice crews have been 
>dispatched to the site and are expected to arrive shortly.
>7/30/14 2:58:44 PM GMT Field Services arrived in 
>Southfield, MI and detected a fault 
>approximately 52 miles from the site. Field 
>Services is proceeding en route to the fault location.
>7/30/14 2:11:36 PM GMTThe Transport NOC detected 
>a loss of signal in Dexter, MI is impacting 
>unprotected services. A fiber cut is suspected 
>and Field Services have been engaged for 
>investigation. Updates will be provided as they are received.
>
>.....................................
>Luke Rockwell
>Systems Administrator
>Information Technology
>
>.....................................
>
>Cal Alumni Association | UC Berkeley
>1 Alumni House, Berkeley, CA 94720
>TÂ 510.900.8196
>FÂ 510.642.6252
>
>.....................................
>
>Commitment, Support, Passion
>
>_____________________________________
><http://alumni.berkeley.edu/>alumni.berkeley.edu
>
>_____________________________________
><https://www.facebook.com/CalAlumni>FacebookÂ |Â 
><http://www.linkedin.com/groups?gid=70245>LinkedIn
>
>
>On Wed, Jul 30, 2014 at 9:41 AM, Bill Wichers 
><<mailto:billw at waveform.net>billw at waveform.net> wrote:
>
>This is affecting XO in Detroit too, I can see 
>them on protect path due to higher than normal latency Detroit<->Chicago.
>
>Â
>
>Â  -Bill
>
>Â
>
>From: Outages 
>[mailto:outages-bounces at outages.org] On Behalf Of Luke Rockwell via Outages
>Sent: Wednesday, July 30, 2014 12:14 PM
>To: Clayton Zekelman
>Cc: <mailto:Outages at outages.org>Outages at outages.org
>Subject: Re: [outages] Level3 10G DWDM Ckt Detroit to Chicago
>
>Â
>
>Latest update from Level 3 trouble board
>
>Â
>
>7/30/14 4:03:40 PM GMT
>
>Field Services continue investigation in 
>conjunction with railway workers to further isolate the suspected fiber cut.
>
>7/30/14 3:41:00 PM GMT
>
>Field Services report that the fault location is 
>in close proximity to a railway. As such, Field 
>Services is coordinating with railway workers to 
>ensure safe access to continue investigation. 
>Splice crews have been dispatched to the site 
>and are expected to arrive shortly.
>
>7/30/14 2:58:44 PM GMT
>
>Field Services arrived in Southfield, MI and 
>detected a fault approximately 52 miles from the 
>site. Field Services is proceeding en route to the fault location.
>
>7/30/14 2:11:36 PM GMT
>
>The Transport NOC detected a loss of signal in 
>Dexter, MI is impacting unprotected services. A 
>fiber cut is suspected and Field Services have 
>been engaged for investigation. Updates will be provided as they are received.
>
>
>.....................................
>
>Luke Rockwell
>Systems Administrator
>Information Technology
>
>.....................................
>
>Cal Alumni Association | UC Berkeley
>1 Alumni House, Berkeley, CA 94720
>TÂ 510.900.8196
>FÂ 510.642.6252
>
>.....................................
>
>Commitment, Support, Passion
>
>_____________________________________
>
><http://alumni.berkeley.edu/>alumni.berkeley.edu
>
>_____________________________________
>
><https://www.facebook.com/CalAlumni>FacebookÂ |Â 
><http://www.linkedin.com/groups?gid=70245>LinkedIn
>
>Â
>
>On Wed, Jul 30, 2014 at 8:47 AM, Clayton 
>Zekelman via Outages <<mailto:outages at outages.org>outages at outages.org> wrote:
>
>
>
>Well, the same 10 Gig DWDM is down again - 
>10:53am. Â Level3 reports a fibre cut this time 
>- near the same location as the card failure on the 19th...
>
>
>
>
>
>At 11:38 AM 19/07/2014, Clayton Zekelman via Outages wrote:
>
>Looks like our circuit came back up a few minutes ago.
>
>We took a couple of brief hits on 2 1Gig DIAs 
>out of Southfield, MI though - not sure if that was related.
>
>
>At 06:48 AM 19/07/2014, Clayton Zekelman wrote:
>
>Level3 10G DWDM Ckt from Detroit to Chicago 
>currently out of service (down at 6:15am EDT).
>
>Possibly related to some sort of transport 
>failure in Dexter, MI that happened around the same time.
>
>
>---
>
>Â
>
>---
>
>Clayton Zekelman
>Managed Network Systems Inc. (MNSi)
>3363 Tecumseh Rd. E
>Windsor, Ontario
>N8W 1H4
>
>tel. 519-985-8410
>fax. 519-985-8409 Â  Â  Â  Â
>_______________________________________________
>Outages mailing list
><mailto:Outages at outages.org>Outages at outages.org
>https://puck.nether.net/mailman/listinfo/outages
>
>Â
>

---

Clayton Zekelman
Managed Network Systems Inc. (MNSi)
3363 Tecumseh Rd. E
Windsor, Ontario
N8W 1H4

tel. 519-985-8410
fax. 519-985-8409        
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20140730/fff620d9/attachment.htm>