<div dir="ltr">As another datapoint, the traditional HV/Datanet/Lightower side of the network has been stable during this reported outage.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Nov 7, 2013 at 4:45 PM, Charles Sprickman <span dir="ltr"><<a href="mailto:spork@bway.net" target="_blank">spork@bway.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div class="im"><div>On Nov 7, 2013, at 4:29 PM, Vincent Aniello wrote:</div><br>
<blockquote type="cite"><font face="sans-serif">Our circuits stabilized mid-afternoon,
but shortly after 4pm EST we started seeing issues again.</font>
<br></blockquote><div><br></div></div><div>Confirmed here as well, 7 out of 7 L2 tail circuits not passing traffic after a good hour of instability.</div><div><br></div><div>FWIW, I don't recall any recent maintenance announcements from them, so this is a bit troubling (I'd expect this if we saw a big rollout of new code the night previous) since it came out of the blue. I can also confirm we saw our first outage much earlier than they acknowledged, one circuit lost connectivity at 5:23 a.m. I can also confirm that we did not see the physical layer go down once on any of our circuits, this seems to be occurring in the core, not the rings that actually service the buildings.</div>
<div><br></div><div>We're legacy ConEd/RCN/Sidera, have no idea if this is happening with the LT network or not.</div><div><br></div><div>We got what appears to be a bogus "all restored" email at 2:50 p.m.</div>
<div><br></div><div>Charles</div><br><blockquote type="cite"><div><div class="h5"><font face="sans-serif">--Vincent</font>
<br>
<br>
<br>
<br><font size="1" color="#5f5f5f" face="sans-serif">From:
</font><font size="1" face="sans-serif">Karl Clapp <<a href="mailto:kclapp@staff.gwi.net" target="_blank">kclapp@staff.gwi.net</a>></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">To:
</font><font size="1" face="sans-serif">"<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>"
<<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Date:
</font><font size="1" face="sans-serif">11/07/2013 03:23 PM</font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Subject:
</font><font size="1" face="sans-serif">Re: [outages]
Lightower/Sidera - NYC</font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Sent by:
</font><font size="1" face="sans-serif">"Outages"
<<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>></font>
<br>
<hr noshade>
<br>
<br>
<br><font face="Tahoma">I contacted their NOC about 45min ago. They
have stated they are still working on resolution. They have reports about
some services being functional, while other still experiencing packet-loss
and latency. It does not sound like they have been able to identify the
root cause and are still troubleshooting.<br>
</font>
<br><font size="3">Here is the number I</font><font face="Tahoma">
used to get through to their NOC: <a href="tel:888-583-4237" value="+18885834237" target="_blank">888-583-4237</a>, option 1. From there, just
follow the prompts.<br>
<br>
Fwiw, my customer circuit from NYC to NJ is still affected.</font><font size="3"><br>
</font>
<br><font size="3">On Thu, Nov 7, 2013 at 10:25 AM, John Kinsella <</font><a href="mailto:jlk@thrashyour.com" target="_blank"><font size="3" color="blue"><u>jlk@thrashyour.com</u></font></a><font size="3">>
wrote:</font>
<br><font size="3">Just as a data point, LT for me is routing into Equinix
NY5 over HE, so at least there's connectivity...</font>
<br>
<br><font size="3">
My traceroute [v0.82]</font>
<br><font size="3">icmp (0.0.0.0)
Thu Nov 7 07:24:09
2013</font>
<br><font size="3">Keys: Help Display mode Restart statistics
Order of fields quit</font>
<br><font size="3">
Packets
Pings</font>
<br><font size="3"> Host
Loss% Snt Last Avg Best Wrst StDev</font>
<br><font size="3"> 1. 10.0.1.1
0.0%
29 6.7 3.2 0.8 12.2
2.7</font>
<br><font size="3"> 2. 10.0.1.49
0.0%
28 8.0 4.3 1.5 13.1
3.0</font>
<br><font size="3"> 3. 10.0.24.73
0.0%
28 6.7 4.5 1.9 14.7
3.4</font>
<br><font size="3"> 4. 25.66.208.</font><a href="http://web-pass.com/" target="_blank"><font size="3" color="blue"><u>web-pass.com</u></font></a><font size="3">
0.0%
28 1.9 11.2 1.9 48.0 12.3</font>
<br><font size="3"> 5. 2-1.760.</font><a href="http://core.fre.web-pass.com/" target="_blank"><font size="3" color="blue"><u>core.fre.web-pass.com</u></font></a><font size="3">
0.0% 28 9.2
6.4 3.1 19.7 4.0</font>
<br><font size="3"> 6. gige-g6-2.</font><a href="http://core1.fmt1.he.net/" target="_blank"><font size="3" color="blue"><u>core1.fmt1.he.net</u></font></a><font size="3">
0.0% 28
9.6 12.4 3.5 26.4 6.2</font>
<br><font size="3"> 7. 10gigabitethernet1-2.</font><a href="http://core1.sjc2.he.net/" target="_blank"><font size="3" color="blue"><u>core1.sjc2.he.net</u></font></a><font size="3">
0.0% 28 11.9 9.5 3.5 18.7
4.3</font>
<br><font size="3"> 8. 10gigabitethernet9-1.</font><a href="http://core1.nyc4.he.net/" target="_blank"><font size="3" color="blue"><u>core1.nyc4.he.net</u></font></a><font size="3">
0.0% 28 87.9 79.5 73.3 92.7
5.5</font>
<br><font size="3"> 9. 10gigabitethernet4-1.</font><a href="http://core1.nyc5.he.net/" target="_blank"><font size="3" color="blue"><u>core1.nyc5.he.net</u></font></a><font size="3">
0.0% 28 76.6 80.5 73.6 106.7
7.0</font>
<br><font size="3">10. lightower-fiber-networks.10gigabitether 0.0%
28 78.0 82.3 74.4 108.0 9.4</font>
<br><font size="3">11. </font><a href="http://xe-8-3-0.nycmnyzrj91.lightower.net/" target="_blank"><font size="3" color="blue"><u>xe-8-3-0.nycmnyzrj91.lightower.net</u></font></a><font size="3">
0.0% 28 89.1 83.6 74.2
144.7 15.8</font>
<br><font size="3">12. </font><a href="http://ae2-jrcynj67j41.lightower.net/" target="_blank"><font size="3" color="blue"><u>ae2-jrcynj67j41.lightower.net</u></font></a><font size="3">
0.0% 28 85.4
88.6 80.1 131.5 12.1</font>
<br><font size="3">13. 192.154.116.14.</font><a href="http://lightower.net/" target="_blank"><font size="3" color="blue"><u>lightower.net</u></font></a><font size="3">
0.0% 28 85.6
83.9 80.2 95.6 3.5</font>
<br><font size="3">14. ???</font>
<br>
<br><font size="3" color="#8f8f8f">John</font>
<br>
<br><font size="3">On Nov 7, 2013, at 7:19 AM, Karl Clapp <</font><a href="mailto:kclapp@staff.gwi.net" target="_blank"><font size="3" color="blue"><u>kclapp@staff.gwi.net</u></font></a><font size="3">>
wrote:</font>
<br>
<br><font face="Tahoma">LightTower/Sidera master ticker for this
is LT1323725.<br>
</font>
<br><font face="Tahoma">I was able to get through their support
line now.</font>
<br><font face="Tahoma"><br>
The ETR is 1-2 hours from now.</font>
<br>
<br><font size="3" face="Tahoma">---<br>
Karl Clapp<br>
Network Engineering Manager<br>
Biddeford Internet Corp. d.b.a. GWI (AS5760)<br>
Office: </font><a href="tel:207.602.1144" target="_blank"><font size="3" color="blue" face="Tahoma"><u>207.602.1144</u></font></a><font size="3" face="Tahoma"><br>
Mobile: </font><a href="tel:207.641.3907" target="_blank"><font size="3" color="blue" face="Tahoma"><u>207.641.3907</u></font></a><font size="3" face="Tahoma"><br>
Fax: </font><a href="tel:207.286.2061" target="_blank"><font size="3" color="blue" face="Tahoma"><u>207.286.2061</u></font></a>
<br><font size="3"><br>
</font>
<br><font size="3">On Thu, Nov 7, 2013 at 9:39 AM, <</font><a href="mailto:SRonan@eexchange.com" target="_blank"><font size="3" color="blue"><u>SRonan@eexchange.com</u></font></a><font size="3">>
wrote:</font>
<br><font size="1" face="Calibri">Likely related to the fiber cut they experienced
on their network earlier this week which they routed around.</font>
<br>
<br><a name="142348a779668117_142332be2a606f1c_142330201e9fe8e9__MailAutoSig"></a><font face="Calibri">Shane
Ronan, Vice President – Technology Architect </font>
<br><font face="Calibri">State Street Global Exchange | 1230
Avenue of the Americas, 18th Fl | New York, NY 10020</font>
<br><font face="Calibri">P </font><a href="tel:%28212%29%20259-3023" target="_blank"><font color="blue" face="Calibri"><u>(212)
259-3023</u></font></a><font face="Calibri"> | M
</font><a href="tel:%28347%29%20413-4503" target="_blank"><font color="blue" face="Calibri"><u>(347)
413-4503</u></font></a>
<br><a href="mailto:sronan@eexchange.com" target="_blank"><font color="blue" face="Calibri"><u>sronan@eexchange.com</u></font></a>
<br>
<br><font face="Calibri"><b>From: </b>Jayson Navitsky <</font><a href="mailto:jnavitsky@gmail.com" target="_blank"><font color="blue" face="Calibri"><u>jnavitsky@gmail.com</u></font></a><font face="Calibri">><b><br>
Date: </b>Thursday, November 7, 2013 at 6:55 AM<b><br>
To: </b>"</font><a href="mailto:outages@outages.org" target="_blank"><font color="blue" face="Calibri"><u>outages@outages.org</u></font></a><font face="Calibri">"
<</font><a href="mailto:outages@outages.org" target="_blank"><font color="blue" face="Calibri"><u>outages@outages.org</u></font></a><font face="Calibri">><b><br>
Subject: </b>[outages] Lightower/Sidera - NYC</font>
<br>
<br><font size="1" face="Calibri">Anyone else seeing any issues in NYC with
Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet
loss across their network (from Sidera legacy) and some of our point to
points bounce. The P to P's have some what stabilized (though we've seen
a couple more bounces since then) the IP side of things have been pretty
much unusable (we shut our session down.) It also looks like their
phone system is having a problem - calls into them seem to drop after trying
to connect to the NOC - which I'm sure is pretty busy. <br>
</font>
<br><font size="1" face="Calibri">-Jayson </font>
<br><font size="1" face="Calibri"><br>
</font>
<hr><font size="1" color="blue" face="Arial"><u><br>
</u></font><a href="http://www.statestreet.com/emaildisclaimer/" target="_blank"><font size="1" color="blue" face="Arial"><u>http://www.statestreet.com/emaildisclaimer/</u></font></a>
<br><font size="3"><br>
_______________________________________________<br>
Outages mailing list</font><font size="3" color="blue"><u><br>
</u></font><a href="mailto:Outages@outages.org" target="_blank"><font size="3" color="blue"><u>Outages@outages.org</u></font></a><font size="3" color="blue"><u><br>
</u></font><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank"><font size="3" color="blue"><u>https://puck.nether.net/mailman/listinfo/outages</u></font></a><font size="3"><br>
</font>
<br>
<br><font size="3">_______________________________________________<br>
Outages mailing list</font><font size="3" color="blue"><u><br>
</u></font><a href="mailto:Outages@outages.org" target="_blank"><font size="3" color="blue"><u>Outages@outages.org</u></font></a><font size="3" color="blue"><u><br>
</u></font><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank"><font size="3" color="blue"><u>https://puck.nether.net/mailman/listinfo/outages</u></font></a>
<br>
<br><font size="3"><br>
_______________________________________________<br>
Outages mailing list</font><font size="3" color="blue"><u><br>
</u></font><a href="mailto:Outages@outages.org" target="_blank"><font size="3" color="blue"><u>Outages@outages.org</u></font></a><font size="3" color="blue"><u><br>
</u></font><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank"><font size="3" color="blue"><u>https://puck.nether.net/mailman/listinfo/outages</u></font></a><font size="3"><br>
</font>
<br><tt><font>_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
</font></tt><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank"><tt><font>https://puck.nether.net/mailman/listinfo/outages</font></tt></a><tt><font><br>
</font></tt>
<br></div></div>
_______________________________________________<br>Outages mailing list<br><a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote></div><br></div><br>_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Tim:>
</div>