[outages] Lightower/Sidera - NYC
James Cornman
james at atlanticmetro.net
Thu Nov 7 17:34:43 EST 2013
This outage is still on-going. I spoke with an executive at LT earlier
today and the word then was that some downstream customer of the
Sidera/ConEd/RCN legacy network is leaking PVST BPDUs and its wreaking
havoc on their network. I'm not sure how legitimate that is anymore
since its been several hours. Apparently though, since our circuits
are fluctuating up and down every few minutes, it would seem they're
not very close to resolution. We're approaching the 12 hour mark for
certain circuits.
They also don't seem to have any tools to notify/communicate with
their customers about the on-going efforts..and they don't tweet
anything but marketing crap. The management of this incident is
asinine.
-James
On Thu, Nov 7, 2013 at 4:54 PM, Tim Durack <tdurack at gmail.com> wrote:
> As another datapoint, the traditional HV/Datanet/Lightower side of the
> network has been stable during this reported outage.
>
>
> On Thu, Nov 7, 2013 at 4:45 PM, Charles Sprickman <spork at bway.net> wrote:
>>
>> On Nov 7, 2013, at 4:29 PM, Vincent Aniello wrote:
>>
>> Our circuits stabilized mid-afternoon, but shortly after 4pm EST we
>> started seeing issues again.
>>
>>
>> Confirmed here as well, 7 out of 7 L2 tail circuits not passing traffic
>> after a good hour of instability.
>>
>> FWIW, I don't recall any recent maintenance announcements from them, so
>> this is a bit troubling (I'd expect this if we saw a big rollout of new code
>> the night previous) since it came out of the blue. I can also confirm we
>> saw our first outage much earlier than they acknowledged, one circuit lost
>> connectivity at 5:23 a.m. I can also confirm that we did not see the
>> physical layer go down once on any of our circuits, this seems to be
>> occurring in the core, not the rings that actually service the buildings.
>>
>> We're legacy ConEd/RCN/Sidera, have no idea if this is happening with the
>> LT network or not.
>>
>> We got what appears to be a bogus "all restored" email at 2:50 p.m.
>>
>> Charles
>>
>> --Vincent
>>
>>
>>
>> From: Karl Clapp <kclapp at staff.gwi.net>
>> To: "outages at outages.org" <outages at outages.org>
>> Date: 11/07/2013 03:23 PM
>> Subject: Re: [outages] Lightower/Sidera - NYC
>> Sent by: "Outages" <outages-bounces at outages.org>
>> ________________________________
>>
>>
>>
>> I contacted their NOC about 45min ago. They have stated they are still
>> working on resolution. They have reports about some services being
>> functional, while other still experiencing packet-loss and latency. It does
>> not sound like they have been able to identify the root cause and are still
>> troubleshooting.
>>
>> Here is the number I used to get through to their NOC: 888-583-4237,
>> option 1. From there, just follow the prompts.
>>
>>
>> Fwiw, my customer circuit from NYC to NJ is still affected.
>>
>>
>> On Thu, Nov 7, 2013 at 10:25 AM, John Kinsella <jlk at thrashyour.com> wrote:
>> Just as a data point, LT for me is routing into Equinix NY5 over HE, so at
>> least there's connectivity...
>>
>> My traceroute [v0.82]
>> icmp (0.0.0.0) Thu Nov 7
>> 07:24:09 2013
>> Keys: Help Display mode Restart statistics Order of fields quit
>> Packets Pings
>> Host Loss% Snt Last Avg Best
>> Wrst StDev
>> 1. 10.0.1.1 0.0% 29 6.7 3.2 0.8
>> 12.2 2.7
>> 2. 10.0.1.49 0.0% 28 8.0 4.3 1.5
>> 13.1 3.0
>> 3. 10.0.24.73 0.0% 28 6.7 4.5 1.9
>> 14.7 3.4
>> 4. 25.66.208.web-pass.com 0.0% 28 1.9 11.2 1.9
>> 48.0 12.3
>> 5. 2-1.760.core.fre.web-pass.com 0.0% 28 9.2 6.4 3.1
>> 19.7 4.0
>> 6. gige-g6-2.core1.fmt1.he.net 0.0% 28 9.6 12.4 3.5
>> 26.4 6.2
>> 7. 10gigabitethernet1-2.core1.sjc2.he.net 0.0% 28 11.9 9.5 3.5
>> 18.7 4.3
>> 8. 10gigabitethernet9-1.core1.nyc4.he.net 0.0% 28 87.9 79.5 73.3
>> 92.7 5.5
>> 9. 10gigabitethernet4-1.core1.nyc5.he.net 0.0% 28 76.6 80.5 73.6
>> 106.7 7.0
>> 10. lightower-fiber-networks.10gigabitether 0.0% 28 78.0 82.3 74.4
>> 108.0 9.4
>> 11. xe-8-3-0.nycmnyzrj91.lightower.net 0.0% 28 89.1 83.6 74.2
>> 144.7 15.8
>> 12. ae2-jrcynj67j41.lightower.net 0.0% 28 85.4 88.6 80.1
>> 131.5 12.1
>> 13. 192.154.116.14.lightower.net 0.0% 28 85.6 83.9 80.2
>> 95.6 3.5
>> 14. ???
>>
>> John
>>
>> On Nov 7, 2013, at 7:19 AM, Karl Clapp <kclapp at staff.gwi.net> wrote:
>>
>> LightTower/Sidera master ticker for this is LT1323725.
>>
>> I was able to get through their support line now.
>>
>> The ETR is 1-2 hours from now.
>>
>> ---
>> Karl Clapp
>> Network Engineering Manager
>> Biddeford Internet Corp. d.b.a. GWI (AS5760)
>> Office: 207.602.1144
>> Mobile: 207.641.3907
>> Fax: 207.286.2061
>>
>>
>> On Thu, Nov 7, 2013 at 9:39 AM, <SRonan at eexchange.com> wrote:
>> Likely related to the fiber cut they experienced on their network earlier
>> this week which they routed around.
>>
>> Shane Ronan, Vice President – Technology Architect
>> State Street Global Exchange | 1230 Avenue of the Americas, 18th Fl | New
>> York, NY 10020
>> P (212) 259-3023 | M (347) 413-4503
>> sronan at eexchange.com
>>
>> From: Jayson Navitsky <jnavitsky at gmail.com>
>> Date: Thursday, November 7, 2013 at 6:55 AM
>> To: "outages at outages.org" <outages at outages.org>
>> Subject: [outages] Lightower/Sidera - NYC
>>
>> Anyone else seeing any issues in NYC with Lightower/Sidera? Starting
>> around 6:05AM (EST) we saw heavy packet loss across their network (from
>> Sidera legacy) and some of our point to points bounce. The P to P's have
>> some what stabilized (though we've seen a couple more bounces since then)
>> the IP side of things have been pretty much unusable (we shut our session
>> down.) It also looks like their phone system is having a problem - calls
>> into them seem to drop after trying to connect to the NOC - which I'm sure
>> is pretty busy.
>>
>> -Jayson
>>
>> ________________________________
>>
>> http://www.statestreet.com/emaildisclaimer/
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
>
>
> --
> Tim:>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
--
James Cornman
Chief Technology Officer
jcornman at atlanticmetro.net
212.792.9950 - ext 101
Atlantic Metro Communications, Inc.
4 Century Drive, Parsippany NJ 07054
Data Centers • Managed Hosting • Internet Access • Cloud Computing
Follow @atlanticmetro on Twitter to learn more about daily specials.
www.atlanticmetro.net
More information about the Outages
mailing list