[outages] Lightower/Sidera - NYC

Jared Geiger jared at compuwizz.net
Thu Nov 7 23:40:10 EST 2013 

Looks like it stopped OTC market trading for a bit.
http://www.cnbc.com/id/101181346


On Thu, Nov 7, 2013 at 2:54 PM, <michael.mccoy at bt.com> wrote:

> Thanks James. I was told of some sort of packet storm causing the outage
> on their MPLS network. Maybe these BPDUs have spanning tree going haywire.
> Maybe they had to isolate some PEs from the network and all of us are on
> those PEs?
>
> -----Original Message-----
> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of James
> Cornman
> Sent: Thursday, November 07, 2013 5:35 PM
> To: outages at outages.org
> Subject: Re: [outages] Lightower/Sidera - NYC
>
> This outage is still on-going. I spoke with an executive at LT earlier
> today and the word then was that some downstream customer of the
> Sidera/ConEd/RCN legacy network is leaking PVST BPDUs and its wreaking
> havoc on their network. I'm not sure how legitimate that is anymore since
> its been several hours. Apparently though, since our circuits are
> fluctuating up and down every few minutes, it would seem they're not very
> close to resolution. We're approaching the 12 hour mark for certain
> circuits.
>
> They also don't seem to have any tools to notify/communicate with their
> customers about the on-going efforts..and they don't tweet anything but
> marketing crap. The management of this incident is asinine.
>
> -James
>
> On Thu, Nov 7, 2013 at 4:54 PM, Tim Durack <tdurack at gmail.com> wrote:
> > As another datapoint, the traditional HV/Datanet/Lightower side of the
> > network has been stable during this reported outage.
> >
> >
> > On Thu, Nov 7, 2013 at 4:45 PM, Charles Sprickman <spork at bway.net>
> wrote:
> >>
> >> On Nov 7, 2013, at 4:29 PM, Vincent Aniello wrote:
> >>
> >> Our circuits stabilized mid-afternoon, but shortly after 4pm EST we
> >> started seeing issues again.
> >>
> >>
> >> Confirmed here as well, 7 out of 7 L2 tail circuits not passing
> >> traffic after a good hour of instability.
> >>
> >> FWIW, I don't recall any recent maintenance announcements from them,
> >> so this is a bit troubling (I'd expect this if we saw a big rollout
> >> of new code the night previous) since it came out of the blue.  I can
> >> also confirm we saw our first outage much earlier than they
> >> acknowledged, one circuit lost connectivity at 5:23 a.m.  I can also
> >> confirm that we did not see the physical layer go down once on any of
> >> our circuits, this seems to be occurring in the core, not the rings
> that actually service the buildings.
> >>
> >> We're legacy ConEd/RCN/Sidera, have no idea if this is happening with
> >> the LT network or not.
> >>
> >> We got what appears to be a bogus "all restored" email at 2:50 p.m.
> >>
> >> Charles
> >>
> >> --Vincent
> >>
> >>
> >>
> >> From:        Karl Clapp <kclapp at staff.gwi.net>
> >> To:        "outages at outages.org" <outages at outages.org>
> >> Date:        11/07/2013 03:23 PM
> >> Subject:        Re: [outages] Lightower/Sidera - NYC
> >> Sent by:        "Outages" <outages-bounces at outages.org>
> >> ________________________________
> >>
> >>
> >>
> >> I contacted their NOC about 45min ago. They have stated they are
> >> still working on resolution. They have reports about some services
> >> being functional, while other still experiencing packet-loss and
> >> latency. It does not sound like they have been able to identify the
> >> root cause and are still troubleshooting.
> >>
> >> Here is the number I used to get through to their NOC: 888-583-4237,
> >> option 1. From there, just follow the prompts.
> >>
> >>
> >> Fwiw, my customer circuit from NYC to NJ is still affected.
> >>
> >>
> >> On Thu, Nov 7, 2013 at 10:25 AM, John Kinsella <jlk at thrashyour.com>
> wrote:
> >> Just as a data point, LT for me is routing into Equinix NY5 over HE,
> >> so at least there's connectivity...
> >>
> >>                                 My traceroute  [v0.82]
> >> icmp (0.0.0.0)                                                Thu Nov  7
> >> 07:24:09 2013
> >> Keys:  Help   Display mode   Restart statistics   Order of fields   quit
> >>                                               Packets
> Pings
> >>  Host                                       Loss%   Snt   Last   Avg
>  Best
> >> Wrst StDev
> >>  1. 10.0.1.1                                 0.0%    29    6.7   3.2
> 0.8
> >> 12.2   2.7
> >>  2. 10.0.1.49                                0.0%    28    8.0   4.3
> 1.5
> >> 13.1   3.0
> >>  3. 10.0.24.73                               0.0%    28    6.7   4.5
> 1.9
> >> 14.7   3.4
> >>  4. 25.66.208.web-pass.com                   0.0%    28    1.9  11.2
> 1.9
> >> 48.0  12.3
> >>  5. 2-1.760.core.fre.web-pass.com            0.0%    28    9.2   6.4
> 3.1
> >> 19.7   4.0
> >>  6. gige-g6-2.core1.fmt1.he.net              0.0%    28    9.6  12.4
> 3.5
> >> 26.4   6.2
> >>  7. 10gigabitethernet1-2.core1.sjc2.he.net   0.0%    28   11.9   9.5
> 3.5
> >> 18.7   4.3
> >>  8. 10gigabitethernet9-1.core1.nyc4.he.net   0.0%    28   87.9  79.5
>  73.3
> >> 92.7   5.5
> >>  9. 10gigabitethernet4-1.core1.nyc5.he.net   0.0%    28   76.6  80.5
>  73.6
> >> 106.7   7.0
> >> 10. lightower-fiber-networks.10gigabitether  0.0%    28   78.0  82.3
>  74.4
> >> 108.0   9.4
> >> 11. xe-8-3-0.nycmnyzrj91.lightower.net       0.0%    28   89.1  83.6
>  74.2
> >> 144.7  15.8
> >> 12. ae2-jrcynj67j41.lightower.net            0.0%    28   85.4  88.6
>  80.1
> >> 131.5  12.1
> >> 13. 192.154.116.14.lightower.net             0.0%    28   85.6  83.9
>  80.2
> >> 95.6   3.5
> >> 14. ???
> >>
> >> John
> >>
> >> On Nov 7, 2013, at 7:19 AM, Karl Clapp <kclapp at staff.gwi.net> wrote:
> >>
> >> LightTower/Sidera master ticker for this is LT1323725.
> >>
> >> I was able to get through their support line now.
> >>
> >> The ETR is 1-2 hours from now.
> >>
> >> ---
> >> Karl Clapp
> >> Network Engineering Manager
> >> Biddeford Internet Corp. d.b.a. GWI (AS5760)
> >> Office: 207.602.1144
> >> Mobile: 207.641.3907
> >> Fax: 207.286.2061
> >>
> >>
> >> On Thu, Nov 7, 2013 at 9:39 AM, <SRonan at eexchange.com> wrote:
> >> Likely related to the fiber cut they experienced on their network
> >> earlier this week which they routed around.
> >>
> >> Shane Ronan, Vice President - Technology Architect State Street
> >> Global Exchange | 1230 Avenue of the Americas, 18th Fl | New York, NY
> >> 10020
> >> P  (212) 259-3023    |  M (347) 413-4503
> >> sronan at eexchange.com
> >>
> >> From: Jayson Navitsky <jnavitsky at gmail.com>
> >> Date: Thursday, November 7, 2013 at 6:55 AM
> >> To: "outages at outages.org" <outages at outages.org>
> >> Subject: [outages] Lightower/Sidera - NYC
> >>
> >> Anyone else seeing any issues in NYC with Lightower/Sidera?  Starting
> >> around 6:05AM (EST) we saw heavy packet loss across their network
> >> (from Sidera legacy) and some of our point to points bounce. The P to
> >> P's have some what stabilized (though we've seen a couple more
> >> bounces since then) the IP side of things have been pretty much
> >> unusable (we shut our session
> >> down.)  It also looks like their phone system is having a problem -
> >> calls into them seem to drop after trying to connect to the NOC -
> >> which I'm sure is pretty busy.
> >>
> >> -Jayson
> >>
> >> ________________________________
> >>
> >> http://www.statestreet.com/emaildisclaimer/
> >>
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >>
> >>
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >>
> >>
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >>
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >>
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >>
> >>
> >>
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >>
> >
> >
> >
> > --
> > Tim:>
> >
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> >
>
>
>
> --
> James Cornman
>
> Chief Technology Officer
> jcornman at atlanticmetro.net
> 212.792.9950 - ext 101
>
> Atlantic Metro Communications, Inc.
>
> 4 Century Drive, Parsippany NJ  07054
>
>
> Data Centers * Managed Hosting * Internet Access * Cloud Computing Follow
> @atlanticmetro on Twitter to learn more about daily specials.
> www.atlanticmetro.net
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20131107/d044cf87/attachment.htm>

More information about the Outages mailing list