[outages] Level3/TW Telecom issues?
David Hubbard
dhubbard at dino.hostasaurus.com
Thu Aug 27 14:37:53 EDT 2015
I'm seeing traffic outbound from Level 3 Tampa going to Frankfurt
Germany currently, so they're definitely doing something wrong.
3 xe-10-3-1.bar1.Tampa1.Level3.net (4.53.172.29) 13.642 ms 13.650 ms
13.645 ms
4 ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7) 117.129 ms
117.100 ms ae-2-70.edge3.Frankfurt1.Level3.net (4.69.154.71) 116.928 ms
5 ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135) 116.724 ms
ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7) 117.079 ms
ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135) 116.756 ms
> -----Original Message-----
> From: Outages [mailto:outages-bounces at outages.org] On Behalf
> Of Matthew Topper via Outages
> Sent: Thursday, August 27, 2015 2:29 PM
> To: outages at outages.org
> Subject: Re: [outages] Level3/TW Telecom issues?
>
> Seeing on and off in upstate NY
>
>
> Matthew Topper
>
> -----Original Message-----
> From: Outages [mailto:outages-bounces at outages.org] On Behalf
> Of Jason Hellenthal via Outages
> Sent: Thursday, August 27, 2015 2:23 PM
> To: outages at outages.org
> Subject: Re: [outages] Level3/TW Telecom issues?
>
> Retracting my statement from yesterday this is still a solid
> issue here. Yes absolutely still continuing. Root Cause would
> be a great explanation at this point.
>
> > On Aug 27, 2015, at 13:10, Ryan Brooks via Outages
> <outages at outages.org> wrote:
> >
> > 24 hours of on and off packet loss, more than just Chicago.
> More communication would be helpful. NOC can't seem to
> decide if it's transport or routing. Any insights?
> >
> >> On Aug 27, 2015, at 1:08 PM, Bobin Joseph via Outages
> <outages at outages.org> wrote:
> >>
> >> We just got another update saying they still have issues.
> >>
> >>
> >> IP services are affected across multiple markets. The IP
> NOC and IP and Transport Tier III are actively
> troubleshooting the issue. Operations Engineering is engaged
> and Field Services is on site in Chicago, IL investigating the issue.
> >> tw telecom has identified a network failure affecting your
> services. tw telecom tier-2 group is investigating the issue
> at this time.
> >>
> >> Thanks,
> >> Bobin
> >>
> >> From: Outages [mailto:outages-bounces at outages.org] On
> Behalf Of Aaron
> >> Gross via Outages
> >> Sent: Thursday, August 27, 2015 10:55 AM
> >> To: Bob Hancock
> >> Cc: outages at outages.org
> >> Subject: Re: [outages] Level3/TW Telecom issues?
> >>
> >> We are seeing the same things that happened yesterday on
> our circuits in Milwaukee and Chicago areas. Lots of packet
> loss and large latency through Chicago exchanges.
> >>
> >> On Thu, Aug 27, 2015 at 12:33 PM, Bob Hancock via Outages
> <outages at outages.org> wrote:
> >> We were seeing substantial loss through Chicago this
> morning but it
> >> seems to have subsided at this time.
> >>
> >>
> >> Bob Hancock
> >>
> >>
> >> On 08/27/2015 12:19 PM, Steven Fischer via Outages wrote:
> >> > we are still seeing significant packet loss to 3 of our sites in
> >> > the Midwest.
> >> >
> >> > On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages
> >> > <outages at outages.org <mailto:outages at outages.org>> wrote:
> >> >
> >> > We got this from Level3____
> >> >
> >> > __ __
> >> >
> >> > Hello,
> >> >
> >> > The network outage has been resolved.
> >> >
> >> > *Event Conclusion Summary*
> >> >
> >> > Start: August 26, 2015 19:15 GMT
> >> > Stop: August 27, 2015 00:14 GMT
> >> >
> >> > Root Cause: An unknown issue in Chicago, IL was
> impacting IP services.
> >> > Fix Action: Adjustments were made to resolve the
> issue and restore
> >> > services.
> >> >
> >> > Summary:
> >> > The IP NOC responded to an unknown issue in Chicago,
> IL that was
> >> > impacting IP services. Tier III Technical Support
> were engaged to
> >> > assist with investigations, and adjustments were
> made to resolve the
> >> > issue and stabilize traffic. Monitoring continues to ensure
> >> > stability. Should further issues be experienced,
> please contact the
> >> > Level 3 Technical Service Center.
> >> >
> >> >
> >> > ____
> >> >
> >> > __ __
> >> >
> >> > Thanks,____
> >> >
> >> > Bobin____
> >> >
> >> > __ __
> >> >
> >> > *From:*Outages [mailto:outages-bounces at outages.org
> >> > <mailto:outages-bounces at outages.org>] *On Behalf Of
> *Ryan Gelobter
> >> > via Outages
> >> > *Sent:* Thursday, August 27, 2015 9:44 AM
> >> > *To:* Rafael Possamai
> >> > *Cc:* outages at outages.org <mailto:outages at outages.org>
> >> > *Subject:* Re: [outages] Level3/TW Telecom issues?____
> >> >
> >> > __ __
> >> >
> >> > Was anyone able to get some useful information out of Level3
> >> > regarding this yesterday? Also is anyone else seeing similar
> >> > issues?____
> >> >
> >> > On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages"
> >> > <outages at outages.org <mailto:outages at outages.org>> wrote:____
> >> >
> >> > Here's a graph showing some packet loss:
> >> > http://imgur.com/KVAzetA
> >> >
> >> >
> <http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFEL
> >> >
> CzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LO
> >> >
> VJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjr1J7HF
> >> >
> kr8kJ6h3PtPo0eiF5KfbU1gp41jVy1KHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQz
> >> > h0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrehG_98MC>____
> >> >
> >> > __ __
> >> >
> >> > __ __
> >> >
> >> > On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages
> >> > <outages at outages.org <mailto:outages at outages.org>> wrote:____
> >> >
> >> > Anyone else having issues on Level3? We're seeing
> packet loss with
> >> > anything that traverses Chicago.____
> >> >
> >> > __ __
> >> >
> >> > Thanks,____
> >> >
> >> > Matt____
> >> >
> >> > This message and attachment(s), if any, is intended
> for the sole use
> >> > of the individual and/or entity for which it is
> addressed, and may
> >> > contain information that is privileged, confidential
> and prohibited
> >> > from disclosure under applicable law. If you are not
> the addressee,
> >> > or authorized to receive this on behalf of the
> addressee, you are
> >> > hereby notified that you may not use, copy, disclose
> or distribute
> >> > to anyone this message or any part thereof. If you
> have received
> >> > this in error, please immediately advise the sender
> by e-mail and
> >> > delete this information and all attachments from
> your computer and
> >> > network. Thank you. ____
> >> >
> >> >
> >> > _______________________________________________
> >> > Outages mailing list
> >> > Outages at outages.org <mailto:Outages at outages.org>
> >> > https://puck.nether.net/mailman/listinfo/outages
> >> >
> >> >
> <http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVs
> >> >
> ppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrug
> >> >
> sotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJcSMrhWWl6O5
> >> >
> bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_j
> >> >
> w09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdBPdH>
> >> > ____
> >> >
> >> > __ __
> >> >
> >> >
> >> > _______________________________________________
> >> > Outages mailing list
> >> > Outages at outages.org <mailto:Outages at outages.org>
> >> > https://puck.nether.net/mailman/listinfo/outages
> >> >
> >> >
> <http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbV
> >> >
> EVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYK
> >> >
> rugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJASMrhWWl
> >> >
> 6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ
> >> >
> 8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWo
> >> > Uq>____
> >> >
> >> >
> >> > _______________________________________________
> >> > Outages mailing list
> >> > Outages at outages.org <mailto:Outages at outages.org>
> >> > https://puck.nether.net/mailman/listinfo/outages
> >> >
> >> >
> <http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOU
> >> >
> OOYrhKCy-qen4TQjqdQTbLfzDSjp0DIblIl1iJlvxdJmvD-LOVJmGLMCSHfP_nVsSYw
> >> >
> UMWw-_R-uoovu7ctuVt5dMQsYyUeod7bdTeEyCJtdmXX_axVZicHs3jr1JwSzRQGdAa
> >> >
> mz8xVKVIDeqR4INpKNnwqj-f0T1dnoovaAVgtHBmVWh-xKHsZ8_gKgGT2TQ1lmVWh-D
> >> >
> 00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrVbh4>
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > To him who is able to keep you from falling and to present you
> >> > before his glorious presence without fault and with great joy
> >> >
> >> >
> >> > _______________________________________________
> >> > Outages mailing list
> >> > Outages at outages.org
> >> >
> http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBB
> >> >
> USztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1Nx
> >> >
> R1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjq9J7HFkr8kJ6
> >> >
> h3PtPpesRG9pyPtyL0QDYu1K2qKMM-l9OwXnaJPQzZ3tmVWh-xsxlK5LE2GJPQzZe00
> >> >
> CQm4PhOUr1vF6y0QJRrDF7W6BQQg1nPfDIU2g5d409Y_RPh1aB6_EidICzASH16Ohmp
> >> >
> >> The information contained in this E-mail message is
> privileged, confidential, and may be protected from
> disclosure; please be aware that any unauthorized use,
> printing, copying, disclosure or dissemination of this
> communication may be subject to legal restriction or
> sanction. If you think that you have received this E-mail
> message in error, please reply to the sender. This E-mail
> message and any attachments have been scanned for viruses and
> are believed to be free of any virus or other defect that
> might affect any computer system into which it is received
> and opened. It is the responsibility, however, of the
> recipient to ensure that it is virus free.
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >>
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
>
>
> --
> Jason Hellenthal
> JJH48-ARIN
>
>
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
More information about the Outages
mailing list