[outages] Level3/TW Telecom issues?

Thu Aug 27 14:10:52 EDT 2015

24 hours of on and off packet loss, more than just Chicago.   More communication would be helpful.   NOC can't seem to decide if it's transport or routing.  Any insights?

> On Aug 27, 2015, at 1:08 PM, Bobin Joseph via Outages <outages at outages.org> wrote:
> 
> We just got another update saying they still have issues.
>  
>  
> IP services are affected across multiple markets. The IP NOC and IP and Transport Tier III are actively troubleshooting the issue. Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 
> tw telecom has identified a network failure affecting your services. tw telecom tier-2 group is investigating the issue at this time.
>  
> Thanks,
> Bobin
>  
> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of Aaron Gross via Outages
> Sent: Thursday, August 27, 2015 10:55 AM
> To: Bob Hancock
> Cc: outages at outages.org
> Subject: Re: [outages] Level3/TW Telecom issues?
>  
> We are seeing the same things that happened yesterday on our circuits in Milwaukee and Chicago areas. Lots of packet loss and large latency through Chicago exchanges.
>  
> On Thu, Aug 27, 2015 at 12:33 PM, Bob Hancock via Outages <outages at outages.org <mailto:outages at outages.org>> wrote:
> We were seeing substantial loss through Chicago this morning but it
> seems to have subsided at this time.
> 
> 
> Bob Hancock
> 
> 
> On 08/27/2015 12:19 PM, Steven Fischer via Outages wrote:
> > we are still seeing significant packet loss to 3 of our sites in the
> > Midwest.
> >
> > On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages
> > <outages at outages.org <mailto:outages at outages.org> <mailto:outages at outages.org <mailto:outages at outages.org>>> wrote:
> >
> >     We got this from Level3____
> >
> >     __ __
> >
> >     Hello,
> >
> >     The network outage has been resolved.
> >
> >     *Event Conclusion Summary*
> >
> >     Start: August 26, 2015 19:15 GMT
> >     Stop: August 27, 2015 00:14 GMT
> >
> >     Root Cause: An unknown issue in Chicago, IL was impacting IP services.
> >     Fix Action: Adjustments were made to resolve the issue and restore
> >     services.
> >
> >     Summary:
> >     The IP NOC responded to an unknown issue in Chicago, IL that was
> >     impacting IP services. Tier III Technical Support were engaged to
> >     assist with investigations, and adjustments were made to resolve the
> >     issue and stabilize traffic. Monitoring continues to ensure
> >     stability. Should further issues be experienced, please contact the
> >     Level 3 Technical Service Center.
> >
> >
> >     ____
> >
> >     __ __
> >
> >     Thanks,____
> >
> >     Bobin____
> >
> >     __ __
> >
> >     *From:*Outages [mailto:outages-bounces at outages.org <mailto:outages-bounces at outages.org>
> >     <mailto:outages-bounces at outages.org <mailto:outages-bounces at outages.org>>] *On Behalf Of *Ryan Gelobter
> >     via Outages
> >     *Sent:* Thursday, August 27, 2015 9:44 AM
> >     *To:* Rafael Possamai
> >     *Cc:* outages at outages.org <mailto:outages at outages.org> <mailto:outages at outages.org <mailto:outages at outages.org>>
> >     *Subject:* Re: [outages] Level3/TW Telecom issues?____
> >
> >     __ __
> >
> >     Was anyone able to get some useful information out of Level3
> >     regarding this yesterday? Also is anyone else seeing similar issues?____
> >
> >     On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages"
> >     <outages at outages.org <mailto:outages at outages.org> <mailto:outages at outages.org <mailto:outages at outages.org>>> wrote:____
> >
> >     Here's a graph showing some packet loss: http://imgur.com/KVAzetA <http://imgur.com/KVAzetA>
> >     <http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjr1J7HFkr8kJ6h3PtPo0eiF5KfbU1gp41jVy1KHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrehG_98MC <http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjr1J7HFkr8kJ6h3PtPo0eiF5KfbU1gp41jVy1KHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrehG_98MC>>____
> >
> >     __ __
> >
> >     __ __
> >
> >     On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages
> >     <outages at outages.org <mailto:outages at outages.org> <mailto:outages at outages.org <mailto:outages at outages.org>>> wrote:____
> >
> >     Anyone else having issues on Level3?  We’re seeing packet loss with
> >     anything that traverses Chicago.____
> >
> >     __ __
> >
> >     Thanks,____
> >
> >     Matt____
> >
> >     This message and attachment(s), if any, is intended for the sole use
> >     of the individual and/or entity for which it is addressed, and may
> >     contain information that is privileged, confidential and prohibited
> >     from disclosure under applicable law. If you are not the addressee,
> >     or authorized to receive this on behalf of the addressee, you are
> >     hereby notified that you may not use, copy, disclose or distribute
> >     to anyone this message or any part thereof. If you have received
> >     this in error, please immediately advise the sender by e-mail and
> >     delete this information and all attachments from your computer and
> >     network. Thank you. ____
> >
> >
> >     _______________________________________________
> >     Outages mailing list
> >     Outages at outages.org <mailto:Outages at outages.org> <mailto:Outages at outages.org <mailto:Outages at outages.org>>
> >     https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
> >     <http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJcSMrhWWl6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdBPdH <http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJcSMrhWWl6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdBPdH>>____
> >
> >     __ __
> >
> >
> >     _______________________________________________
> >     Outages mailing list
> >     Outages at outages.org <mailto:Outages at outages.org> <mailto:Outages at outages.org <mailto:Outages at outages.org>>
> >     https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
> >     <http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJASMrhWWl6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWoUq <http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJASMrhWWl6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWoUq>>____
> >
> >
> >     _______________________________________________
> >     Outages mailing list
> >     Outages at outages.org <mailto:Outages at outages.org> <mailto:Outages at outages.org <mailto:Outages at outages.org>>
> >     https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
> >     <http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOUOOYrhKCy-qen4TQjqdQTbLfzDSjp0DIblIl1iJlvxdJmvD-LOVJmGLMCSHfP_nVsSYwUMWw-_R-uoovu7ctuVt5dMQsYyUeod7bdTeEyCJtdmXX_axVZicHs3jr1JwSzRQGdAamz8xVKVIDeqR4INpKNnwqj-f0T1dnoovaAVgtHBmVWh-xKHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrVbh4 <http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOUOOYrhKCy-qen4TQjqdQTbLfzDSjp0DIblIl1iJlvxdJmvD-LOVJmGLMCSHfP_nVsSYwUMWw-_R-uoovu7ctuVt5dMQsYyUeod7bdTeEyCJtdmXX_axVZicHs3jr1JwSzRQGdAamz8xVKVIDeqR4INpKNnwqj-f0T1dnoovaAVgtHBmVWh-xKHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrVbh4>>
> >
> >
> >
> >
> > --
> > To him who is able to keep you from falling and to present you before
> > his glorious presence without fault and with great joy
> >
> >
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org <mailto:Outages at outages.org>
> > http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjq9J7HFkr8kJ6h3PtPpesRG9pyPtyL0QDYu1K2qKMM-l9OwXnaJPQzZ3tmVWh-xsxlK5LE2GJPQzZe00CQm4PhOUr1vF6y0QJRrDF7W6BQQg1nPfDIU2g5d409Y_RPh1aB6_EidICzASH16Ohmp <http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjq9J7HFkr8kJ6h3PtPpesRG9pyPtyL0QDYu1K2qKMM-l9OwXnaJPQzZ3tmVWh-xsxlK5LE2GJPQzZe00CQm4PhOUr1vF6y0QJRrDF7W6BQQg1nPfDIU2g5d409Y_RPh1aB6_EidICzASH16Ohmp>
> >
> The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. It is the responsibility, however, of the recipient to ensure that it is virus free.
> _______________________________________________
> Outages mailing list
> Outages at outages.org <mailto:Outages at outages.org>
> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
>  
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20150827/c54399c0/attachment.htm>