<div dir="ltr">just got off the phone with our Level3 Service Manager - he indicated all should be well. Watching and Waiting...</div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 27, 2015 at 2:33 PM, Jameel Akari via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Thu, 27 Aug 2015, Matthew Topper via Outages wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
Seeing on and off in upstate NY<br>
<br>
<br>
Matthew Topper<br>
</blockquote>
<br></span>
Confirmed upstate NY (Albany) as well. We've rerouted in the meanwhile; were seeing lost routes and split routing back to us.<br>
<br>
-- <br>
Jameel Akari<div class="HOEnZb"><div class="h5"><br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<br>
-----Original Message-----<br>
From: Outages [mailto:<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>] On Behalf Of Jason Hellenthal via Outages<br>
Sent: Thursday, August 27, 2015 2:23 PM<br>
To: <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br>
Subject: Re: [outages] Level3/TW Telecom issues?<br>
<br>
Retracting my statement from yesterday this is still a solid issue here. Yes absolutely still continuing. Root Cause would be a great explanation at this point.<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
On Aug 27, 2015, at 13:10, Ryan Brooks via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br>
<br>
24 hours of on and off packet loss, more than just Chicago. More communication would be helpful. NOC can't seem to decide if it's transport or routing. Any insights?<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
On Aug 27, 2015, at 1:08 PM, Bobin Joseph via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br>
<br>
We just got another update saying they still have issues.<br>
<br>
<br>
IP services are affected across multiple markets. The IP NOC and IP and Transport Tier III are actively troubleshooting the issue. Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue.<br>
tw telecom has identified a network failure affecting your services. tw telecom tier-2 group is investigating the issue at this time.<br>
<br>
Thanks,<br>
Bobin<br>
<br>
From: Outages [mailto:<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>] On Behalf Of Aaron<br>
Gross via Outages<br>
Sent: Thursday, August 27, 2015 10:55 AM<br>
To: Bob Hancock<br>
Cc: <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br>
Subject: Re: [outages] Level3/TW Telecom issues?<br>
<br>
We are seeing the same things that happened yesterday on our circuits in Milwaukee and Chicago areas. Lots of packet loss and large latency through Chicago exchanges.<br>
<br>
On Thu, Aug 27, 2015 at 12:33 PM, Bob Hancock via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br>
We were seeing substantial loss through Chicago this morning but it<br>
seems to have subsided at this time.<br>
<br>
<br>
Bob Hancock<br>
<br>
<br>
On 08/27/2015 12:19 PM, Steven Fischer via Outages wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
we are still seeing significant packet loss to 3 of our sites in<br>
the Midwest.<br>
<br>
On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages<br>
<<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a> <mailto:<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>>> wrote:<br>
<br>
We got this from Level3____<br>
<br>
__ __<br>
<br>
Hello,<br>
<br>
The network outage has been resolved.<br>
<br>
*Event Conclusion Summary*<br>
<br>
Start: August 26, 2015 19:15 GMT<br>
Stop: August 27, 2015 00:14 GMT<br>
<br>
Root Cause: An unknown issue in Chicago, IL was impacting IP services.<br>
Fix Action: Adjustments were made to resolve the issue and restore<br>
services.<br>
<br>
Summary:<br>
The IP NOC responded to an unknown issue in Chicago, IL that was<br>
impacting IP services. Tier III Technical Support were engaged to<br>
assist with investigations, and adjustments were made to resolve the<br>
issue and stabilize traffic. Monitoring continues to ensure<br>
stability. Should further issues be experienced, please contact the<br>
Level 3 Technical Service Center.<br>
<br>
<br>
____<br>
<br>
__ __<br>
<br>
Thanks,____<br>
<br>
Bobin____<br>
<br>
__ __<br>
<br>
*From:*Outages [mailto:<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a><br>
<mailto:<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>>] *On Behalf Of *Ryan Gelobter<br>
via Outages<br>
*Sent:* Thursday, August 27, 2015 9:44 AM<br>
*To:* Rafael Possamai<br>
*Cc:* <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a> <mailto:<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>><br>
*Subject:* Re: [outages] Level3/TW Telecom issues?____<br>
<br>
__ __<br>
<br>
Was anyone able to get some useful information out of Level3<br>
regarding this yesterday? Also is anyone else seeing similar<br>
issues?____<br>
<br>
On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages"<br>
<<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a> <mailto:<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>>> wrote:____<br>
<br>
Here's a graph showing some packet loss: <a href="http://imgur.com/KVAzetA" target="_blank" rel="noreferrer">http://imgur.com/KVAzetA</a><br>
<br>
<<a href="http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFEL" target="_blank" rel="noreferrer">http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFEL</a><br>
CzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LO<br>
VJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjr1J7HF<br>
kr8kJ6h3PtPo0eiF5KfbU1gp41jVy1KHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQz<br>
h0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrehG_98MC>____<br>
<br>
__ __<br>
<br>
__ __<br>
<br>
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages<br>
<<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a> <mailto:<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>>> wrote:____<br>
<br>
Anyone else having issues on Level3? We’re seeing packet loss with<br>
anything that traverses Chicago.____<br>
<br>
__ __<br>
<br>
Thanks,____<br>
<br>
Matt____<br>
<br>
This message and attachment(s), if any, is intended for the sole use<br>
of the individual and/or entity for which it is addressed, and may<br>
contain information that is privileged, confidential and prohibited<br>
from disclosure under applicable law. If you are not the addressee,<br>
or authorized to receive this on behalf of the addressee, you are<br>
hereby notified that you may not use, copy, disclose or distribute<br>
to anyone this message or any part thereof. If you have received<br>
this in error, please immediately advise the sender by e-mail and<br>
delete this information and all attachments from your computer and<br>
network. Thank you. ____<br>
<br>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a> <mailto:<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a>><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank" rel="noreferrer">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
<<a href="http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVs" target="_blank" rel="noreferrer">http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVs</a><br>
ppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrug<br>
sotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJcSMrhWWl6O5<br>
bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_j<br>
w09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdBPdH><br>
____<br>
<br>
__ __<br>
<br>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a> <mailto:<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a>><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank" rel="noreferrer">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
<<a href="http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbV" target="_blank" rel="noreferrer">http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbV</a><br>
EVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYK<br>
rugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJASMrhWWl<br>
6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ<br>
8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWo<br>
Uq>____<br>
<br>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a> <mailto:<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a>><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank" rel="noreferrer">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
<<a href="http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOU" target="_blank" rel="noreferrer">http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOU</a><br>
OOYrhKCy-qen4TQjqdQTbLfzDSjp0DIblIl1iJlvxdJmvD-LOVJmGLMCSHfP_nVsSYw<br>
UMWw-_R-uoovu7ctuVt5dMQsYyUeod7bdTeEyCJtdmXX_axVZicHs3jr1JwSzRQGdAa<br>
mz8xVKVIDeqR4INpKNnwqj-f0T1dnoovaAVgtHBmVWh-xKHsZ8_gKgGT2TQ1lmVWh-D<br>
00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrVbh4><br>
<br>
<br>
<br>
<br>
--<br>
To him who is able to keep you from falling and to present you<br>
before his glorious presence without fault and with great joy<br>
<br>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBB" target="_blank" rel="noreferrer">http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBB</a><br>
USztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1Nx<br>
R1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjq9J7HFkr8kJ6<br>
h3PtPpesRG9pyPtyL0QDYu1K2qKMM-l9OwXnaJPQzZ3tmVWh-xsxlK5LE2GJPQzZe00<br>
CQm4PhOUr1vF6y0QJRrDF7W6BQQg1nPfDIU2g5d409Y_RPh1aB6_EidICzASH16Ohmp<br>
<br>
</blockquote>
The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. It is the responsibility, however, of the recipient to ensure that it is virus free.<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank" rel="noreferrer">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank" rel="noreferrer">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank" rel="noreferrer">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote>
<br>
<br>
--<br>
Jason Hellenthal<br>
JJH48-ARIN<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank" rel="noreferrer">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
</blockquote>
<br></div></div><span class="HOEnZb"><font color="#888888">
-- <br>
Jameel Akari</font></span><br>_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank" rel="noreferrer">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy</div>
</div>