<p dir="ltr">FYI<br></p>
<p dir="ltr">*** CASCADED EXTERNAL NOTES 15-Dec-2015 23:10:18 GMT From CASE: 10125524 - Event<br>
Event Conclusion Summary<br>
Outage Start: December 15, 2015 18:18 GMT<br>
Outage Stop: December 15, 2015 19:20 GMT<br>
Root Cause: A router addition and subsequent card failure<br>
Fix Action: The IP NOC made adjustments to normalize traffic.<br>
Reason for Outage (RFO) Summary: The IP NOC investigated the reported trouble<br>
and determined that the incident resulted from a router addition and subsequent<br>
card failure. Traffic between Chicago, IL and Atlanta, GA was impacted for<br>
multiple users across the East Coast. The IP NOC made adjustments to normalize<br>
traffic and no further impact is anticipated.</p>
<p dir="ltr">Corrective Actions: The defective card will be replaced during an upcoming<br>
maintenance window, but is not expected to be service affecting.<br></p>
<p dir="ltr">Mike </p>
<div class="gmail_quote">On Dec 15, 2015 12:26, "Joshua McDonald via Outages" <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div style="font-family:-webkit-standard">Just got off the phone with Level3, this basically explains the issue.</div><div style="font-family:-webkit-standard"><br></div><div style="font-family:-webkit-standard">*** CASCADED EXTERNAL NOTES 15-Dec-2015 20:20:05 GMT From CASE: 10125524 - Event</div><div style="font-family:-webkit-standard">The IP NOC reported that an unknown issue was impacting IP services for multiple users across the East Coast. Traffic between Chicago, IL and Atlanta, GA was experiencing latency and routing issues. The IP NOC made adjustments to normalize traffic. Tier III Technical Support and the IP NOC will continue investigations to isolate the root cause. A final notification will be sent when stability has been confirmed.</div></div><div><br></div><div><br></div><div><br></div><div><blockquote type="cite"><div>On Dec 15, 2015, at 2:23 PM, Ryan Rittgarn via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:</div><br><div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)"><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Coming from Atlanta to my facility in Chicago<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> 1 <1 ms <1 ms <1 ms 10.233.208.1<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> 2 <1 ms <1 ms <1 ms 172.16.0.12<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> 3 1 ms <1 ms <1 ms 68.208.9.1<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> 4 3 ms 3 ms 3 ms 12.250.9.101<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> 5 7 ms 7 ms 7 ms <span> </span><a href="http://cr1.attga.ip.att.net/" style="color:purple;text-decoration:underline" target="_blank">cr1.attga.ip.att.net</a><span> </span>[12.122.141.186]<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> 6 4 ms 7 ms 7 ms 12.122.141.233<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> 7 * * * Request timed out.<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Same thing… advertising on backup ISP and working fine. No word on my open ticket with L3<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></div><div><div style="border-style:solid none none;border-top-color:rgb(181,196,223);border-top-width:1pt;padding:3pt 0in 0in"><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><b><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:windowtext">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:windowtext"><span> </span>Outages [<a href="mailto:outages-bounces@outages.org" style="color:purple;text-decoration:underline" target="_blank">mailto:outages-bounces@outages.org</a>]<span> </span><b>On Behalf Of<span> </span></b>Bob Colon via Outages<br><b>Sent:</b><span> </span>Tuesday, December 15, 2015 1:16 PM<br><b>To:</b><span> </span><a href="mailto:outages@outages.org" style="color:purple;text-decoration:underline" target="_blank">outages@outages.org</a><br><b>Subject:</b><span> </span>Re: [outages] Level3 Chicago?<u></u><u></u></span></div></div></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><u></u> <u></u></div><p class="MsoNormal" style="margin:0in 0in 12pt;font-size:12pt;font-family:'Times New Roman',serif">Seeing the same here, my VPN endpoint is in Milwaukee, I'm trying to connect in Nashville, TN and not able to connect via AT&T Uverse. Traceroute seems to die on the ATT network, but I assume this is the last hop before hitting level 3?<br><br>traceroute to 66.195.129.28 (66.195.129.28), 30 hops max, 60 byte packets<br> 1 192.168.10.1 (192.168.10.1) 0.756 ms 1.188 ms 1.099 ms<br> 2 <span> </span><a href="http://99-98-180-2.lightspeed.nsvltn.sbcglobal.net/" style="color:purple;text-decoration:underline" target="_blank">99-98-180-2.lightspeed.nsvltn.sbcglobal.net</a><span> </span>(99.98.180.2) 20.989 ms 21.522 ms 22.053 ms<br> 3 99.131.205.92 (99.131.205.92) 22.597 ms 23.212 ms 23.743 ms<br> 4 12.83.112.21 (12.83.112.21) 27.580 ms 24.973 ms 28.006 ms<br> 5 12.122.141.233 (12.122.141.233) 33.381 ms 37.744 ms 38.289 ms<br> 6 * * *<br> 7 * * *<br> 8 * * *<br> 9 * * *<u></u><u></u></p><div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><u></u> <u></u></div><div style="margin:0in 0in 0.0001pt;font-size:8.5pt;font-family:Arial,sans-serif"><span><span style="color:gray">To ensure prompt service, please use "Reply All" when responding<span> </span></span></span><span style="color:gray"><br><br><span>Robert Colon | TCSA Tier 3 | EthoStream LLC | P: <a href="tel:%28877%29%20282-2519%20x2114" value="+18772822519" target="_blank">(877) 282-2519 x2114</a> | F: <a href="tel:%28414%29%20258-8307" value="+14142588307" target="_blank">(414) 258-8307</a> | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 |<span> </span><a href="http://www.telkonet.com/" style="color:purple;text-decoration:underline" target="_blank">www.telkonet.com</a><span> </span>|<span> </span><a href="http://www.ethostream.com/" style="color:purple;text-decoration:underline" target="_blank">www.ethostream.com</a><span> </span>|<span> </span><a href="https://twitter.com/telkonet" style="color:purple;text-decoration:underline" target="_blank">@Telkonet</a></span></span><u></u><u></u></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><u></u> <u></u></div></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">On 12/15/2015 1:11 PM, Savage, Tony via Outages wrote:<u></u><u></u></div></div><blockquote style="margin-top:5pt;margin-bottom:5pt"><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">My datacenter is in Madison Wisconsin and we are experiencing issues from stores across the US having issues connecting to the VPN. Looks like there is definitely something going on there. Working on getting a root cause.<u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'"><u></u> <u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">-----Original Message-----<u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">From: Outages [<a href="mailto:outages-bounces@outages.org" style="color:purple;text-decoration:underline" target="_blank">mailto:outages-bounces@outages.org</a>] On Behalf Of Chris Adams via Outages<u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">Sent: Tuesday, December 15, 2015 1:07 PM<u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">To: <a href="mailto:outages@outages.org" style="color:purple;text-decoration:underline" target="_blank">outages@outages.org</a><u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">Subject: [outages] Level3 Chicago?<u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'"><u></u> <u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">We were having trouble with reaching some customer sites through Level3 Chicago (one customer is connected to TWTC for example). We shut down our BGP and all is well.<u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'"><u></u> <u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">Someone is contacting Level3 now, but curious if anyone else is seeing similar.<u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'"><u></u> <u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">--<u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">Chris Adams <a href="mailto:cma@cmadams.net" style="color:purple;text-decoration:underline" target="_blank"><cma@cmadams.net></a><u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">_______________________________________________<u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">Outages mailing list<u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'"><a href="mailto:Outages@outages.org" style="color:purple;text-decoration:underline" target="_blank">Outages@outages.org</a><u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'"><a href="https://puck.nether.net/mailman/listinfo/outages" style="color:purple;text-decoration:underline" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">_______________________________________________<u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">Outages mailing list<u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'"><a href="mailto:Outages@outages.org" style="color:purple;text-decoration:underline" target="_blank">Outages@outages.org</a><u></u><u></u></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'"><a href="https://puck.nether.net/mailman/listinfo/outages" style="color:purple;text-decoration:underline" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><u></u><u></u></pre></blockquote><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><u></u> <u></u></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="color:windowtext"><u></u> <u></u></span></div><div class="MsoNormal" align="center" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif;text-align:center"><span style="color:windowtext"><hr size="2" width="100%" align="center"></span></div><p style="margin-right:0in;margin-left:0in;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:15.5pt;font-family:Arial,sans-serif;color:windowtext"><a href="http://www.telkonet.com/" style="color:purple;text-decoration:underline" target="_blank"><span style="border:1pt solid windowtext;padding:0in;text-decoration:none"><span><~WRD000.jpg></span><span><~WRD000.jpg></span></span></a></span><a href="http://www.ethostream.com/" style="color:purple;text-decoration:underline" target="_blank"><span style="font-size:15.5pt;font-family:Arial,sans-serif;border:1pt solid windowtext;padding:0in;text-decoration:none"><span><~WRD000.jpg></span></span></a><span style="font-size:15.5pt;font-family:Arial,sans-serif;color:windowtext"><u></u><u></u></span></p><p style="margin-right:0in;margin-left:0in;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:7.5pt;font-family:Arial,sans-serif;color:gray">This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at <a href="tel:%28414%29%20223-0473" value="+14142230473" target="_blank">(414) 223-0473</a>.</span><span style="color:windowtext"><u></u><u></u></span></p></div><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important">_______________________________________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important">Outages mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)"><a href="mailto:Outages@outages.org" style="color:purple;text-decoration:underline;font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" target="_blank">Outages@outages.org</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)"><a href="https://puck.nether.net/mailman/listinfo/outages" style="color:purple;text-decoration:underline;font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)"></div></blockquote></div><br></div><br>_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br></blockquote></div>