[outages] Level3 Chicago?

Tue Dec 15 19:12:13 EST 2015

FYI

*** CASCADED EXTERNAL NOTES 15-Dec-2015 23:10:18 GMT From CASE: 10125524 -
Event
Event Conclusion Summary
Outage Start:  December 15, 2015 18:18 GMT
Outage Stop: December 15, 2015 19:20 GMT
Root Cause:  A router addition and subsequent card failure
Fix Action: The IP NOC made adjustments to normalize traffic.
Reason for Outage (RFO) Summary: The IP NOC investigated the reported
trouble
and determined that the incident resulted from a router addition and
subsequent
card failure. Traffic between Chicago, IL and Atlanta, GA was impacted for
multiple users across the East Coast. The IP NOC made adjustments to
normalize
traffic and no further impact is anticipated.

Corrective Actions: The defective card will be replaced during an upcoming
maintenance window, but is not expected to be service affecting.

Mike
On Dec 15, 2015 12:26, "Joshua McDonald via Outages" <outages at outages.org>
wrote:

> Just got off the phone with Level3, this basically explains the issue.
>
> *** CASCADED EXTERNAL NOTES 15-Dec-2015 20:20:05 GMT From CASE: 10125524 -
> Event
> The IP NOC reported that an unknown issue was impacting IP services for
> multiple users across the East Coast. Traffic between Chicago, IL and
> Atlanta, GA was experiencing latency and routing issues. The IP NOC made
> adjustments to normalize traffic. Tier III Technical Support and the IP NOC
> will continue investigations to isolate the root cause. A final
> notification will be sent when stability has been confirmed.
>
>
>
> On Dec 15, 2015, at 2:23 PM, Ryan Rittgarn via Outages <
> outages at outages.org> wrote:
>
> Coming from Atlanta to my facility in Chicago
>
>   1    <1 ms    <1 ms    <1 ms  10.233.208.1
>   2    <1 ms    <1 ms    <1 ms  172.16.0.12
>   3     1 ms    <1 ms    <1 ms  68.208.9.1
>   4     3 ms     3 ms     3 ms  12.250.9.101
>   5     7 ms     7 ms     7 ms  cr1.attga.ip.att.net [12.122.141.186]
>   6     4 ms     7 ms     7 ms  12.122.141.233
>   7     *        *        *     Request timed out.
>
> Same thing… advertising on backup ISP and working fine. No word on my open
> ticket with L3
>
> *From:* Outages [mailto:outages-bounces at outages.org
> <outages-bounces at outages.org>] *On Behalf Of *Bob Colon via Outages
> *Sent:* Tuesday, December 15, 2015 1:16 PM
> *To:* outages at outages.org
> *Subject:* Re: [outages] Level3 Chicago?
>
>
> Seeing the same here, my VPN endpoint is in Milwaukee, I'm trying to
> connect in Nashville, TN and not able to connect via AT&T Uverse.
> Traceroute seems to die on the ATT network, but I assume this is the last
> hop before hitting level 3?
>
> traceroute to 66.195.129.28 (66.195.129.28), 30 hops max, 60 byte packets
>  1  192.168.10.1 (192.168.10.1)  0.756 ms  1.188 ms  1.099 ms
>  2  99-98-180-2.lightspeed.nsvltn.sbcglobal.net (99.98.180.2)  20.989 ms
> 21.522 ms  22.053 ms
>  3  99.131.205.92 (99.131.205.92)  22.597 ms  23.212 ms  23.743 ms
>  4  12.83.112.21 (12.83.112.21)  27.580 ms  24.973 ms  28.006 ms
>  5  12.122.141.233 (12.122.141.233)  33.381 ms  37.744 ms  38.289 ms
>  6  * * *
>  7  * * *
>  8  * * *
>  9  * * *
>
> To ensure prompt service, please use "Reply All" when responding
>
> Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 |
> F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 |
> www.telkonet.com | www.ethostream.com | @Telkonet
> <https://twitter.com/telkonet>
>
> On 12/15/2015 1:11 PM, Savage, Tony via Outages wrote:
>
> My datacenter is in Madison Wisconsin and we are experiencing issues from stores across the US having issues connecting to the VPN.  Looks like there is definitely something going on there.  Working on getting a root cause.
>
>
>
> -----Original Message-----
>
> From: Outages [mailto:outages-bounces at outages.org <outages-bounces at outages.org>] On Behalf Of Chris Adams via Outages
>
> Sent: Tuesday, December 15, 2015 1:07 PM
>
> To: outages at outages.org
>
> Subject: [outages] Level3 Chicago?
>
>
>
> We were having trouble with reaching some customer sites through Level3 Chicago (one customer is connected to TWTC for example).  We shut down our BGP and all is well.
>
>
>
> Someone is contacting Level3 now, but curious if anyone else is seeing similar.
>
>
>
> --
>
> Chris Adams <cma at cmadams.net> <cma at cmadams.net>
>
> _______________________________________________
>
> Outages mailing list
>
> Outages at outages.org
>
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
>
> Outages mailing list
>
> Outages at outages.org
>
> https://puck.nether.net/mailman/listinfo/outages
>
>
>
> ------------------------------
>
> <~WRD000.jpg><~WRD000.jpg> <http://www.telkonet.com/><~WRD000.jpg>
> <http://www.ethostream.com/>
>
> This e-mail and any files transmitted with it are private and confidential
> and are solely for the use of the addressee. It may contain material which
> is legally privileged. If you are not the addressee or the person
> responsible for delivering to the addressee, be advised that you have
> received this e-mail in error and that any use of it is strictly
> prohibited. If you have received this e-mail in error please contact us at (414)
> 223-0473.
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20151215/ce8fc42f/attachment.htm>