[Outages-discussion] [outages] [QUAR] Re: Denver Level3 problems?

[virendra rode]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Just wondering if something broke at denver that caused traffic to shift
to a RING path (w/ a higher segmentation traffic)?

If my memory serves me correct, I thought L3 have engineered their
network to intelligently re-route traffic to a better optimized path
(lower segmentation traffic) for reasons that you've mentioned.

Has any L3 customer(s) requested for a RFO?


regards,
/virendra



Bryan Campbell wrote:
> We have a ticket open.
> 
> MTR between our network (level 3 customer) and another network (level 3
> customer) was showing that ae-1-100.ebr2.Denver1.Level3.net was running
> 50%-90% packet loss for the past 6+ hours.  Well, that doesn't really
> tell the whole story since routers routinely drop ICMP as a matter of
> due course during periods of high load.  But, point to point pings to
> far end destinations that would normally be 40-60ms were varying
> regularly (every 60 seconds) to 240-260ms.  And, of course the voice
> calls on that path sounded like crap.
> 
> What it looks like is a router that is experiencing consistently high
> CPU load that is chewing on some process every 60 seconds.  That
> explains the MTR packet loss numbers that are probably innocuous.  But,
> while the processor is chewing on that process, the throughput latency
> goes through the roof.  This is a very common thing that we have seen
> before . . . just not in the middle of the Level3 network.
> 
> The Level3 tech said that since he could not replicate the problem, he
> would leave the ticket open until tomorrow.
> 
> Bryan
> 
> 
> 
> Sean Crandall wrote:
>> Sorry to dredge up an old issue, but did anyone get anything better
>> than "PEBKAC".
>>
>> We were having some issues with voice that was going to Level3 last
>> Tuesday and trying to figure out if what you all saw was related or
>> not (I use Level3 for transit and voice termination), but since I
>> didn't open a ticket at the time (we thought it was an internal
>> issue), Level3 says they can't research the issue.
>>
>> Any help?
>>
>> -Sean
>>
>>
>> -----Original Message-----
>> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org]
>> On Behalf Of Kevin Broderick
>> Sent: Wednesday, May 19, 2010 10:24 AM
>> To: outages at outages.org
>> Subject: Re: [outages] Denver Level3 problems?
>>
>> L3 NOC called us back with reason for outage yesterday: PEBKAC.
>>
>> - K
>>
>> Bradley Bopp wrote:
>>> An internal source has informed me there was an issue with a Data
>>> Center in
>>> the Denver area in which something ³blew up² and this caused a chain
>>> effect
>>> with anything going through Denver to any other pop in the USA.
>>>
>>> The problem since has been resolved and was for a very short period.
>>>
>>> Brad
>>>
>>>
>>>
>>> On 5/18/10 2:32 PM, "Joseph Jackson" <jjackson at aninetworks.net> wrote:
>>>
>>>  
>>>> We also saw issues with customer end points that took level 3 around
>>>> the same
>>>> time but in NYC ­ LA and DFW POPS.
>>>>  
>>>> Joseph
>>>>  
>>>>  
>>>>
>>>> From: outages-bounces at outages.org
>>>> [mailto:outages-bounces at outages.org] On
>>>> Behalf Of Sean Clark
>>>> Sent: Tuesday, May 18, 2010 11:53 AM
>>>> To: Jayson Baker; randal k
>>>> Cc: outages at outages.org
>>>> Subject: Re: [outages] Denver Level3 problems?
>>>>  
>>>> We had many customer complaints and all our  IP SLA tests spiked
>>>> between 1110
>>>> and 1135 est for Level 3 in our Denver market.  But we also saw the
>>>> problems
>>>> in DAL on our level 3 peer (SLA test as well). Some event happened
>>>> in the
>>>> Denver area on level 3, we have opened a ticket with them and they
>>>> reported to
>>>> us they have had a lot of customer call in for the same thing. They
>>>> have not
>>>> given us an RFO yetŠ
>>>>  
>>>>  
>>>>
>>>> From: outages-bounces at outages.org
>>>> [mailto:outages-bounces at outages.org] On
>>>> Behalf Of Jayson Baker
>>>> Sent: Tuesday, May 18, 2010 12:31
>>>> To: randal k
>>>> Cc: outages at outages.org
>>>> Subject: Re: [outages] Denver Level3 problems?
>>>>  
>>>> We've been having issues staying connected to Google Chat for the
>>>> last hour or
>>>> so.
>>>>
>>>> And it just so happens we're connected through you.  :-)
>>>>
>>>>
>>>> Best Wishes,
>>>> Jayson A. Baker
>>>> Peak Internet
>>>>
>>>> On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek at gmail.com> wrote:
>>>> Did anybody else see Denver Level3 service take a hit this morning?
>>>> Our logs
>>>> indicated nothing, but we saw various blackholing all over their
>>>> network for
>>>> ~20 minutes, starting around 9:15am MST.
>>>>
>>>> Everybody have a great day -
>>>> Randal
>>>>
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>  
>>>>
>>>>
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>     
>>>
>>> Bradley D. Bopp
>>> Director Of  Engineering
>>> http://www.nationalnet.com
>>> http://as22384.peeringdb.net
>>>
>>> NationalNet is committed to the highest level of Customer Service
>>> available
>>> in the Web Hosting business.  If you have any questions, comments or
>>> concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free,
>>> 888-4-NATNET).
>>>  
>>>
>>>
>>>  
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>   
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFL+0aBpbZvCIJx1bcRAlHBAKDiAz8FpfEd3pOZqzuW/M7DHb1yUQCfSeix
tu4ac5VDG6IVzI0g2pNuH4U=
=S51R
-----END PGP SIGNATURE-----