[outages] AT&T <-> Level3 Packet Loss in SF Bay Area

[virendra rode]

On 03/09/2011 03:46 PM, Cornelius_Lewis at ahm.honda.com wrote:
> According to Level 3
>
>
> They have routed traffic from the problem module in the switch, and high
> CPU utilization has dropped. They are currently checking with their
> customers to confirm improvement.
>
>
>   Regards
>
>
> Neal Lewis
---------------------------
Thanks Neal!


regards,
/virendra

>
>
>
>
>
>
>               Jonathan Lassoff
>               <jof at thejof.com>
>               Sent by:                                                   To
>               outages-bounces at o         virendra.rode at outages.org
>               utages.org                                                 cc
>                                         outages at outages.org
>                                                                     Subject
>               03/09/2011 03:39          Re: [outages] AT&T<->  Level3
>               PM                        Packet Loss in SF Bay Area
>
>
>
>
>
>
>
>
>
>
> On Wed, Mar 9, 2011 at 1:46 PM, virendra rode<virendra.rode at outages.org>
> wrote:
>> Hi -
>>
>> On 03/09/2011 11:21 AM, Jonathan Lassoff wrote:
>>>
>>> We've been observing a small about of intermittent packet loss to/from
>>> AT&T and Level3 here in the San Francisco Bay Area this morning. In
>>> examining the situation, it seems like a evenly-spaced dropping of
>>> packets to certain IPs.
>>
>> -------------------
>> We experienced similar issue last week off our level3 connection for a
>> specific /21 customer block. After further troubleshooting we decided to
>> re-route our /21 to the other provider where packet loss issue went away.
> We
>> are still scratching our heads over this issue.
>
> Hrm. That's been my solution for this type of problem as well: just
> temporarily switch providers in both directions. This can be a little
> tricker if course if you're not a directly-downstream customer of the
> affected transit AS.
>
> I've been able to identify this problem by running mtr or traceroute
> simultaneously to several (at least 16+) IPs that are adjacent in a
> remote network and seeing different source IPs coming back for the
> ICMP type 11/code 0 (TTL Exceeded in transit) responses, and
> correlating loss to some subset of those IPs. Of course control plane
> policing or loaded CPUs on remote routers can confound these numbers.
>
> I feel like MTR does this especially well.
>
> Hope that helps. Good luck. Enjoy your routers.
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>