[outages] AT&T <-> Level3 Packet Loss in SF Bay Area

[Cornelius_Lewis at ahm.honda.com]

According to Level 3


They have routed traffic from the problem module in the switch, and high
CPU utilization has dropped. They are currently checking with their
customers to confirm improvement.


 Regards


Neal Lewis





                                                                           
             Jonathan Lassoff                                              
             <jof at thejof.com>                                              
             Sent by:                                                   To 
             outages-bounces at o         virendra.rode at outages.org           
             utages.org                                                 cc 
                                       outages at outages.org                 
                                                                   Subject 
             03/09/2011 03:39          Re: [outages] AT&T <-> Level3       
             PM                        Packet Loss in SF Bay Area          
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           




On Wed, Mar 9, 2011 at 1:46 PM, virendra rode <virendra.rode at outages.org>
wrote:
> Hi -
>
> On 03/09/2011 11:21 AM, Jonathan Lassoff wrote:
>>
>> We've been observing a small about of intermittent packet loss to/from
>> AT&T and Level3 here in the San Francisco Bay Area this morning. In
>> examining the situation, it seems like a evenly-spaced dropping of
>> packets to certain IPs.
>
> -------------------
> We experienced similar issue last week off our level3 connection for a
> specific /21 customer block. After further troubleshooting we decided to
> re-route our /21 to the other provider where packet loss issue went away.
We
> are still scratching our heads over this issue.

Hrm. That's been my solution for this type of problem as well: just
temporarily switch providers in both directions. This can be a little
tricker if course if you're not a directly-downstream customer of the
affected transit AS.

I've been able to identify this problem by running mtr or traceroute
simultaneously to several (at least 16+) IPs that are adjacent in a
remote network and seeing different source IPs coming back for the
ICMP type 11/code 0 (TTL Exceeded in transit) responses, and
correlating loss to some subset of those IPs. Of course control plane
policing or loaded CPUs on remote routers can confound these numbers.

I feel like MTR does this especially well.

Hope that helps. Good luck. Enjoy your routers.
_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages