[outages] AT&T <-> Level3 Packet Loss in SF Bay Area

[Jonathan Lassoff]

On Wed, Mar 9, 2011 at 1:46 PM, virendra rode <virendra.rode at outages.org> wrote:
> Hi -
>
> On 03/09/2011 11:21 AM, Jonathan Lassoff wrote:
>>
>> We've been observing a small about of intermittent packet loss to/from
>> AT&T and Level3 here in the San Francisco Bay Area this morning. In
>> examining the situation, it seems like a evenly-spaced dropping of
>> packets to certain IPs.
>
> -------------------
> We experienced similar issue last week off our level3 connection for a
> specific /21 customer block. After further troubleshooting we decided to
> re-route our /21 to the other provider where packet loss issue went away. We
> are still scratching our heads over this issue.

Hrm. That's been my solution for this type of problem as well: just
temporarily switch providers in both directions. This can be a little
tricker if course if you're not a directly-downstream customer of the
affected transit AS.

I've been able to identify this problem by running mtr or traceroute
simultaneously to several (at least 16+) IPs that are adjacent in a
remote network and seeing different source IPs coming back for the
ICMP type 11/code 0 (TTL Exceeded in transit) responses, and
correlating loss to some subset of those IPs. Of course control plane
policing or loaded CPUs on remote routers can confound these numbers.

I feel like MTR does this especially well.

Hope that helps. Good luck. Enjoy your routers.