[outages] Level3 ICMP Filtering in Chicago?

[Justin Krejci]

True. We had verified that the ICMP echo is not arriving to the destination, I forgot to mention that already.
Thanks for the pointers to the LG's. I can't seem to find the correct combo on their LG's to replicate this behavior. However if you use http://lg.he.net and choose the origin of Minneapolis the trace goes through this same Telia link in Chicago. 

I will try and reach out more to the parties involved.

Thank you!


________________________________________
From: Jeremy Chadwick [jdc at koitsu.org]
Sent: Monday, September 30, 2013 4:47 PM
To: Justin Krejci
Cc: outages at outages.org
Subject: Re: [outages] Level3 ICMP Filtering in Chicago?

It could be the return path that is blocking ICMP echo-reply (i.e.
destination receives ICMP echo, responds with ICMP echo-reply, but goes
out a path/interface that has it filtered).

If you can get on to 204.11.209.66 and capture ICMP (both directions)
you should be able to determine if it's even seeing the echo.

L3 and Telia both have looking glasses:

http://lookingglass.level3.net/
http://looking-glass.telia.net/

--
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

On Mon, Sep 30, 2013 at 09:07:00PM +0000, Justin Krejci wrote:
> At about 3:30PM central time today we started seeing ICMP packet loss going into Level3 networks, we believe it is likely some sort of Level3 filtering of some sort.
>
> Two traces below from the same origin to the same destination at the same time, one is ICMP and one is UDP.
>
> It is not just traceroute ICMP packets, we cannot ping our own equipment when it takes a Level3 path. Other UDP/TCP traffic so far seems unaffected.
>
> Is anyone else seeing this? This appears to be a Chicago-area connection.
>
>
>
>
> Using ICMP to trace to:
>
> 204.11.209.66
>
> Tracing to host 204.11.209.66
>
> traceroute to 204.11.209.66 (204.11.209.66), 30 hops max, 60 byte packets
>  1  216.17.34.1 (216.17.34.1) [as10242/AS10242]  0.558 ms  0.554 ms  0.552 ms
>  2  v103.usi-cr02-mpls.usinternet.com (216.17.36.25) [as10242/AS10242]  1.094 ms  1.199 ms  1.555 ms
>  3  10gigabitethernet2-6.core1.msp1.he.net (216.66.73.173) [AS6939]  1.203 ms  1.530 ms  1.533 ms
>  4  100gigabitethernet7-1.core1.chi1.he.net (184.105.223.177) [AS46841]  9.076 ms  9.084 ms  9.154 ms
>  5  chi-bb1-link.telia.net (213.248.104.213) [AS1299]  9.155 ms  9.163 ms  9.163 ms
>  6  * * *
>  7  * * *
>  8  * * *
>  9  * * *
> 10  * * *
> 11  * * *
> 12  * * *
> 13  * * *
> 14  * * *
> 15  * * *
>
>
>
>
> Using UDP to trace to:
>
> 204.11.209.66
>
> Tracing to host 204.11.209.66
>
> traceroute to 204.11.209.66 (204.11.209.66), 30 hops max, 60 byte packets
>  1  216.17.34.1 (216.17.34.1) [as10242/AS10242]  0.438 ms  0.480 ms  0.495 ms
>  2  v103.usi-cr02-mpls.usinternet.com (216.17.36.25) [as10242/AS10242]  1.106 ms  1.166 ms  1.235 ms
>  3  10gigabitethernet2-6.core1.msp1.he.net (216.66.73.173) [AS6939]  4.447 ms  4.445 ms  4.440 ms
>  4  100gigabitethernet7-1.core1.chi1.he.net (184.105.223.177) [AS46841]  12.821 ms  12.898 ms  12.893 ms
>  5  chi-bb1-link.telia.net (213.248.104.213) [AS1299]  9.056 ms  9.052 ms  9.054 ms
>  6  level3-ic-300135-chi-bb1.c.telia.net (213.248.87.238) [AS1299]  23.943 ms  24.000 ms  18.147 ms
>  7  ae-32-52.ebr2.Chicago1.Level3.net (4.69.138.62) [AS3356]  20.890 ms  18.162 ms  18.189 ms
>  8  ae-5-5.ebr2.Chicago2.Level3.net (4.69.140.194) [AS3356]  20.993 ms  20.982 ms  22.160 ms
>  9  ae-22-52.car2.Chicago2.Level3.net (4.69.138.165) [AS3356]  18.164 ms  22.165 ms  18.231 ms
> 10  RED-ANVIL-L.car2.Chicago2.Level3.net (4.30.14.162) [AS3356]  44.853 ms  44.822 ms  44.810 ms
> 11  ge-1-0-1-0.cr1.noc.ip.redanvil.net (204.15.100.173) [AS33693]  19.849 ms  19.888 ms  19.797 ms
>

> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages