<div dir="ltr"><div><div><div><div><div><br><br></div>I've been hating on Comcast this past week, because<br>they keep losing connectivity to the outside world in<br></div>Palo Alto for 3-5 minutes at a time. When it happens,<br>
</div>I can traceroute towards places like google, make it<br></div>5 or 6 hops through comcast, at which point it <br>consistently dies at 529bryant, like this:<br><br>mpetach@markbestgrew-lm:~% traceroute <a href="http://www.youtube.com">www.youtube.com</a><br>
traceroute: Warning: <a href="http://www.youtube.com">www.youtube.com</a> has multiple addresses; using 74.125.239.130<br>traceroute to <a href="http://youtube-ui.l.google.com">youtube-ui.l.google.com</a> (74.125.239.130), 64 hops max, 52 byte packets<br>
1 rtr (172.16.0.1) 1.395 ms 0.958 ms 0.727 ms<br> 2 76.102.48.1 (76.102.48.1) 113.310 ms 62.540 ms 58.273 ms<br> 3 <a href="http://te-0-1-0-1-ur05.santaclara.ca.sfba.comcast.net">te-0-1-0-1-ur05.santaclara.ca.sfba.comcast.net</a> (68.87.196.113) 9.965 ms 11.765 ms 9.556 ms<br>
4 <a href="http://te-1-1-0-13-ar01.sfsutro.ca.sfba.comcast.net">te-1-1-0-13-ar01.sfsutro.ca.sfba.comcast.net</a> (69.139.199.106) 11.827 ms<br> <a href="http://te-1-1-0-12-ar01.sfsutro.ca.sfba.comcast.net">te-1-1-0-12-ar01.sfsutro.ca.sfba.comcast.net</a> (69.139.199.94) 15.437 ms<br>
<a href="http://te-1-1-0-11-ar01.sfsutro.ca.sfba.comcast.net">te-1-1-0-11-ar01.sfsutro.ca.sfba.comcast.net</a> (69.139.199.90) 13.214 ms<br> 5 <a href="http://he-1-6-0-0-cr01.sanjose.ca.ibone.comcast.net">he-1-6-0-0-cr01.sanjose.ca.ibone.comcast.net</a> (68.86.90.157) 13.895 ms 18.774 ms 16.012 ms<br>
6 <a href="http://pos-0-3-0-0-pe01.529bryant.ca.ibone.comcast.net">pos-0-3-0-0-pe01.529bryant.ca.ibone.comcast.net</a> (68.86.87.142) 15.177 ms 17.322 ms 15.498 ms<br> 7 * * *<br> 8 * * *<br> 9 * * *<br>10 * * *<br>
11 * * *<br><br><br></div>running pings shows large scale packet loss to google:<br><br>Request timeout for icmp_seq 23270<br>Request timeout for icmp_seq 23271<br>Request timeout for icmp_seq 23272<br>64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23273 ttl=50 time=75.800 ms<br>
64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23274 ttl=50 time=141.421 ms<br>64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23275 ttl=50 time=521.741 ms<br>64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23276 ttl=50 time=253.848 ms<br>
64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23277 ttl=50 time=174.845 ms<br>64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23278 ttl=50 time=162.240 ms<br>64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23279 ttl=50 time=57.788 ms<br>
64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23280 ttl=50 time=226.018 ms<br>64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23281 ttl=50 time=134.495 ms<br>64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23282 ttl=50 time=72.048 ms<br>
64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23283 ttl=50 time=229.693 ms<br>64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23284 ttl=50 time=208.210 ms<br>64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23285 ttl=50 time=184.294 ms<br>
64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23286 ttl=50 time=71.336 ms<br>64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23287 ttl=50 time=48.964 ms<br>64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23288 ttl=50 time=273.211 ms<br>
64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23289 ttl=50 time=37.991 ms<br>64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23290 ttl=50 time=203.306 ms<br>64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23291 ttl=50 time=291.084 ms<br>
64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23292 ttl=50 time=518.671 ms<br>64 bytes from <a href="http://74.125.239.133">74.125.239.133</a>: icmp_seq=23293 ttl=50 time=199.055 ms<br>^C<br>--- <a href="http://google.com">google.com</a> ping statistics ---<br>
23294 packets transmitted, 12496 packets received, 46.4% packet loss<br>round-trip min/avg/max/stddev = 32.792/120.653/3023.649/147.378 ms<br>mpetach@markbestgrew-lm:~% <br><div><br></div><div>I was hoping I wouldn't have order another T1 to the<br>
new house, but it looks like that might be my only<br>option for stable connectivity. :(<br><br></div><div>Matt<br><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Oct 25, 2013 at 8:49 PM, Jeremy Chadwick <span dir="ltr"><<a href="mailto:jdc@koitsu.org" target="_blank">jdc@koitsu.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Here's another one of these incredibly annoying situations that baffles<br>
me (and of course figuring out who's responsible is not something I have<br>
the power to do).<br>
<br>
Been watching this one happen every day, usually between the hours of<br>
1700 and 2000 PDT (UTC-0700). Packet loss and latency lasts anywhere<br>
between 30 to 75 minutes. From today around 1915:<br>
<br>
src IP: 204.8.213.80<br>
dst IP: 76.102.14.35<br>
<br>
Host Loss% Snt Last Avg Best Wrst StDev<br>
1. --- 0.0% 161 21.5 1.4 0.3 59.3 6.0<br>
2. --- 0.0% 160 0.4 1.2 0.3 41.4 4.6<br>
3. --- 0.0% 160 0.3 2.2 0.2 44.5 6.5<br>
4. --- 0.0% 160 1.5 3.3 1.2 47.2 6.7<br>
5. <a href="http://xe0-1-0-1.core1.sv8.layer42.net" target="_blank">xe0-1-0-1.core1.sv8.layer42.net</a> 0.0% <a href="tel:160%20%20%20%202.2%20%20%202.1%20%20%201.9%20%20%202.4" value="+16022211924">160 2.2 2.1 1.9 2.4</a> 0.1<br>
6. <a href="tel:193.159.165.73" value="+19315916573">193.159.165.73</a> 0.0% <a href="tel:160%20%20%20%204.9%20%20%204.1%20%20%201.9%20%20%208.1" value="+16049411981">160 4.9 4.1 1.9 8.1</a> 1.3<br>
7. 80.156.163.154 24.5% 160 1000. 933.0 6.3 1738. 624.0<br>
8. pos-1-12-0-0-cr01.sanjose.ca.ibone.comcast 27.7% 160 987.5 932.7 7.2 1706. 647.7<br>
9. <a href="http://he-0-6-0-0-ar01.sfsutro.ca.sfba.comcast.ne" target="_blank">he-0-6-0-0-ar01.sfsutro.ca.sfba.comcast.ne</a> 26.4% 160 981.4 917.1 7.2 1719. 642.9<br>
10. te-0-4-0-10-ur05.santaclara.ca.sfba.comcas 30.8% 160 980.0 895.1 9.0 1710. 636.9<br>
11. <a href="http://te-6-0-acr03.santaclara.ca.sfba.comcast.ne" target="_blank">te-6-0-acr03.santaclara.ca.sfba.comcast.ne</a> 17.0% 160 967.3 614.0 6.9 1224. 518.1<br>
12. <a href="http://c-76-102-14-35.hsd1.ca.comcast.net" target="_blank">c-76-102-14-35.hsd1.ca.comcast.net</a> 18.9% 160 970.2 609.1 16.2 1229. 517.8<br>
<br>
<br>
src IP: 76.102.14.35<br>
dst IP: 204.8.213.80<br>
<br>
Host Loss% Snt Rcv Last Avg Best Wrst<br>
1. gw.home.lan 0.0% 563 563 0.3 0.2 0.2 1.9<br>
2. 76.102.12.1 0.0% 563 563 8.4 8.8 7.9 22.8<br>
3. te-0-2-0-5-ur06.santaclara.ca.sfba.comcast. 0.0% 563 563 8.7 8.9 8.1 23.1<br>
4. <a href="http://te-1-1-0-9-ar01.oakland.ca.sfba.comcast.net" target="_blank">te-1-1-0-9-ar01.oakland.ca.sfba.comcast.net</a> 0.0% 563 563 13.5 12.4 9.7 30.2<br>
5. <a href="http://be-90-ar01.sfsutro.ca.sfba.comcast.net" target="_blank">be-90-ar01.sfsutro.ca.sfba.comcast.net</a> 0.0% 563 563 12.5 13.7 11.1 41.5<br>
6. <a href="http://he-3-8-0-0-cr01.sanjose.ca.ibone.comcast.ne" target="_blank">he-3-8-0-0-cr01.sanjose.ca.ibone.comcast.ne</a> 0.0% 563 563 13.2 15.0 12.5 25.0<br>
7. <a href="http://be-14-pe02.11greatoaks.ca.ibone.comcast.net" target="_blank">be-14-pe02.11greatoaks.ca.ibone.comcast.net</a> 0.0% 562 562 16.5 16.5 15.6 43.1<br>
8. <a href="http://23-30-206-94-static.hfc.comcastbusiness.net" target="_blank">23-30-206-94-static.hfc.comcastbusiness.net</a> 0.0% 562 562 16.4 22.5 15.6 40.2<br>
9. <a href="http://0.xe-2-0-8.XL3.SJC7.ALTER.NET" target="_blank">0.xe-2-0-8.XL3.SJC7.ALTER.NET</a> 0.7% 562 558 20.3 29.6 15.3 124.7<br>
10. <a href="http://0.so-4-0-0.XL1.SJC1.ALTER.NET" target="_blank">0.so-4-0-0.XL1.SJC1.ALTER.NET</a> 1.1% 562 556 26.6 22.9 16.8 174.2<br>
<a href="http://0.so-7-0-0.XL1.SJC1.ALTER.NET" target="_blank">0.so-7-0-0.XL1.SJC1.ALTER.NET</a><br>
11. <a href="http://POS1-0.XR1.SJC1.ALTER.NET" target="_blank">POS1-0.XR1.SJC1.ALTER.NET</a> 23.0% 562 432 196.6 150.0 17.8 459.6<br>
12. <a href="http://193.ATM7-0.GW4.SJC1.ALTER.NET" target="_blank">193.ATM7-0.GW4.SJC1.ALTER.NET</a> 24.6% 562 424 1133. 669.3 17.9 1835.<br>
13. ???<br>
<br>
What interests me: hops #6 and #7 shown in the first mtr:<br>
<br>
<a href="tel:193.159.165.73" value="+19315916573">193.159.165.73</a> = AS3320 = Deutsche Telekom AG<br>
80.156.163.154 = AS3320 = Deutsche Telekom AG<br>
<br>
Using Layer42's own looking glass long after the issue was over:<br>
<br>
lg-west>show ip bgp 76.102.14.35<br>
BGP routing table entry for <a href="http://76.96.0.0/11" target="_blank">76.96.0.0/11</a>, version 4070745<br>
Paths: (1 available, best #1, table default)<br>
Not advertised to any peer<br>
Refresh Epoch 1<br>
8121 3320 7922<br>
69.36.239.8 from 69.36.239.8 (69.36.239.8)<br>
Origin IGP, metric 301, localpref 100, valid, external, best<br>
rx pathid: 0, tx pathid: 0x0<br>
<br>
lg-west>traceroute 76.102.14.35<br>
Type escape sequence to abort.<br>
Tracing the route to <a href="http://c-76-102-14-35.hsd1.ca.comcast.net" target="_blank">c-76-102-14-35.hsd1.ca.comcast.net</a> (76.102.14.35)<br>
VRF info: (vrf in name/id, vrf out name/id)<br>
1 69.36.229.177 [AS 8121] 1 msec 1 msec 1 msec<br>
2 <a href="http://xe0-0-0-4.core1.sv8.layer42.net" target="_blank">xe0-0-0-4.core1.sv8.layer42.net</a> (65.50.198.161) [AS 8121] 3 msec 2 msec 2 msec<br>
3 <a href="tel:193.159.165.73" value="+19315916573">193.159.165.73</a> [AS 3320] 4 msec 2 msec 3 msec<br>
4 80.156.163.154 [AS 3320] 4 msec 5 msec 3 msec<br>
5 <a href="http://pos-1-12-0-0-cr01.sanjose.ca.ibone.comcast.net" target="_blank">pos-1-12-0-0-cr01.sanjose.ca.ibone.comcast.net</a> (68.86.87.141) [AS 7922] 7 msec 6 msec 4 msec<br>
6 <a href="http://he-0-5-0-0-ar01.sfsutro.ca.sfba.comcast.net" target="_blank">he-0-5-0-0-ar01.sfsutro.ca.sfba.comcast.net</a> (68.86.91.46) [AS 7922] 6 msec 5 msec 8 msec<br>
7 <a href="http://te-0-4-0-6-ur05.santaclara.ca.sfba.comcast.net" target="_blank">te-0-4-0-6-ur05.santaclara.ca.sfba.comcast.net</a> (69.139.198.169) [AS 7922] 5 msec 6 msec 7 msec<br>
8 <a href="http://te-6-0-acr03.santaclara.ca.sfba.comcast.net" target="_blank">te-6-0-acr03.santaclara.ca.sfba.comcast.net</a> (68.86.249.66) [AS 7922] 5 msec 6 msec 5 msec<br>
9 <a href="http://c-76-102-14-35.hsd1.ca.comcast.net" target="_blank">c-76-102-14-35.hsd1.ca.comcast.net</a> (76.102.14.35) [AS 7922] 14 msec 18 msec 14 msec<br>
<br>
Two final things to note:<br>
<br>
1. When the packet loss/latency issue is not occurring (e.g. presently),<br>
the paths are still the same. If packets were actually going from<br>
Sunnyvale (Layer42) to Germany I'd expect a lot higher than 4ms-7ms<br>
latency consistently,<br>
<br>
2. The source/destination of 204.8.213.80 is a box at my place of work;<br>
border of network intentionally filters. We do peer with Layer42<br>
directly, but I have not pursued this with them yet (as said initially,<br>
it's hard for me to accurately point fingers).<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
| Jeremy Chadwick <a href="mailto:jdc@koitsu.org">jdc@koitsu.org</a> |<br>
| UNIX Systems Administrator <a href="http://jdc.koitsu.org/" target="_blank">http://jdc.koitsu.org/</a> |<br>
| Making life hard for others since 1977. PGP 4BD6C0CB |<br>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
</font></span></blockquote></div><br></div>