<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">Those of you having issues, can you check to see if they're still occurring?</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">Thanks,</div><div class="gmail_default" style="font-family:tahoma,sans-serif">John</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 19, 2015 at 4:25 PM, Jeremy Chadwick via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Here's another destination showing the same problem (I can't even load<br>
<a href="http://www.dslreports.com/" target="_blank">http://www.dslreports.com/</a> at present -- handshake works, but either<br>
client->server GET request gets lost, or payload response gets lost,<br>
followed by TCP retrans, eventually RST from the server, rinse lather<br>
repeat), with things again looking centralised around LA.<br>
<br>
Hopefully having multiple test endpoints will come in handy.<br>
<br>
Source IP: 69.181.142.213<br>
Destination IP: 64.91.255.98<br>
<span class=""><br>
Packets Pings<br>
Host Loss% Snt Rcv Last Avg Best Wrst<br>
</span> 1. gw.home.lan (192.168.1.1) 0.0% 171 171 0.2 0.2 0.2 0.4<br>
2. 69.181.136.1 0.0% 171 171 12.7 9.6 7.7 19.2<br>
3. <a href="http://te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net" target="_blank">te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net</a> (68.86.249.69) 0.0% 171 171 9.4 9.2 8.0 17.1<br>
4. <a href="http://te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net" target="_blank">te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net</a> (162.151.78.134 0.0% 171 171 12.1 12.2 9.5 18.4<br>
5. <a href="http://he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net" target="_blank">he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net</a> (68.86.90.93) 98.8% 171 2 50456 50095 49735 50456<br>
6. <a href="http://he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net" target="_blank">he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net</a> (68.86.86.102) 2.9% 171 166 14.1 13.5 10.9 22.7<br>
7. <a href="http://he-2-12-0-0-cr01.seattle.wa.ibone.comcast.net" target="_blank">he-2-12-0-0-cr01.seattle.wa.ibone.comcast.net</a> (68.86.86.229) 1.8% 171 168 29.2 30.2 27.4 36.3<br>
8. <a href="http://he-0-15-0-0-cr01.denver.co.ibone.comcast.net" target="_blank">he-0-15-0-0-cr01.denver.co.ibone.comcast.net</a> (68.86.86.213) 9.9% 171 154 50.1 47.6 44.9 53.1<br>
9. <a href="http://he-5-13-0-0-cr01.350ecermak.il.ibone.comcast.net" target="_blank">he-5-13-0-0-cr01.350ecermak.il.ibone.comcast.net</a> (68.86.85.245) 4.1% 171 163 72.9 72.3 68.6 84.7<br>
10. <a href="http://hu-0-10-0-1.pe03.350ecermak.il.ibone.comcast.net" target="_blank">hu-0-10-0-1.pe03.350ecermak.il.ibone.comcast.net</a> (68.86.86.6) 5.3% 170 161 69.8 69.4 68.1 84.7<br>
11. 50.242.150.126 7.1% 170 158 106.0 89.7 67.4 400.5<br>
50.242.150.130<br>
12. <a href="http://lw-dc3-core1-te8-16.rtr.liquidweb.com" target="_blank">lw-dc3-core1-te8-16.rtr.liquidweb.com</a> (209.59.157.244) 9.4% 170 154 75.7 80.9 75.2 242.7<br>
13. <a href="http://lw-dc3-dist15-po5.rtr.liquidweb.com" target="_blank">lw-dc3-dist15-po5.rtr.liquidweb.com</a> (69.167.128.241) 2.4% 170 166 75.0 75.9 74.7 83.6<br>
14. <a href="http://www.dslreports.com" target="_blank">www.dslreports.com</a> (64.91.255.98) 5.3% 170 161 74.2 74.3 73.2 82.5<br>
<span class="im HOEnZb"><br>
--<br>
| Jeremy Chadwick <a href="mailto:jdc@koitsu.org">jdc@koitsu.org</a> |<br>
| UNIX Systems Administrator <a href="http://jdc.koitsu.org/" target="_blank">http://jdc.koitsu.org/</a> |<br>
| Making life hard for others since 1977. PGP 4BD6C0CB |<br>
<br>
</span><div class="HOEnZb"><div class="h5">On Thu, Feb 19, 2015 at 03:07:04PM -0800, Jeremy Chadwick wrote:<br>
> Something weird is going on with Comcast in Los Angeles: small packets<br>
> appear to be getting discarded with a loss rate of ~6-10% while larger<br>
> packets (I've tried 1000 and 1200) seem to make it through without any<br>
> loss. Things like SSH and IMAP+SSL are impacted consistently<br>
> (particularly during key exchange), while HTTP does not appear to be, so<br>
> the issue appears to be affecting both TCP and ICMP (have not tested<br>
> UDP).<br>
><br>
> First sign of this issue, according to my logs, is at 11:51 PT (UTC-8),<br>
> and has gradually gotten worse over the past few hours.<br>
><br>
> Looks like the issue is with the forward path (Source->Destination<br>
> listed below), as the return path only shows loss at the final<br>
> destination. (I currently cannot provide a return-path mtr because the<br>
> only way I can access my box is through VNC, and I don't like doing<br>
> things like screenshots of a VNC window, so you'll just have to trust<br>
> me)<br>
><br>
> Nothing anomalous shows up when using L3's looking glass to ping things,<br>
> and the only Comcast LG I know of is in NYC.<br>
><br>
> I most likely won't see Email responses from anyone (directly or to the<br>
> list) until this problem is addressed, since my mail is hosted on shown<br>
> box and cannot currently be fetched due to the problem. Frustrating.<br>
><br>
> Source IP: 69.181.142.213<br>
> Destination IP: 174.136.108.146<br>
><br>
> ICMP 56 bytes:<br>
><br>
> Packets Pings<br>
> Host Loss% Snt Rcv Last Avg Best Wrst<br>
> 1. gw.home.lan (192.168.1.1) 0.0% 185 185 0.2 0.2 0.2 0.3<br>
> 2. 69.181.136.1 0.0% 185 185 8.9 9.9 7.0 21.1<br>
> 3. <a href="http://te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net" target="_blank">te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net</a> (68.86.249.69) 0.0% 185 185 8.7 9.9 7.9 17.0<br>
> 4. <a href="http://te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net" target="_blank">te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net</a> (162.151.78.134 0.0% 185 185 10.7 12.9 9.7 20.7<br>
> 5. <a href="http://he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net" target="_blank">he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net</a> (68.86.90.93) 98.4% 184 3 49496 49006 47906 49615<br>
> 6. <a href="http://he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net" target="_blank">he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net</a> (68.86.86.102) 5.4% 184 174 11.9 14.1 10.9 23.5<br>
> 7. 50.242.148.34 6.5% 184 172 11.7 13.4 10.4 69.9<br>
> 8. <a href="http://ae-3-80.edge6.LosAngeles1.Level3.net" target="_blank">ae-3-80.edge6.LosAngeles1.Level3.net</a> (4.69.144.142) 66.7% 184 61 44.9 20.6 18.0 44.9<br>
> 9. <a href="http://ae-3-80.edge6.LosAngeles1.Level3.net" target="_blank">ae-3-80.edge6.LosAngeles1.Level3.net</a> (4.69.144.142) 59.0% 184 75 18.7 19.8 18.0 26.0<br>
> 10. <a href="http://ae-27-27.car2.LosAngeles1.Level3.net" target="_blank">ae-27-27.car2.LosAngeles1.Level3.net</a> (4.69.201.18) 2.2% 184 180 20.8 63.1 18.2 293.3<br>
> 11. <a href="http://GIGLINX-INC.car2.LosAngeles1.Level3.net" target="_blank">GIGLINX-INC.car2.LosAngeles1.Level3.net</a> (4.30.195.86) 6.0% 184 173 19.2 22.2 19.1 39.3<br>
> 12. <a href="http://mambo.koitsu.org" target="_blank">mambo.koitsu.org</a> (174.136.108.146) 6.5% 184 172 18.8 20.6 18.5 36.9<br>
><br>
> Note: Loss shown at above hops #5, #8, and #9 are normal according to<br>
> historic data (i.e. ICMP prioritisation is in effect) -- however loss at<br>
> hop #6 is new.<br>
><br>
><br>
> ICMP 1200 bytes:<br>
><br>
> Packets Pings<br>
> Host Loss% Snt Rcv Last Avg Best Wrst<br>
> 1. gw.home.lan (192.168.1.1) 0.0% 43 43 0.3 0.3 0.2 0.4<br>
> 2. 69.181.136.1 0.0% 43 43 9.7 10.4 8.6 14.3<br>
> 3. <a href="http://te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net" target="_blank">te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net</a> (68.86.249.69) 0.0% 42 42 10.0 10.0 8.7 16.0<br>
> 4. <a href="http://te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net" target="_blank">te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net</a> (162.151.78.134 0.0% 42 42 11.7 13.3 10.3 18.9<br>
> 5. ???<br>
> 6. <a href="http://he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net" target="_blank">he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net</a> (68.86.86.102) 0.0% 42 42 14.4 15.9 12.4 25.1<br>
> 7. 50.242.148.34 0.0% 42 42 12.7 13.6 11.2 23.1<br>
> 8. <a href="http://ae-3-80.edge6.LosAngeles1.Level3.net" target="_blank">ae-3-80.edge6.LosAngeles1.Level3.net</a> (4.69.144.142) 9.8% 42 37 20.0 20.7 18.8 25.0<br>
> 9. <a href="http://ae-3-80.edge6.LosAngeles1.Level3.net" target="_blank">ae-3-80.edge6.LosAngeles1.Level3.net</a> (4.69.144.142) 0.0% 42 42 19.1 21.1 18.9 29.9<br>
> 10. <a href="http://ae-27-27.car2.LosAngeles1.Level3.net" target="_blank">ae-27-27.car2.LosAngeles1.Level3.net</a> (4.69.201.18) 0.0% 42 42 64.3 105.7 20.9 412.5<br>
> 11. <a href="http://GIGLINX-INC.car2.LosAngeles1.Level3.net" target="_blank">GIGLINX-INC.car2.LosAngeles1.Level3.net</a> (4.30.195.86) 0.0% 42 42 20.6 22.5 19.9 31.2<br>
> 12. <a href="http://mambo.koitsu.org" target="_blank">mambo.koitsu.org</a> (174.136.108.146) 0.0% 42 42 20.7 22.0 20.1 26.4<br>
><br>
> --<br>
> | Jeremy Chadwick <a href="mailto:jdc@koitsu.org">jdc@koitsu.org</a> |<br>
> | UNIX Systems Administrator <a href="http://jdc.koitsu.org/" target="_blank">http://jdc.koitsu.org/</a> |<br>
> | Making life hard for others since 1977. PGP 4BD6C0CB |<br>
><br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</div></div></blockquote></div><br></div>