[outages] Comcast LA issue?
John Neiberger
jneiberger at gmail.com
Thu Feb 19 19:47:11 EST 2015
Yep. A 100G link had lost its mind. Sounds like all is well now. Thanks for
the info!
John
On Thu, Feb 19, 2015 at 5:41 PM, Jeremy Chadwick <jdc at koitsu.org> wrote:
> Looks resolved as of about 16:23 PT (UTC-8) -- everything appears
> reliable now (all my mail just came flooding in ;-) ).
>
> Looks like there was a link flap or some reconvergence on the forward
> path that happened at about 16:21, followed by recovery. (I saw nothing
> on the return path). Look about right?
>
> === Thu Feb 19 16:21:00 PST 2015 (1424391660)
> Start: Thu Feb 19 16:21:00 2015
> HOST: icarus.home.lan
> Loss% Snt Rcv Last Avg Best Wrst
> 1.|-- gw.home.lan (192.168.1.1)
> 0.0% 30 30 0.2 0.2 0.2 0.3
> 2.|-- 69.181.136.1
> 0.0% 30 30 9.5 8.8 7.8 15.4
> 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)
> 0.0% 30 30 9.4 8.9 8.2 15.7
> 4.|-- te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134)
> 0.0% 30 30 11.9 11.8 9.5 13.8
> 5.|-- ???
> 100.0 30 0 0.0 0.0 0.0 0.0
> 6.|-- he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102)
> 16.7% 30 25 12.5 13.2 11.2 16.2
> 7.|-- 50.242.148.34
> 23.3% 30 23 11.6 13.8 10.9 47.6
> 8.|-- ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)
> 16.7% 30 25 18.7 19.0 18.3 21.7
> 9.|-- ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)
> 50.0% 30 15 18.6 19.7 18.2 26.2
> 10.|-- ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18)
> 16.7% 30 25 370.2 87.7 25.9 370.2
> 11.|-- GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86)
> 10.0% 30 27 133.6 26.8 19.1 133.6
> 12.|-- mambo.koitsu.org (174.136.108.146)
> 13.3% 30 26 19.1 19.4 18.4 26.9
> === END
>
> === Thu Feb 19 16:22:00 PST 2015 (1424391720)
> Start: Thu Feb 19 16:22:00 2015
> HOST: icarus.home.lan
> Loss% Snt Rcv Last Avg Best Wrst
> 1.|-- gw.home.lan (192.168.1.1)
> 0.0% 30 30 0.2 0.2 0.2 0.3
> 2.|-- 69.181.136.1
> 0.0% 30 30 9.1 9.0 8.0 13.6
> 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)
> 0.0% 30 30 9.0 9.2 8.1 15.9
> 4.|-- te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134)
> 0.0% 30 30 11.7 12.6 9.8 21.6
> 5.|-- ???
> 100.0 30 0 0.0 0.0 0.0 0.0
> === END
>
> === Thu Feb 19 16:23:00 PST 2015 (1424391780)
> Start: Thu Feb 19 16:23:00 2015
> HOST: icarus.home.lan
> Loss% Snt Rcv Last Avg Best Wrst
> 1.|-- gw.home.lan (192.168.1.1)
> 0.0% 30 30 0.2 0.2 0.2 0.3
> 2.|-- 69.181.136.1
> 0.0% 30 30 9.0 9.0 7.7 15.0
> 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)
> 0.0% 30 30 9.2 8.9 8.2 10.1
> 4.|-- te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134)
> 0.0% 30 30 12.5 12.4 10.0 19.2
> 5.|-- ???
> 100.0 30 0 0.0 0.0 0.0 0.0
> 6.|-- he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102)
> 0.0% 30 30 14.6 14.0 11.1 23.2
> 7.|-- 50.242.148.34
> 0.0% 30 30 12.2 11.7 10.7 19.0
> 8.|-- ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)
> 16.7% 30 25 19.5 24.6 18.2 53.5
> 9.|-- ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)
> 46.7% 30 16 19.2 18.8 18.2 19.9
> 10.|-- ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18)
> 0.0% 30 30 76.1 53.7 20.2 109.3
> 11.|-- GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86)
> 0.0% 30 30 22.2 21.4 18.9 32.7
> 12.|-- mambo.koitsu.org (174.136.108.146)
> 0.0% 30 30 18.6 19.3 18.5 21.5
> === END
>
> --
> | Jeremy Chadwick jdc at koitsu.org |
> | UNIX Systems Administrator http://jdc.koitsu.org/ |
> | Making life hard for others since 1977. PGP 4BD6C0CB |
>
> On Thu, Feb 19, 2015 at 05:24:08PM -0700, John Neiberger wrote:
> > Those of you having issues, can you check to see if they're still
> occurring?
> >
> > Thanks,
> > John
> >
> > On Thu, Feb 19, 2015 at 4:25 PM, Jeremy Chadwick via Outages <
> > outages at outages.org> wrote:
> >
> > > Here's another destination showing the same problem (I can't even load
> > > http://www.dslreports.com/ at present -- handshake works, but either
> > > client->server GET request gets lost, or payload response gets lost,
> > > followed by TCP retrans, eventually RST from the server, rinse lather
> > > repeat), with things again looking centralised around LA.
> > >
> > > Hopefully having multiple test endpoints will come in handy.
> > >
> > > Source IP: 69.181.142.213
> > > Destination IP: 64.91.255.98
> > >
> > >
> > > Packets Pings
> > > Host
> Loss%
> > > Snt Rcv Last Avg Best Wrst
> > > 1. gw.home.lan (192.168.1.1)
> 0.0%
> > > 171 171 0.2 0.2 0.2 0.4
> > > 2. 69.181.136.1
> 0.0%
> > > 171 171 12.7 9.6 7.7 19.2
> > > 3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)
> > > 0.0% 171 171 9.4 9.2 8.0 17.1
> > > 4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134
> > > 0.0% 171 171 12.1 12.2 9.5 18.4
> > > 5. he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.90.93)
> > > 98.8% 171 2 50456 50095 49735 50456
> > > 6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102)
> > > 2.9% 171 166 14.1 13.5 10.9 22.7
> > > 7. he-2-12-0-0-cr01.seattle.wa.ibone.comcast.net (68.86.86.229)
> > > 1.8% 171 168 29.2 30.2 27.4 36.3
> > > 8. he-0-15-0-0-cr01.denver.co.ibone.comcast.net (68.86.86.213)
> > > 9.9% 171 154 50.1 47.6 44.9 53.1
> > > 9. he-5-13-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.85.245)
> > > 4.1% 171 163 72.9 72.3 68.6 84.7
> > > 10. hu-0-10-0-1.pe03.350ecermak.il.ibone.comcast.net (68.86.86.6)
> > > 5.3% 170 161 69.8 69.4 68.1 84.7
> > > 11. 50.242.150.126
> 7.1%
> > > 170 158 106.0 89.7 67.4 400.5
> > > 50.242.150.130
> > > 12. lw-dc3-core1-te8-16.rtr.liquidweb.com (209.59.157.244)
> > > 9.4% 170 154 75.7 80.9 75.2 242.7
> > > 13. lw-dc3-dist15-po5.rtr.liquidweb.com (69.167.128.241)
> > > 2.4% 170 166 75.0 75.9 74.7 83.6
> > > 14. www.dslreports.com (64.91.255.98)
> > > 5.3% 170 161 74.2 74.3 73.2 82.5
> > >
> > > --
> > > | Jeremy Chadwick jdc at koitsu.org |
> > > | UNIX Systems Administrator http://jdc.koitsu.org/ |
> > > | Making life hard for others since 1977. PGP 4BD6C0CB |
> > >
> > > On Thu, Feb 19, 2015 at 03:07:04PM -0800, Jeremy Chadwick wrote:
> > > > Something weird is going on with Comcast in Los Angeles: small
> packets
> > > > appear to be getting discarded with a loss rate of ~6-10% while
> larger
> > > > packets (I've tried 1000 and 1200) seem to make it through without
> any
> > > > loss. Things like SSH and IMAP+SSL are impacted consistently
> > > > (particularly during key exchange), while HTTP does not appear to
> be, so
> > > > the issue appears to be affecting both TCP and ICMP (have not tested
> > > > UDP).
> > > >
> > > > First sign of this issue, according to my logs, is at 11:51 PT
> (UTC-8),
> > > > and has gradually gotten worse over the past few hours.
> > > >
> > > > Looks like the issue is with the forward path (Source->Destination
> > > > listed below), as the return path only shows loss at the final
> > > > destination. (I currently cannot provide a return-path mtr because
> the
> > > > only way I can access my box is through VNC, and I don't like doing
> > > > things like screenshots of a VNC window, so you'll just have to trust
> > > > me)
> > > >
> > > > Nothing anomalous shows up when using L3's looking glass to ping
> things,
> > > > and the only Comcast LG I know of is in NYC.
> > > >
> > > > I most likely won't see Email responses from anyone (directly or to
> the
> > > > list) until this problem is addressed, since my mail is hosted on
> shown
> > > > box and cannot currently be fetched due to the problem. Frustrating.
> > > >
> > > > Source IP: 69.181.142.213
> > > > Destination IP: 174.136.108.146
> > > >
> > > > ICMP 56 bytes:
> > > >
> > > >
> > > Packets Pings
> > > > Host
> > > Loss% Snt Rcv Last Avg Best Wrst
> > > > 1. gw.home.lan (192.168.1.1)
> > > 0.0% 185 185 0.2 0.2 0.2 0.3
> > > > 2. 69.181.136.1
> > > 0.0% 185 185 8.9 9.9 7.0 21.1
> > > > 3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)
> > > 0.0% 185 185 8.7 9.9 7.9 17.0
> > > > 4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134
> > > 0.0% 185 185 10.7 12.9 9.7 20.7
> > > > 5. he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.90.93)
> > > 98.4% 184 3 49496 49006 47906 49615
> > > > 6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102)
> > > 5.4% 184 174 11.9 14.1 10.9 23.5
> > > > 7. 50.242.148.34
> > > 6.5% 184 172 11.7 13.4 10.4 69.9
> > > > 8. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)
> > > 66.7% 184 61 44.9 20.6 18.0 44.9
> > > > 9. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)
> > > 59.0% 184 75 18.7 19.8 18.0 26.0
> > > > 10. ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18)
> > > 2.2% 184 180 20.8 63.1 18.2 293.3
> > > > 11. GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86)
> > > 6.0% 184 173 19.2 22.2 19.1 39.3
> > > > 12. mambo.koitsu.org (174.136.108.146)
> > > 6.5% 184 172 18.8 20.6 18.5 36.9
> > > >
> > > > Note: Loss shown at above hops #5, #8, and #9 are normal according to
> > > > historic data (i.e. ICMP prioritisation is in effect) -- however
> loss at
> > > > hop #6 is new.
> > > >
> > > >
> > > > ICMP 1200 bytes:
> > > >
> > > >
> > > Packets Pings
> > > > Host
> > > Loss% Snt Rcv Last Avg Best Wrst
> > > > 1. gw.home.lan (192.168.1.1)
> > > 0.0% 43 43 0.3 0.3 0.2 0.4
> > > > 2. 69.181.136.1
> > > 0.0% 43 43 9.7 10.4 8.6 14.3
> > > > 3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)
> > > 0.0% 42 42 10.0 10.0 8.7 16.0
> > > > 4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134
> > > 0.0% 42 42 11.7 13.3 10.3 18.9
> > > > 5. ???
> > > > 6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102)
> > > 0.0% 42 42 14.4 15.9 12.4 25.1
> > > > 7. 50.242.148.34
> > > 0.0% 42 42 12.7 13.6 11.2 23.1
> > > > 8. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)
> > > 9.8% 42 37 20.0 20.7 18.8 25.0
> > > > 9. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)
> > > 0.0% 42 42 19.1 21.1 18.9 29.9
> > > > 10. ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18)
> > > 0.0% 42 42 64.3 105.7 20.9 412.5
> > > > 11. GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86)
> > > 0.0% 42 42 20.6 22.5 19.9 31.2
> > > > 12. mambo.koitsu.org (174.136.108.146)
> > > 0.0% 42 42 20.7 22.0 20.1 26.4
> > > >
> > > > --
> > > > | Jeremy Chadwick jdc at koitsu.org |
> > > > | UNIX Systems Administrator http://jdc.koitsu.org/ |
> > > > | Making life hard for others since 1977. PGP 4BD6C0CB |
> > > >
> > > _______________________________________________
> > > Outages mailing list
> > > Outages at outages.org
> > > https://puck.nether.net/mailman/listinfo/outages
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20150219/021900bc/attachment.htm>
More information about the Outages
mailing list