[outages] Comcast in Silicon Valley -- Oakland or thereabouts just broke?
Jeremy Chadwick
jdc at koitsu.org
Wed Oct 2 20:05:33 EDT 2013
Acknowledged.
I'm in Mountain View but you can clearly see where the reconvergence
takes place in the mtrs (between Oakland and SF Sutro, at least from the
forward path). Naturally I have no way of knowing where the actual
issue is.
I'm sure you know by now most of our (as a customer) packets get sent
either down to San Jose or out of Oakland (and I believe the Oakland
network is something that got rolled out somewhat recently; I remember
seeing it suddenly appear in my mtrs one night not that long ago).
I sent a tweet to @ComcastCares pointing them to this thread.
I also imagine the issue will get worse since it's now 1700 and people
are just starting to get off work...
--
| Jeremy Chadwick jdc at koitsu.org |
| UNIX Systems Administrator http://jdc.koitsu.org/ |
| Making life hard for others since 1977. PGP 4BD6C0CB |
On Wed, Oct 02, 2013 at 04:58:57PM -0700, Scott Howard wrote:
> Not just Oakland - I'm seeing issues in both the South Bay as well as on
> the Peninsular so I'm guessing it's pretty much the whole SF Bay Area.
>
> Same symptoms as you're seeing - packet loss that seems to be within their
> network, and as a result fairly ordinary speeds.
>
> Scott
>
>
>
>
> On Wed, Oct 2, 2013 at 4:54 PM, Jeremy Chadwick <jdc at koitsu.org> wrote:
>
> > This just happened -- looks like something in Comcast's network within
> > Oakland broke, reconverged (quickly and seamless), but issues remain.
> >
> > Ren Provo et al, if you're lurking, might want to make someone aware of
> > this (if they aren't already). :-)
> >
> > src: 76.102.14.35
> > dst: 208.79.90.130
> >
> > === Wed Oct 2 16:14:00 PDT 2013 (1380755640)
> > Start: Wed Oct 2 16:14:00 2013
> > HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst
> > 1.|-- 192.168.1.1 0.0% 40 40 0.4 0.3 0.2 0.4
> > 2.|-- 76.102.12.1 0.0% 40 40 9.6 10.8 8.1 73.5
> > 3.|-- 68.86.249.253 0.0% 40 40 8.9 10.2 8.3 62.2
> > 4.|-- 69.139.199.98 0.0% 40 40 10.6 13.1 9.8 47.4
> > 5.|-- 68.85.155.14 0.0% 40 40 12.0 16.8 11.4 87.7
> > 6.|-- 68.86.90.157 0.0% 40 40 14.6 17.8 12.6 87.5
> > 7.|-- 68.86.89.210 0.0% 40 40 23.7 24.7 20.6 92.9
> > 8.|-- 68.86.87.210 0.0% 40 40 25.9 25.0 20.7 99.0
> > 9.|-- 173.167.57.138 0.0% 40 40 22.1 24.7 21.3 89.0
> > 10.|-- 69.31.127.141 0.0% 40 40 20.5 22.0 19.8 67.1
> > 11.|-- 69.31.124.114 0.0% 40 40 20.6 21.6 20.0 45.9
> > 12.|-- 67.199.135.102 0.0% 40 40 39.9 36.5 21.1 231.0
> > 13.|-- 208.79.90.130 0.0% 40 40 21.2 20.7 20.2 21.7
> > === END
> >
> > === Wed Oct 2 16:15:00 PDT 2013 (1380755700)
> > Start: Wed Oct 2 16:15:00 2013
> > HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst
> > 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4
> > 2.|-- 76.102.12.1 0.0% 40 40 8.1 9.4 8.1 13.6
> > 3.|-- 68.86.249.253 0.0% 40 40 8.7 9.7 8.4 14.1
> > 4.|-- 68.86.143.98 0.0% 40 40 13.1 13.1 10.2 21.1
> > 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
> > 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
> > 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
> > 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
> > 9.|-- 173.167.57.138 0.0% 40 40 33.0 33.9 31.6 37.1
> > 10.|-- 69.31.127.141 5.0% 40 38 30.2 31.4 29.7 39.3
> > 11.|-- 69.31.124.114 7.5% 40 37 31.8 38.5 29.8 104.0
> > 12.|-- 67.199.135.102 7.5% 40 37 31.1 36.6 30.6 119.7
> > 13.|-- 208.79.90.130 7.5% 40 37 30.9 31.9 30.4 38.9
> > === END
> >
> > 68.86.249.253 = te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net
> > 69.139.199.98 = te-1-1-0-11-ar01.oakland.ca.sfba.comcast.net
> > 68.85.155.14 = be-90-ar01.sfsutro.ca.sfba.comcast.net
> > 68.86.143.98 = te-1-1-0-5-ar01.oakland.ca.sfba.comcast.net
> >
> > And packet loss has gradually been getting worse (impacting all forms of
> > packets, not just TCP).
> >
> > === Wed Oct 2 16:45:00 PDT 2013 (1380757500)
> > Start: Wed Oct 2 16:45:00 2013
> > HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst
> > 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4
> > 2.|-- 76.102.12.1 0.0% 40 40 8.6 9.9 8.1 14.3
> > 3.|-- 68.86.249.253 0.0% 40 40 9.2 10.2 8.4 16.7
> > 4.|-- 68.86.143.98 0.0% 40 40 13.9 13.1 10.6 16.6
> > 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
> > 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
> > 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
> > 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
> > 9.|-- 173.167.57.138 15.0% 40 34 34.6 34.8 31.7 41.7
> > 10.|-- 69.31.127.141 25.0% 40 30 31.3 32.6 30.0 41.7
> > 11.|-- 69.31.124.114 5.0% 40 38 31.8 33.5 30.5 58.6
> > 12.|-- 67.199.135.102 12.8% 39 34 30.9 37.1 30.9 77.3
> > 13.|-- 208.79.90.130 7.7% 39 36 32.9 32.3 30.8 37.3
> > === END
> >
> > And here's what I see on the return path, for a brief run:
> >
> > src: 208.79.90.130
> > dst: 76.102.14.35
> >
> > Host Loss% Snt Rcv Last
> > Avg Best Wrst
> > 1. 208.79.90.129 0.0% 93 93 1.5
> > 6.1 0.9 163.0
> > 2. s7.lax.arpnetworks.com 0.0% 93 93 0.7
> > 11.2 0.5 150.5
> > 3. ge0-15.as01.lax07.mzima.net 0.0% 93 93 4.1 2.4 0.4 64.6
> > 4. ae3-50g.cr1.lax1.us.nlayer.net 0.0% 93 93 0.6 2.2 0.4 56.0
> > 5. ae1-50g.ar1.lax1.us.nlayer.net 1.1% 93 92 3.1 3.3 1.3 6.4
> > 6. be-102-pe01.600wseventh.ca.ibone.comcast.ne 0.0% 93 93 3.8 3.1 1.1 5.4
> > 7. pos-0-8-0-0-cr01.losangeles.ca.ibone.comcas 0.0% 93 93 3.0
> > 2.9 1.5 6.1
> > 8. he-2-15-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 92 92 8.6
> > 10.3 8.1 12.5
> > 9. he-0-9-0-0-ar01.sfsutro.ca.sfba.comcast.net 0.0% 92 92 12.5
> > 11.6 9.8 13.6
> > 10. be-100-ar01.oakland.ca.sfba.comcast.net 3.3% 92 88 22.0
> > 25.2 21.9 49.6
> > 11. te-0-4-0-2-ur06.santaclara.ca.sfba.comcast. 3.3% 92 89 23.8
> > 23.8 23.0 25.4
> > 12. te-6-0-acr03.santaclara.ca.sfba.comcast.net 9.8% 92 83 23.8
> > 24.4 22.6 46.3
> > 13. c-76-102-14-35.hsd1.ca.comcast.net 6.5% 92 86 36.3
> > 31.7 30.5 38.3
> >
> > --
> > | Jeremy Chadwick jdc at koitsu.org |
> > | UNIX Systems Administrator http://jdc.koitsu.org/ |
> > | Making life hard for others since 1977. PGP 4BD6C0CB |
> >
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> >
More information about the Outages
mailing list