[outages] Comcast in Silicon Valley -- Oakland or thereabouts just broke?
Jeremy Chadwick
jdc at koitsu.org
Wed Oct 2 19:54:34 EDT 2013
This just happened -- looks like something in Comcast's network within
Oakland broke, reconverged (quickly and seamless), but issues remain.
Ren Provo et al, if you're lurking, might want to make someone aware of
this (if they aren't already). :-)
src: 76.102.14.35
dst: 208.79.90.130
=== Wed Oct 2 16:14:00 PDT 2013 (1380755640)
Start: Wed Oct 2 16:14:00 2013
HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst
1.|-- 192.168.1.1 0.0% 40 40 0.4 0.3 0.2 0.4
2.|-- 76.102.12.1 0.0% 40 40 9.6 10.8 8.1 73.5
3.|-- 68.86.249.253 0.0% 40 40 8.9 10.2 8.3 62.2
4.|-- 69.139.199.98 0.0% 40 40 10.6 13.1 9.8 47.4
5.|-- 68.85.155.14 0.0% 40 40 12.0 16.8 11.4 87.7
6.|-- 68.86.90.157 0.0% 40 40 14.6 17.8 12.6 87.5
7.|-- 68.86.89.210 0.0% 40 40 23.7 24.7 20.6 92.9
8.|-- 68.86.87.210 0.0% 40 40 25.9 25.0 20.7 99.0
9.|-- 173.167.57.138 0.0% 40 40 22.1 24.7 21.3 89.0
10.|-- 69.31.127.141 0.0% 40 40 20.5 22.0 19.8 67.1
11.|-- 69.31.124.114 0.0% 40 40 20.6 21.6 20.0 45.9
12.|-- 67.199.135.102 0.0% 40 40 39.9 36.5 21.1 231.0
13.|-- 208.79.90.130 0.0% 40 40 21.2 20.7 20.2 21.7
=== END
=== Wed Oct 2 16:15:00 PDT 2013 (1380755700)
Start: Wed Oct 2 16:15:00 2013
HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst
1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4
2.|-- 76.102.12.1 0.0% 40 40 8.1 9.4 8.1 13.6
3.|-- 68.86.249.253 0.0% 40 40 8.7 9.7 8.4 14.1
4.|-- 68.86.143.98 0.0% 40 40 13.1 13.1 10.2 21.1
5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
9.|-- 173.167.57.138 0.0% 40 40 33.0 33.9 31.6 37.1
10.|-- 69.31.127.141 5.0% 40 38 30.2 31.4 29.7 39.3
11.|-- 69.31.124.114 7.5% 40 37 31.8 38.5 29.8 104.0
12.|-- 67.199.135.102 7.5% 40 37 31.1 36.6 30.6 119.7
13.|-- 208.79.90.130 7.5% 40 37 30.9 31.9 30.4 38.9
=== END
68.86.249.253 = te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net
69.139.199.98 = te-1-1-0-11-ar01.oakland.ca.sfba.comcast.net
68.85.155.14 = be-90-ar01.sfsutro.ca.sfba.comcast.net
68.86.143.98 = te-1-1-0-5-ar01.oakland.ca.sfba.comcast.net
And packet loss has gradually been getting worse (impacting all forms of
packets, not just TCP).
=== Wed Oct 2 16:45:00 PDT 2013 (1380757500)
Start: Wed Oct 2 16:45:00 2013
HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst
1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4
2.|-- 76.102.12.1 0.0% 40 40 8.6 9.9 8.1 14.3
3.|-- 68.86.249.253 0.0% 40 40 9.2 10.2 8.4 16.7
4.|-- 68.86.143.98 0.0% 40 40 13.9 13.1 10.6 16.6
5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0
9.|-- 173.167.57.138 15.0% 40 34 34.6 34.8 31.7 41.7
10.|-- 69.31.127.141 25.0% 40 30 31.3 32.6 30.0 41.7
11.|-- 69.31.124.114 5.0% 40 38 31.8 33.5 30.5 58.6
12.|-- 67.199.135.102 12.8% 39 34 30.9 37.1 30.9 77.3
13.|-- 208.79.90.130 7.7% 39 36 32.9 32.3 30.8 37.3
=== END
And here's what I see on the return path, for a brief run:
src: 208.79.90.130
dst: 76.102.14.35
Host Loss% Snt Rcv Last Avg Best Wrst
1. 208.79.90.129 0.0% 93 93 1.5 6.1 0.9 163.0
2. s7.lax.arpnetworks.com 0.0% 93 93 0.7 11.2 0.5 150.5
3. ge0-15.as01.lax07.mzima.net 0.0% 93 93 4.1 2.4 0.4 64.6
4. ae3-50g.cr1.lax1.us.nlayer.net 0.0% 93 93 0.6 2.2 0.4 56.0
5. ae1-50g.ar1.lax1.us.nlayer.net 1.1% 93 92 3.1 3.3 1.3 6.4
6. be-102-pe01.600wseventh.ca.ibone.comcast.ne 0.0% 93 93 3.8 3.1 1.1 5.4
7. pos-0-8-0-0-cr01.losangeles.ca.ibone.comcas 0.0% 93 93 3.0 2.9 1.5 6.1
8. he-2-15-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 92 92 8.6 10.3 8.1 12.5
9. he-0-9-0-0-ar01.sfsutro.ca.sfba.comcast.net 0.0% 92 92 12.5 11.6 9.8 13.6
10. be-100-ar01.oakland.ca.sfba.comcast.net 3.3% 92 88 22.0 25.2 21.9 49.6
11. te-0-4-0-2-ur06.santaclara.ca.sfba.comcast. 3.3% 92 89 23.8 23.8 23.0 25.4
12. te-6-0-acr03.santaclara.ca.sfba.comcast.net 9.8% 92 83 23.8 24.4 22.6 46.3
13. c-76-102-14-35.hsd1.ca.comcast.net 6.5% 92 86 36.3 31.7 30.5 38.3
--
| Jeremy Chadwick jdc at koitsu.org |
| UNIX Systems Administrator http://jdc.koitsu.org/ |
| Making life hard for others since 1977. PGP 4BD6C0CB |
More information about the Outages
mailing list