[outages] Comcast in Silicon Valley -- Oakland or thereabouts just broke?

John Neiberger jneiberger at gmail.com
Wed Oct 2 20:18:28 EDT 2013


I just checked into this. Looks like some sort of transport issue in sfba.
The appropriate teams have been working on it for a little while. I haven't
heard what the expected time to repair is yet.

John
On Oct 2, 2013 5:57 PM, "Jeremy Chadwick" <jdc at koitsu.org> wrote:

> This just happened -- looks like something in Comcast's network within
> Oakland broke, reconverged (quickly and seamless), but issues remain.
>
> Ren Provo et al, if you're lurking, might want to make someone aware of
> this (if they aren't already).  :-)
>
> src: 76.102.14.35
> dst: 208.79.90.130
>
> === Wed Oct  2 16:14:00 PDT 2013  (1380755640)
> Start: Wed Oct  2 16:14:00 2013
> HOST: icarus.home.lan             Loss%   Snt   Rcv  Last   Avg  Best  Wrst
>   1.|-- 192.168.1.1                0.0%    40    40   0.4   0.3   0.2   0.4
>   2.|-- 76.102.12.1                0.0%    40    40   9.6  10.8   8.1  73.5
>   3.|-- 68.86.249.253              0.0%    40    40   8.9  10.2   8.3  62.2
>   4.|-- 69.139.199.98              0.0%    40    40  10.6  13.1   9.8  47.4
>   5.|-- 68.85.155.14               0.0%    40    40  12.0  16.8  11.4  87.7
>   6.|-- 68.86.90.157               0.0%    40    40  14.6  17.8  12.6  87.5
>   7.|-- 68.86.89.210               0.0%    40    40  23.7  24.7  20.6  92.9
>   8.|-- 68.86.87.210               0.0%    40    40  25.9  25.0  20.7  99.0
>   9.|-- 173.167.57.138             0.0%    40    40  22.1  24.7  21.3  89.0
>  10.|-- 69.31.127.141              0.0%    40    40  20.5  22.0  19.8  67.1
>  11.|-- 69.31.124.114              0.0%    40    40  20.6  21.6  20.0  45.9
>  12.|-- 67.199.135.102             0.0%    40    40  39.9  36.5  21.1 231.0
>  13.|-- 208.79.90.130              0.0%    40    40  21.2  20.7  20.2  21.7
> === END
>
> === Wed Oct  2 16:15:00 PDT 2013  (1380755700)
> Start: Wed Oct  2 16:15:00 2013
> HOST: icarus.home.lan             Loss%   Snt   Rcv  Last   Avg  Best  Wrst
>   1.|-- 192.168.1.1                0.0%    40    40   0.2   0.3   0.2   0.4
>   2.|-- 76.102.12.1                0.0%    40    40   8.1   9.4   8.1  13.6
>   3.|-- 68.86.249.253              0.0%    40    40   8.7   9.7   8.4  14.1
>   4.|-- 68.86.143.98               0.0%    40    40  13.1  13.1  10.2  21.1
>   5.|-- ???                       100.0    40     0   0.0   0.0   0.0   0.0
>   6.|-- ???                       100.0    40     0   0.0   0.0   0.0   0.0
>   7.|-- ???                       100.0    40     0   0.0   0.0   0.0   0.0
>   8.|-- ???                       100.0    40     0   0.0   0.0   0.0   0.0
>   9.|-- 173.167.57.138             0.0%    40    40  33.0  33.9  31.6  37.1
>  10.|-- 69.31.127.141              5.0%    40    38  30.2  31.4  29.7  39.3
>  11.|-- 69.31.124.114              7.5%    40    37  31.8  38.5  29.8 104.0
>  12.|-- 67.199.135.102             7.5%    40    37  31.1  36.6  30.6 119.7
>  13.|-- 208.79.90.130              7.5%    40    37  30.9  31.9  30.4  38.9
> === END
>
> 68.86.249.253 = te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net
> 69.139.199.98 = te-1-1-0-11-ar01.oakland.ca.sfba.comcast.net
> 68.85.155.14  = be-90-ar01.sfsutro.ca.sfba.comcast.net
> 68.86.143.98  = te-1-1-0-5-ar01.oakland.ca.sfba.comcast.net
>
> And packet loss has gradually been getting worse (impacting all forms of
> packets, not just TCP).
>
> === Wed Oct  2 16:45:00 PDT 2013  (1380757500)
> Start: Wed Oct  2 16:45:00 2013
> HOST: icarus.home.lan             Loss%   Snt   Rcv  Last   Avg  Best  Wrst
>   1.|-- 192.168.1.1                0.0%    40    40   0.2   0.3   0.2   0.4
>   2.|-- 76.102.12.1                0.0%    40    40   8.6   9.9   8.1  14.3
>   3.|-- 68.86.249.253              0.0%    40    40   9.2  10.2   8.4  16.7
>   4.|-- 68.86.143.98               0.0%    40    40  13.9  13.1  10.6  16.6
>   5.|-- ???                       100.0    40     0   0.0   0.0   0.0   0.0
>   6.|-- ???                       100.0    40     0   0.0   0.0   0.0   0.0
>   7.|-- ???                       100.0    40     0   0.0   0.0   0.0   0.0
>   8.|-- ???                       100.0    40     0   0.0   0.0   0.0   0.0
>   9.|-- 173.167.57.138            15.0%    40    34  34.6  34.8  31.7  41.7
>  10.|-- 69.31.127.141             25.0%    40    30  31.3  32.6  30.0  41.7
>  11.|-- 69.31.124.114              5.0%    40    38  31.8  33.5  30.5  58.6
>  12.|-- 67.199.135.102            12.8%    39    34  30.9  37.1  30.9  77.3
>  13.|-- 208.79.90.130              7.7%    39    36  32.9  32.3  30.8  37.3
> === END
>
> And here's what I see on the return path, for a brief run:
>
> src: 208.79.90.130
> dst: 76.102.14.35
>
>  Host                                           Loss%   Snt   Rcv  Last
> Avg  Best  Wrst
>  1. 208.79.90.129                                0.0%    93    93   1.5
> 6.1   0.9 163.0
>  2. s7.lax.arpnetworks.com                       0.0%    93    93   0.7
>  11.2   0.5 150.5
>  3. ge0-15.as01.lax07.mzima.net                  0.0%    93 93 4.1 2.4 0.4 64.6
>  4. ae3-50g.cr1.lax1.us.nlayer.net               0.0%    93 93 0.6 2.2 0.4 56.0
>  5. ae1-50g.ar1.lax1.us.nlayer.net               1.1%    93 92 3.1 3.3 1.3  6.4
>  6. be-102-pe01.600wseventh.ca.ibone.comcast.ne  0.0%    93 93 3.8 3.1 1.1  5.4
>  7. pos-0-8-0-0-cr01.losangeles.ca.ibone.comcas  0.0%    93    93   3.0
> 2.9   1.5   6.1
>  8. he-2-15-0-0-cr01.sanjose.ca.ibone.comcast.n  0.0%    92    92   8.6
>  10.3   8.1  12.5
>  9. he-0-9-0-0-ar01.sfsutro.ca.sfba.comcast.net  0.0%    92    92  12.5
>  11.6   9.8  13.6
> 10. be-100-ar01.oakland.ca.sfba.comcast.net      3.3%    92    88  22.0
>  25.2  21.9  49.6
> 11. te-0-4-0-2-ur06.santaclara.ca.sfba.comcast.  3.3%    92    89  23.8
>  23.8  23.0  25.4
> 12. te-6-0-acr03.santaclara.ca.sfba.comcast.net  9.8%    92    83  23.8
>  24.4  22.6  46.3
> 13. c-76-102-14-35.hsd1.ca.comcast.net           6.5%    92    86  36.3
>  31.7  30.5  38.3
>
> --
> | Jeremy Chadwick                                   jdc at koitsu.org |
> | UNIX Systems Administrator                http://jdc.koitsu.org/ |
> | Making life hard for others since 1977.             PGP 4BD6C0CB |
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20131002/6db72942/attachment.htm>


More information about the Outages mailing list