[outages] Bay Area packet loss at Comcast (gblx.net, he.net)

Constantine A. Murenin mureninc at gmail.com
Sat Jan 19 15:20:22 EST 2013


On Sat, Jan 19, 2013 at 11:52:04AM -0800, Constantine A. Murenin wrote:
> On Sat, 19 Jan 2013 at 11:04:14 -0800, Dylan N wrote:
> > I get packetloss on te-1-12-0-0-ar01.sfsutro.ca.sfba.comcast.net to my
> > linode @ fmt at weekends and at nights, looks more like just too much
> > load rather than an outage
> 
> In my case, all packet loss seem to be within gblx.net 
> (e.g. no packet loss between Comcast and hetzner.de).
> 
> What I find strange is that the latency is still very good, 
> even when there is a significant packet loss.
> 
> But my ssh connection between Comcast and Linode/Fremont is completely 
> unusable for several seconds at a time during these events.
> 
> 
> The loss seems to have continued throughout last night:
> 
> 2013-01-19T01:50:04-0800
> HOST: li163-###                                          Snt   Rcv Loss%   Best   Avg  Wrst StDev
>   1. router1-fmt.linode.com                             600   600  0.0%    0.5   6.6 268.8  18.5
>   2. 10gigabitethernet2-3.core1.fmt1.he.net             600   600  0.0%    0.5  12.8 1385.  66.3
>   3. 10gigabitethernet1-2.core1.sjc2.he.net             600   600  0.0%    0.7   4.4  28.9   5.1
>   4. Port-channel100.ar3.SJC2.gblx.net                  600   600  0.0%    0.8  28.2 234.0  54.8
>   5. po6-20G.ar4.SJC2.gblx.net                          600   579  3.5%    0.9  32.1 888.1 66.0
>   6. lag8.ar5.SJC2.gblx.net                             600   570  5.0%    0.9   4.4  20.9   5.1
>   7. 208.178.58.2                                       600   572  4.7%    1.8   4.1  14.4   1.3
>   8. pos-2-7-0-0-cr01.sanjose.ca.ibone.comcast.net      600   575  4.2%    4.8   7.7  43.1   2.9
>   9. pos-1-4-0-0-cr01.sacramento.ca.ibone.comcast.net   600   574  4.3%    7.1   9.3  20.2   1.5
>  10. te-7-3-ar02.saltlakecity.ut.utah.comcast.net       600   573  4.5%    7.3  11.9 115.7  17.6
>  11. te-3-1-ur08.sacramento.ca.ccal.comcast.net         600   574  4.3%    7.5   8.3 155.6   6.6
>  12. te-1-0-0-ten05.sacramento.ca.ccal.comcast.net      600   581  3.2%   14.1  14.3  25.1   1.3
>  13. 74-93-180-##-Sacramento.hfc.comcastbusiness.net    600   572  4.7%   14.1  20.5  83.9   8.8
> 2013-01-19T02:00:04-0800
> 
> 2013-01-19T01:50:44-0800
> HOST: hype.####.##########                                      Snt   Rcv Loss%   Best   Avg  Wrst StDev
>   1. logout.lan                                                 60    60  0.0%    2.1   7.1  35.2   6.5
>   2. 192.168.1.1                                                60    60  0.0%    2.5   8.6  41.3   6.7
>   3. 10.1.10.1                                                  60    60  0.0%    3.1  10.3  49.8   8.4
>   4. 96.120.14.165                                              60    60  0.0%   11.3  22.7  75.3  11.7
>   5. te-6-5-ur08.sacramento.ca.ccal.comcast.net                 60    60  0.0%   12.3  22.8  66.1  12.1
>   6. te-0-2-0-2-ar03.sacramento.ca.sacra.comcast.net            60    60  0.0%   11.8  26.3 101.6  18.3
>   7. pos-0-3-0-0-cr01.sacramento.ca.ibone.comcast.net           60    60  0.0%   14.8  26.8  85.3  14.1
>   8. pos-0-3-0-0-cr01.sanjose.ca.ibone.comcast.net              60    60  0.0%   15.6  28.2  71.6  10.3
>   9. pos-0-1-0-0-pe01.11greatoaks.ca.ibone.comcast.net          60    60  0.0%   20.8  30.6  92.6  12.2
>  10. 208.178.58.1                                               60    60  0.0%   18.6  34.8  98.6  16.6
>  11. po4-20G.ar2.SJC2.gblx.net                                  60    60  0.0%   17.5  61.6 273.3  66.9
>  12. po1-20G.ar3.SJC2.gblx.net                                  60    47 21.7%   17.8  48.2 238.7  51.7
>  13. Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net    60    44 26.7%   17.1  30.5  69.8  11.6
>  14. 10gigabitethernet1-1.core1.fmt1.he.net                     60    43 28.3%   21.5  55.2 598.8  87.9
>  15. router1-fmt.linode.com                                     60    44 26.7%   18.3  44.0 187.1  34.4
>  16. li163-###.members.linode.com                               60    43 28.3%   17.8  29.0  68.1   8.6
> 2013-01-19T01:51:55-0800


It's been a 61% packet loss in the last minute...  Ridiculous.

2013-01-19T12:15:15-0800
HOST: hype.####.##########                                      Snt   Rcv Loss%   Best   Avg  Wrst StDev
  1. logout.lan                                                 60    60  0.0%    2.1  18.8 222.7  50.1
  2. 192.168.1.1                                                60    60  0.0%    2.3  20.4 421.0  65.5
  3. 10.1.10.1                                                  60    60  0.0%    3.0  22.9 357.1  57.2
  4. 96.120.14.165                                              60    60  0.0%    9.6  29.3 306.7  54.4
  5. te-6-5-ur08.sacramento.ca.ccal.comcast.net                 60    60  0.0%   10.3  34.2 265.3  51.9
  6. te-0-2-0-2-ar03.sacramento.ca.sacra.comcast.net            60    60  0.0%    9.8  33.1 185.6  43.3
  7. pos-0-4-0-0-cr01.sacramento.ca.ibone.comcast.net           60    60  0.0%   11.8  32.3 234.3  42.8
  8. pos-0-3-0-0-cr01.sanjose.ca.ibone.comcast.net              60    60  0.0%   15.6  31.7 282.1  41.5
  9. pos-0-1-0-0-pe01.11greatoaks.ca.ibone.comcast.net          60    60  0.0%   18.1  36.0 221.3  38.0
 10. 208.178.58.1                                               60    60  0.0%   17.1  40.8 366.8  51.5
 11. po4-20G.ar2.SJC2.gblx.net                                  60    59  1.7%   16.8  58.3 291.6  68.0
 12. po1-20G.ar3.SJC2.gblx.net                                  60    46 23.3%   16.8  67.3 258.6  66.5
 13. Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net    60    48 20.0%   17.3  34.9 155.8  32.0
 14. 10gigabitethernet1-1.core1.fmt1.he.net                     60    19 68.3%   18.3  50.4 117.8  32.2
 15. router1-fmt.linode.com                                     60    22 63.3%   19.1  66.3 287.3  64.2
 16. li163-###.members.linode.com                               60    23 61.7%   18.3  42.5 213.8  48.7
2013-01-19T12:16:27-0800


Anyone from he.net / gblx.net / comcast.net to comment?

Best regards,
Constantine.



More information about the Outages mailing list