[outages] cocmast packetloss and high latency

Grant McChesney grantmc at gmail.com
Mon Nov 6 14:14:32 EST 2017


On Mon, Nov 6, 2017 at 11:54 AM, John Neiberger via Outages <
outages at outages.org> wrote:

> Thanks! Every trace I've seen has indicated a problem in Level 3. That
> would certainly explain why this is so widespread.
>
> On Mon, Nov 6, 2017 at 11:52 AM, Dovid Bender <dovid at telecurve.com> wrote:
>
>> From NTT in NYC:
>>  3. 165.254.26.201                                        0.0%  1777    0
>> <1777    0>.8   0.8   0.7  10.0   0.3
>>  4. ae-3.r25.nycmny01.us.bb.gin.ntt.net                   0.0%  1777    0
>> <1777    0>.7   0.7   0.6 149.4   3.9
>>  5. ae-9.r22.asbnva02.us.bb.gin.ntt.net                  18.5%  1777    6
>> <1777    6>.3   6.4   6.1  24.9   1.8
>>  6. ae-1.r05.asbnva02.us.bb.gin.ntt.net                   0.0%  1777    5
>> <1777    5>.9   5.9   5.8  15.6   0.5
>>  7. ae-0.a02.asbnva02.us.bb.gin.ntt.net                  19.8%  1777    6
>> <1777    6>.5   7.7   6.0  57.0   5.1
>>  8. lag-12.ear2.Washington12.Level3.net                  98.6%  1777
>> 7208 <1777  7208>. 7377. 6927. 8791. 368.8
>>  9. ae-2-3.bear1.Boston1.Level3.net                      98.1%  1777  131
>> <1777  131>.9 1047. 120.0 6493. 1842.
>> 10. 4.68.71.218                                          45.2%  1777  129
>> <1777  129>.0 133.7 119.6 143.6   4.4
>> 11. be-2-ar01.chartford.ct.hartford.comcast.net          45.9%  1777  139
>> <1777  139>.2 136.3 122.4 146.4   4.5
>> 12. be-31-sur01.waterbury.ct.hartford.comcast.net        43.7%  1777  143
>> <1777  143>.0 137.4 121.7 147.8   4.3
>> 13. 68.87.158.6                                          44.7%  1777  139
>> <1777  139>.5 137.1 115.0 148.1   4.5
>> 14. c-73-218-149-14.hsd1.ma.comcast.net                  45.3%  1776  139
>> <1776  139>.6 146.0 130.5 177.1   5.8
>> 15. ???
>>
>>
>> On Mon, Nov 6, 2017 at 1:51 PM, Dovid Bender <dovid at telecurve.com> wrote:
>>
>>> From DO (NYC):
>>> Keys:  Help   Display mode   Restart statistics   Order of fields   quit
>>>                                                             Packets
>>>          Pings
>>>  Host                                                     Loss%   Snt
>>>  Last   Avg  Best  Wrst StDev
>>>  1. 104.131.255.253                                        0.0%    12
>>> 1.1   1.3   0.3   4.1   1.5
>>>  2. 138.197.248.160                                        0.0%    11
>>>  13.0   1.6   0.3  13.0   3.8
>>>  3. ix-ae-3-0.tcore1.N75-New-York.as6453.net               0.0%    11
>>>   0.4   0.4   0.3   0.8   0.1
>>>  4. if-ae-12-2.tcore2.NTO-New-York.as6453.net              0.0%    11
>>>   8.5   9.8   6.2  40.8  10.3
>>>  5. if-ae-30-2.tcore1.AEQ-Ashburn.as6453.net               0.0%    11
>>>   6.2   8.8   5.9  24.4   5.7
>>>  6. ???
>>>  7. ???
>>>  8. 4.68.71.218                                           63.6%    11
>>>  27.7  28.4  27.5  30.7   1.6
>>>  9. be-2-ar01.chartford.ct.hartford.comcast.net           40.0%    11
>>>  31.7  32.2  31.7  32.9   0.5
>>> 10. be-31-sur01.waterbury.ct.hartford.comcast.net         40.0%    11
>>>  33.6  32.7  32.2  33.6   0.6
>>> 11. 68.87.158.14                                          50.0%    11
>>>  33.5  32.5  32.1  33.5   0.6
>>> 12. c-73-218-149-14.hsd1.ct.comcast.net                   70.0%    11
>>>  39.1  38.7  38.4  39.1   0.3
>>> 13. ???
>>>
>>> From rackspace (Texas)
>>> dev (0.0.0.0)
>>>   Mon Nov  6 18:50:13 2017
>>> Keys:  Help   Display mode   Restart statistics   Order of fields   quit
>>>                                                            Packets
>>>          Pings
>>>  Host                                                    Loss%   Snt
>>>  Last   Avg  Best  Wrst StDev
>>>  1. 50.56.237.2                                           0.0%    11
>>> 1.0   1.1   0.9   1.8   0.3
>>>  2. aggr170a-8-core10.dfw3.rackspace.net                  0.0%    11
>>> 0.9   1.1   0.9   1.5   0.2
>>>  3. core10-coreb.coreb.dfw3.rackspace.net                 0.0%    11
>>> 3.4   3.5   3.3   3.8   0.1
>>>  4. be32.dcpe2.dfw1.rackspace.net                         0.0%    11
>>> 3.6   3.8   3.4   4.4   0.3
>>>  5. be2-mspe4.dfw30.rackspace.net                         0.0%    11
>>> 5.5   5.7   5.5   6.6   0.3
>>>  6. ae-22.a01.dllstx04.us.bb.gin.ntt.net                  0.0%    11
>>> 8.5   6.3   4.8   9.1   1.7
>>>  7. ae-9.r11.dllstx09.us.bb.gin.ntt.net                   0.0%    11
>>>  52.3  43.8  42.8  52.3   2.8
>>>  8. ae-0.r22.dllstx09.us.bb.gin.ntt.net                   0.0%    11
>>>  14.0   5.8   4.4  14.0   2.9
>>>  9. ae-1.r22.asbnva02.us.bb.gin.ntt.net                  45.5%    11
>>>  43.0  43.8  42.2  49.9   3.0
>>> 10. ae-1.r05.asbnva02.us.bb.gin.ntt.net                   0.0%    11
>>>  42.9  43.2  42.9  44.9   0.6
>>> 11. ae-0.a02.asbnva02.us.bb.gin.ntt.net                  50.0%    11
>>>  42.6  42.6  42.3  43.0   0.3
>>> 12. ???
>>> 13. ae-2-3.bear1.Boston1.Level3.net                      90.0%    11
>>> 170.9 170.9 170.9 170.9   0.0
>>> 14. 4.68.71.218                                          20.0%    11
>>> 169.2 175.8 169.2 198.9   9.6
>>> 15. be-2-ar01.chartford.ct.hartford.comcast.net          60.0%    11
>>> 175.0 179.3 167.8 190.8  10.0
>>> 16. be-31-sur01.waterbury.ct.hartford.comcast.net        60.0%    11
>>> 180.1 179.1 175.4 183.0   3.2
>>> 17. 68.87.158.10                                         44.4%    10
>>> 174.8 181.7 174.8 189.7   5.6
>>> 18. c-73-218-149-14.hsd1.ma.comcast.net                  60.0%    10
>>> 173.0 181.0 173.0 191.2   8.7
>>> 19. ???
>>>
>>>
>>> From COX in SoCal
>>>
>>>  3. wsip-216-231-24-181.oc.oc.cox.net                     0.0%     8
>>> 1.8   1.5   1.4   1.8   0.1
>>>  4. ip68-4-11-200.oc.oc.cox.net                           0.0%     8
>>> 1.2   1.2   1.2   1.4   0.1
>>>  5. 68.1.1.167                                            0.0%     8
>>> 2.7   2.8   2.7   2.9   0.1
>>>  6. ae-7.a00.lsanca20.us.bb.gin.ntt.net                   0.0%     8
>>> 3.3   3.3   2.7   4.2   0.6
>>>  7. ae-3.r01.lsanca20.us.bb.gin.ntt.net                   0.0%     8
>>>  69.4  69.5  69.4  69.8   0.1
>>>  8. ae-8.r23.lsanca07.us.bb.gin.ntt.net                   0.0%     8
>>> 2.9   4.6   2.9  12.0   3.1
>>>  9. ae-6.r22.asbnva02.us.bb.gin.ntt.net                  12.5%     8
>>>  66.3  66.5  66.2  67.1   0.3
>>> 10. ae-7.r06.asbnva02.us.bb.gin.ntt.net                   0.0%     8
>>>  66.4  66.4  66.2  66.6   0.1
>>> 11. ae-1.a02.asbnva02.us.bb.gin.ntt.net                   0.0%     8
>>>  66.2  67.2  66.1  72.8   2.3
>>> 12. ???
>>> 13. ???
>>> 14. 4.68.71.218                                          75.0%     8
>>> 202.6 204.3 202.6 206.0   2.4
>>> 15. be-2-ar01.chartford.ct.hartford.comcast.net          12.5%     8
>>> 202.3 203.3 197.6 208.0   4.4
>>> 16. be-31-sur01.waterbury.ct.hartford.comcast.net        37.5%     8
>>> 192.5 199.6 192.5 205.0   4.9
>>> 17. 68.87.158.14                                         42.9%     8
>>> 200.4 202.3 197.8 208.6   4.6
>>> 18. c-73-218-149-14.hsd1.ct.comcast.net                  42.9%     8
>>> 207.9 208.2 203.0 214.8   4.9
>>> 19. ???
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Nov 6, 2017 at 1:38 PM, John Neiberger via Outages <
>>> outages at outages.org> wrote:
>>>
>>>> Our on-net tools don't indicate a problem within the network, but I've
>>>> seen a couple of traceroutes that indicate a problem on Level 3, which
>>>> certainly would affect us. We're trying to nail this down. Any traceroutes
>>>> you have would be helpful.
>>>>
>>>> On Mon, Nov 6, 2017 at 11:35 AM, Brian R. Watters via Outages <
>>>> outages at outages.org> wrote:
>>>>
>>>>> Same issue is wide spread in California via both COAX and Fiber
>>>>> connections (Comcast).
>>>>>
>>>>> ------------------------------
>>>>> *From: *"Gary Leato via Outages" <outages at outages.org>
>>>>> *To: *"Jim Limmer" <jlimmer at vna-commh.org>
>>>>> *Cc: *outages at outages.org
>>>>> *Sent: *Monday, November 6, 2017 10:27:35 AM
>>>>> *Subject: *Re: [outages] cocmast packetloss and high latency
>>>>>
>>>>>
>>>>> we are seeing similiar comcast problems in central illinois rig
>>>>>
>>>>> On Mon, Nov 6, 2017 at 12:25 PM, Jim Limmer via Outages <
>>>>> outages at outages.org> wrote:
>>>>>
>>>>>> Yes. And whats worse is my deli nearby uses Comcast business for
>>>>>> their credit card reader.. guess I owe them for lunch tomorrow..
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Jim Limmer, CISSP
>>>>>>
>>>>>> < Information Security Officer
>>>>>>
>>>>>>  Systems and Network Architect >
>>>>>>
>>>>>> VNA Community Healthcare
>>>>>>
>>>>>> www.connecticuthomecare.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> [image: cid:image002.jpg at 01D332E1.E5C48B90] [image:
>>>>>> smCISSP-logo-2lines]
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From:* Outages [mailto:outages-bounces at outages.org] *On Behalf Of *Audrius
>>>>>> Peciukas via Outages
>>>>>> *Sent:* Monday, November 06, 2017 1:24 PM
>>>>>> *To:* outages at outages.org
>>>>>> *Subject:* [outages] cocmast packetloss and high latency
>>>>>>
>>>>>>
>>>>>>
>>>>>> anyone else is seeing lots of packet loss and 100ms+ latency mostly
>>>>>> in east coast on comcast network?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>>
>>>>>> Audrius Peciukas
>>>>>>
>>>>>> 1Wire Communications
>>>>>>
>>>>>> http://www.1wirecom.com
>>>>>>
>>>>>> _______________________________________________
>>>>>> Outages mailing list
>>>>>> Outages at outages.org
>>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Outages mailing list
>>>>> Outages at outages.org
>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> *Brian Watters*
>>>>>
>>>>>
>>>>> 559-579-1803 <(559)%20579-1803>
>>>>>
>>>>> brianwatters at totlcom.com
>>>>>
>>>>> http://www.totlcom.com
>>>>>
>>>>> *Very satisfied customers are our standard, not our goal.*
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Outages mailing list
>>>>> Outages at outages.org
>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>
>>>>
>>>
>>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>

It's not just just Level3. I'm seeing PL through XO as well.

  5.|-- 97.75.189.72               0
<https://mail.google.com/mail/u/0/72%C2%A0%20%C2%A0%20%C2%A0%20%C2%A0%20%C2%A0%20%C2%A0%20%C2%A0%20%C2%A00>.0%
  10    1.0   2.9   0.9  12.3   3.9
  6.|-- ae1-prv02-rt01.veracityne  0.0%    10    1.5   1.5   1.4   1.6   0.0
  7.|-- ip65-46-63-49 <https://mail.google.com/mail/u/0/65-46-63-49>.z
63-46-65 <https://mail.google.com/mail/u/0/63-46-65>.c  0.0%    10    3.0
 3.0   2.8   3.1   0.0
  8.|-- 216.156.16.24.ptr.us.xo.n  0.0%    10   61.7  60.6  60.3  61.7   0.3
  9.|-- 207.88.12.190.ptr.us.xo.n  0.0%    10   60.4  65.5  60.3  96.9  11.6
 10.|-- te0-12-0-0 <https://mail.google.com/mail/u/0/0-12-0-0>.rar3.sanjose-c
0.0%    10   60.3  60.8  60.3  61.9   0.3
 11.|-- 207.88.12.164.ptr.us.xo.n  0.0%    10   60.3  65.1  60.2  85.1   7.4
 12.|-- 207.88.12.213.ptr.us.xo.n  0.0%    10   62.0  61.0  60.2  64.0   1.2
 13.|-- 207.88.12.214.ptr.us.xo.n  0.0%    10   60.2  60.4  60.2  61.4   0.0
 14.|-- 207.88.14.181.ptr.us.xo.n  0.0%    10   60.1  61.4  60.1  72.4   3.9
 15.|-- ???                       100.0    10    0
<https://mail.google.com/mail/u/0/0%C2%A0%20%C2%A0%2010%C2%A0%20%C2%A0%200>.0
 0.0   0.0   0.0   0.0
 16.|-- 4.69.137.41               40
<https://mail.google.com/mail/u/0/41%C2%A0%20%C2%A0%20%C2%A0%20%C2%A0%20%C2%A0%20%C2%A0%20%C2%A0%20%C2%A040>.0%
  10   92.4  92.7  91.7  93.3   0.4
 17.|-- 4.68.63.18                40
<https://mail.google.com/mail/u/0/18%C2%A0%20%C2%A0%20%C2%A0%20%C2%A0%20%C2%A0%20%C2%A0%20%C2%A0%20%C2%A0%2040>.0%
  10  120.6 121.1 120.3 122.1   0.0
 18.|-- po-1-rur01.orem.ut.utah.c 40.0%    10  121.6 121.7 120.8 122.3   0.4
 19.|-- lag-1-acr04.orem.ut.utah. 30.0%    10  121.8 122.4 121.4 123.5   0.4
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20171106/45f0c68c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: TOTLCOM-MANAGED-SERVICES-LOGO-h60.gif
Type: image/gif
Size: 3404 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/outages/attachments/20171106/45f0c68c/attachment-0001.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 4569 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/outages/attachments/20171106/45f0c68c/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 5367 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/outages/attachments/20171106/45f0c68c/attachment-0001.jpg>


More information about the Outages mailing list