[outages] packet loss to us-east-1

Grant Ridder shortdudey123 at gmail.com
Mon Nov 23 14:54:32 EST 2015 

It is related for some since Slack is in us-east-1

On Mon, Nov 23, 2015 at 11:52 AM, Chris Swingler <chris at chrisswingler.com>
wrote:

> I'm not sure these are related (I'm seeing zero packet loss to Slack from
> Chicago on Cogent) but Slack's definitely broken. Seemed to have been a
> cascading failure, a colleague of mine complained over IRC he got booted
> from his, then I got booted from one Slack team, then another.
>
> On Nov 23, 2015, at 1:49 PM, Justin Oeder via Outages <outages at outages.org>
> wrote:
>
> Slack, which is hosted with Amazon is currently down.
>
>
>
> ------------------------------
> *From: *"Grant Ridder via Outages" <outages at outages.org>
> *To: *"Mike Lyon" <mike.lyon at gmail.com>
> *Cc: *outages at outages.org, "Tomoc Field" <tomoc.field at sonic.com>
> *Sent: *Monday, November 23, 2015 2:43:57 PM
> *Subject: *Re: [outages] packet loss to us-east-1
>
> Seeing this issue again
>   4. AS46375 50.ae4.gw.pao1.sonic.net                   0.0%    10   15.5
>  23.2   2.9  89.3  28.6
>   5. AS8121  199.127.107.189                            0.0%    10   64.2
>  26.6   3.7  85.7  28.2
>   6. AS8121  xe0-0-0-7.core2.sv8.layer42.net            0.0%    10   66.6
>  24.8   3.5 102.1  33.4
>   7. AS16509 72.21.221.176                              0.0%    10   64.2
>  26.9   3.0  99.1  32.4
>   8. AS???   54.240.243.50                              0.0%    10  147.4
> 115.9  88.2 176.9  27.9
>   9. AS???   54.240.243.61                             90.0%    10   78.4
>  78.4  78.4  78.4   0.0
>  10. AS???   54.239.42.38                              10.0%    10  130.4
> 106.9  71.8 167.6  31.8
>  11. AS16509 54.239.108.120                            70.0%    10  178.9
> 125.2  90.0 178.9  47.2
>  12. AS16509 54.239.111.235                            20.0%    10  143.1
>  99.0  70.8 158.1  34.7
>  13. AS16509 54.239.111.230                            70.0%    10   89.5
>  93.9  88.7 103.5   8.3
>
> -Grant
>
> On Fri, Nov 20, 2015 at 5:27 PM, Mike Lyon <mike.lyon at gmail.com> wrote:
>
>> Hmmm,
>> It shows as if it is getting there but look at that jump from 15ms to
>> 85ms on hops 7 and 8:
>>
>> traceroute amazon.com
>>
>> traceroute: Warning: amazon.com has multiple addresses; using
>> 176.32.103.205
>>
>> traceroute to amazon.com (176.32.103.205), 64 hops max, 52 byte packets
>>
>>  1  192.168.20.1 (192.168.20.1)  0.708 ms  0.395 ms  0.328 ms
>>
>>  2  162.250.208.141 (162.250.208.141)  0.671 ms  0.622 ms  0.588 ms
>>
>>  3  10.25.0.1 (10.25.0.1)  1.985 ms  4.165 ms  21.626 ms
>>
>>  4  10.20.0.1 (10.20.0.1)  7.852 ms  7.626 ms  5.214 ms
>>
>>  5  10.0.0.1 (10.0.0.1)  8.041 ms  17.839 ms  34.089 ms
>>
>>  6  69.36.226.193 (69.36.226.193)  5.477 ms  14.228 ms  20.203 ms
>>
>>  7  xe0-2-0-4.core4.scl.layer42.net (65.50.198.237)  15.153 ms  13.475
>> ms  8.706 ms
>>
>>  8  205.251.229.145 (205.251.229.145)  85.231 ms
>>
>>     54.240.242.108 (54.240.242.108)  96.157 ms
>>
>>     205.251.229.145 (205.251.229.145)  175.633 ms
>>
>>  9  54.240.242.141 (54.240.242.141)  100.007 ms  234.602 ms
>>
>>     54.240.242.121 (54.240.242.121)  94.263 ms
>>
>> 10  54.239.109.102 (54.239.109.102)  110.966 ms
>>
>>     54.239.109.136 (54.239.109.136)  123.322 ms
>>
>>     54.239.110.50 (54.239.110.50)  105.029 ms
>>
>> 11  54.239.109.143 (54.239.109.143)  84.239 ms
>>
>>     54.239.110.23 (54.239.110.23)  91.540 ms
>>
>>     54.239.110.57 (54.239.110.57)  86.620 ms
>>
>> 12  54.239.109.143 (54.239.109.143)  91.382 ms
>>
>>     205.251.245.168 (205.251.245.168)  87.930 ms
>>
>>     54.239.109.143 (54.239.109.143)  76.879 ms
>>
>> 13  * 205.251.245.168 (205.251.245.168)  90.780 ms  96.386 ms
>>
>> On Fri, Nov 20, 2015 at 5:24 PM, Grant Ridder <shortdudey123 at gmail.com>
>> wrote:
>>
>>> Looks like they solved it by routing around layer42 and go via cogent.
>>> That in itself is a bad idea :/
>>>  4. AS46375 50.ae4.gw.pao1.sonic.net                    1.2%    81
>>>  3.5  12.0   2.6  91.0  16.2
>>>  5. AS174   te0-0-0-15.ccr21.sjc04.atlas.cogentco.com   6.2%    81
>>>  6.6  10.5   3.2  79.9  11.7
>>>  6. AS174   be2016.ccr22.sfo01.atlas.cogentco.com       3.7%    81
>>> 15.3  14.6   4.2  98.9  16.3
>>>  7. AS174   be2133.ccr22.mci01.atlas.cogentco.com       4.9%    81
>>> 47.7  50.4  41.8 113.6  13.3
>>>  8. AS174   be2157.ccr42.ord01.atlas.cogentco.com      70.0%    81
>>> 54.1  67.2  53.9 140.2  21.3
>>>  9. AS174   be2217.ccr41.ord03.atlas.cogentco.com      33.3%    81
>>> 57.3  63.6  54.3 158.4  17.0
>>> 10. AS174   ntt.ord03.atlas.cogentco.com               66.2%    81
>>> 56.9  72.0  56.5 126.8  20.6
>>> 11. AS2914  ae-3.amazon.chcgil09.us.bb.gin.ntt.net     34.6%    81
>>> 61.5  61.8  55.9  89.7   8.1
>>>
>>> -Grant
>>>
>>> On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon <mike.lyon at gmail.com> wrote:
>>>
>>>> Oh thank god! I was seeing it too but wasn't sure if was Layer42 or
>>>> Amazon that was having the issue...
>>>> Thanks,
>>>> Mike
>>>>
>>>>
>>>> On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages <
>>>> outages at outages.org> wrote:
>>>>
>>>>> Saw that, thanks!
>>>>> -Grant
>>>>>
>>>>> On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <tomoc.field at sonic.com>
>>>>> wrote:
>>>>>
>>>>>> Yes. 70%+ packet loss from 7065 to all Amazon services as far as I
>>>>>> can tell. This was experienced via both layer42 and XO, but Amazon appears
>>>>>> to have resolved their issues at this time.
>>>>>>
>>>>>> -Tomoc
>>>>>>
>>>>>>
>>>>>> On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages
>>>>>> wrote:
>>>>>> > Anyone else seeing packet log into us-east-1?
>>>>>> >
>>>>>> > 54.240.243.121 (AS16509) is Amazon
>>>>>> >
>>>>>> > MTR report excerpt
>>>>>> >   4. AS46375 50.ae4.gw.pao1.sonic.net                 0.0%    10
>>>>>>   2.5
>>>>>> > 8.3 2.5 37.5 10.7
>>>>>> >   5. AS8121  199.127.107.189                          0.0%    10
>>>>>> 3.8
>>>>>> > 7.9   3.4  34.9   9.5
>>>>>> >   6. AS8121  xe0-0-0-7.core2.sv8.layer42.net          0.0%    10
>>>>>>   3.9
>>>>>> > 6.0   3.4  17.6   4.6
>>>>>> >   7. AS16509 72.21.221.176                            0.0%    10
>>>>>> 4.6
>>>>>> > 5.2   3.0  15.5   3.6
>>>>>> >   8. AS???   54.240.243.110                           0.0%    10
>>>>>>  98.6
>>>>>> >  96.4  86.9 104.4   6.2
>>>>>> >   9. AS???   54.240.243.121                          90.0%    10
>>>>>>  85.7
>>>>>> >  85.7  85.7  85.7   0.0
>>>>>> >  10. AS16509 205.251.229.104                         20.0%    10
>>>>>>  87.5
>>>>>> >  85.2  82.5  87.8   2.0
>>>>>> >  11. AS???   54.239.42.29                            70.0%    10
>>>>>>  85.0
>>>>>> >  84.3  82.8  85.2   1.0
>>>>>> >  12. AS16509 54.239.109.238                          30.0%    10
>>>>>>  87.4
>>>>>> >  96.8  87.4 105.2   6.5
>>>>>> >      AS16509 54.239.110.16
>>>>>> >
>>>>>> >
>>>>>> > -Grant
>>>>>>
>>>>>> > _______________________________________________
>>>>>> > Outages mailing list
>>>>>> > Outages at outages.org
>>>>>> > https://puck.nether.net/mailman/listinfo/outages
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Outages mailing list
>>>>> Outages at outages.org
>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Mike Lyon
>>>> 408-621-4826
>>>> mike.lyon at gmail.com
>>>>
>>>> http://www.linkedin.com/in/mlyon
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Mike Lyon
>> 408-621-4826
>> mike.lyon at gmail.com
>>
>> http://www.linkedin.com/in/mlyon
>>
>>
>>
>>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20151123/63141d65/attachment.htm>

More information about the Outages mailing list