[outages] packet loss to us-east-1

Mon Nov 23 14:49:49 EST 2015

Slack, which is hosted with Amazon is currently down. 

From: "Grant Ridder via Outages" <outages at outages.org> 
To: "Mike Lyon" <mike.lyon at gmail.com> 
Cc: outages at outages.org, "Tomoc Field" <tomoc.field at sonic.com> 
Sent: Monday, November 23, 2015 2:43:57 PM 
Subject: Re: [outages] packet loss to us-east-1 

Seeing this issue again 
4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 15.5 23.2 2.9 89.3 28.6 
5. AS8121 199.127.107.189 0.0% 10 64.2 26.6 3.7 85.7 28.2 
6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 66.6 24.8 3.5 102.1 33.4 
7. AS16509 72.21.221.176 0.0% 10 64.2 26.9 3.0 99.1 32.4 
8. AS??? 54.240.243.50 0.0% 10 147.4 115.9 88.2 176.9 27.9 
9. AS??? 54.240.243.61 90.0% 10 78.4 78.4 78.4 78.4 0.0 
10. AS??? 54.239.42.38 10.0% 10 130.4 106.9 71.8 167.6 31.8 
11. AS16509 54.239.108.120 70.0% 10 178.9 125.2 90.0 178.9 47.2 
12. AS16509 54.239.111.235 20.0% 10 143.1 99.0 70.8 158.1 34.7 
13. AS16509 54.239.111.230 70.0% 10 89.5 93.9 88.7 103.5 8.3 

-Grant 

On Fri, Nov 20, 2015 at 5:27 PM, Mike Lyon < mike.lyon at gmail.com > wrote: 

Hmmm, 
It shows as if it is getting there but look at that jump from 15ms to 85ms on hops 7 and 8: 

traceroute amazon.com 

traceroute: Warning: amazon.com has multiple addresses; using 176.32.103.205 

traceroute to amazon.com (176.32.103.205), 64 hops max, 52 byte packets 

1 192.168.20.1 (192.168.20.1) 0.708 ms 0.395 ms 0.328 ms 

2 162.250.208.141 (162.250.208.141) 0.671 ms 0.622 ms 0.588 ms 

3 10.25.0.1 (10.25.0.1) 1.985 ms 4.165 ms 21.626 ms 

4 10.20.0.1 (10.20.0.1) 7.852 ms 7.626 ms 5.214 ms 

5 10.0.0.1 (10.0.0.1) 8.041 ms 17.839 ms 34.089 ms 

6 69.36.226.193 (69.36.226.193) 5.477 ms 14.228 ms 20.203 ms 

7 xe0-2-0-4.core4.scl.layer42.net (65.50.198.237) 15.153 ms 13.475 ms 8.706 ms 

8 205.251.229.145 (205.251.229.145) 85.231 ms 

54.240.242.108 (54.240.242.108) 96.157 ms 

205.251.229.145 (205.251.229.145) 175.633 ms 

9 54.240.242.141 (54.240.242.141) 100.007 ms 234.602 ms 

54.240.242.121 (54.240.242.121) 94.263 ms 

10 54.239.109.102 (54.239.109.102) 110.966 ms 

54.239.109.136 (54.239.109.136) 123.322 ms 

54.239.110.50 (54.239.110.50) 105.029 ms 

11 54.239.109.143 (54.239.109.143) 84.239 ms 

54.239.110.23 (54.239.110.23) 91.540 ms 

54.239.110.57 (54.239.110.57) 86.620 ms 

12 54.239.109.143 (54.239.109.143) 91.382 ms 

205.251.245.168 (205.251.245.168) 87.930 ms 

54.239.109.143 (54.239.109.143) 76.879 ms 

13 * 205.251.245.168 (205.251.245.168) 90.780 ms 96.386 ms 

On Fri, Nov 20, 2015 at 5:24 PM, Grant Ridder < shortdudey123 at gmail.com > wrote: 

BQ_BEGIN

Looks like they solved it by routing around layer42 and go via cogent. That in itself is a bad idea :/ 
4. AS46375 50.ae4.gw.pao1.sonic.net 1.2% 81 3.5 12.0 2.6 91.0 16.2 
5. AS174 te0-0-0-15.ccr21.sjc04.atlas.cogentco.com 6.2% 81 6.6 10.5 3.2 79.9 11.7 
6. AS174 be2016.ccr22.sfo01.atlas.cogentco.com 3.7% 81 15.3 14.6 4.2 98.9 16.3 
7. AS174 be2133.ccr22.mci01.atlas.cogentco.com 4.9% 81 47.7 50.4 41.8 113.6 13.3 
8. AS174 be2157.ccr42.ord01.atlas.cogentco.com 70.0% 81 54.1 67.2 53.9 140.2 21.3 
9. AS174 be2217.ccr41.ord03.atlas.cogentco.com 33.3% 81 57.3 63.6 54.3 158.4 17.0 
10. AS174 ntt.ord03.atlas.cogentco.com 66.2% 81 56.9 72.0 56.5 126.8 20.6 
11. AS2914 ae-3.amazon.chcgil09.us.bb.gin.ntt.net 34.6% 81 61.5 61.8 55.9 89.7 8.1 

-Grant 

On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon < mike.lyon at gmail.com > wrote: 

BQ_BEGIN

Oh thank god! I was seeing it too but wasn't sure if was Layer42 or Amazon that was having the issue... 
Thanks, 
Mike 

On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages < outages at outages.org > wrote: 

BQ_BEGIN

Saw that, thanks! 
-Grant 

On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field < tomoc.field at sonic.com > wrote: 

BQ_BEGIN
Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. This was experienced via both layer42 and XO, but Amazon appears to have resolved their issues at this time. 

-Tomoc 

On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote: 
> Anyone else seeing packet log into us-east-1? 
> 
> 54.240.243.121 (AS16509) is Amazon 
> 
> MTR report excerpt 
> 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 2.5 
> 8.3 2.5 37.5 10.7 
> 5. AS8121 199.127.107.189 0.0% 10 3.8 
> 7.9 3.4 34.9 9.5 
> 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 3.9 
> 6.0 3.4 17.6 4.6 
> 7. AS16509 72.21.221.176 0.0% 10 4.6 
> 5.2 3.0 15.5 3.6 
> 8. AS??? 54.240.243.110 0.0% 10 98.6 
> 96.4 86.9 104.4 6.2 
> 9. AS??? 54.240.243.121 90.0% 10 85.7 
> 85.7 85.7 85.7 0.0 
> 10. AS16509 205.251.229.104 20.0% 10 87.5 
> 85.2 82.5 87.8 2.0 
> 11. AS??? 54.239.42.29 70.0% 10 85.0 
> 84.3 82.8 85.2 1.0 
> 12. AS16509 54.239.109.238 30.0% 10 87.4 
> 96.8 87.4 105.2 6.5 
> AS16509 54.239.110.16 
> 
> 
> -Grant 

> _______________________________________________ 
> Outages mailing list 
> Outages at outages.org 
> https://puck.nether.net/mailman/listinfo/outages 

_______________________________________________ 
Outages mailing list 
Outages at outages.org 
https://puck.nether.net/mailman/listinfo/outages 

BQ_END

-- 
Mike Lyon 
408-621-4826 
mike.lyon at gmail.com 

http://www.linkedin.com/in/mlyon 

BQ_END

BQ_END

-- 
Mike Lyon 
408-621-4826 
mike.lyon at gmail.com 

http://www.linkedin.com/in/mlyon 

BQ_END

_______________________________________________ 
Outages mailing list 
Outages at outages.org 
https://puck.nether.net/mailman/listinfo/outages 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20151123/8b0f76bc/attachment.htm>