[outages] packet loss to us-east-1
Chris Swingler
chris at chrisswingler.com
Mon Nov 23 14:52:50 EST 2015
I'm not sure these are related (I'm seeing zero packet loss to Slack from Chicago on Cogent) but Slack's definitely broken. Seemed to have been a cascading failure, a colleague of mine complained over IRC he got booted from his, then I got booted from one Slack team, then another.
> On Nov 23, 2015, at 1:49 PM, Justin Oeder via Outages <outages at outages.org> wrote:
>
> Slack, which is hosted with Amazon is currently down.
>
>
>
> From: "Grant Ridder via Outages" <outages at outages.org>
> To: "Mike Lyon" <mike.lyon at gmail.com>
> Cc: outages at outages.org, "Tomoc Field" <tomoc.field at sonic.com>
> Sent: Monday, November 23, 2015 2:43:57 PM
> Subject: Re: [outages] packet loss to us-east-1
>
> Seeing this issue again
> 4. AS46375 50.ae4.gw.pao1.sonic.net <http://50.ae4.gw.pao1.sonic.net/> 0.0% 10 15.5 23.2 2.9 89.3 28.6
> 5. AS8121 199.127.107.189 0.0% 10 64.2 26.6 3.7 85.7 28.2
> 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net <http://xe0-0-0-7.core2.sv8.layer42.net/> 0.0% 10 66.6 24.8 3.5 102.1 33.4
> 7. AS16509 72.21.221.176 0.0% 10 64.2 26.9 3.0 99.1 32.4
> 8. AS??? 54.240.243.50 0.0% 10 147.4 115.9 88.2 176.9 27.9
> 9. AS??? 54.240.243.61 90.0% 10 78.4 78.4 78.4 78.4 0.0
> 10. AS??? 54.239.42.38 10.0% 10 130.4 106.9 71.8 167.6 31.8
> 11. AS16509 54.239.108.120 70.0% 10 178.9 125.2 90.0 178.9 47.2
> 12. AS16509 54.239.111.235 20.0% 10 143.1 99.0 70.8 158.1 34.7
> 13. AS16509 54.239.111.230 70.0% 10 89.5 93.9 88.7 103.5 8.3
>
> -Grant
>
> On Fri, Nov 20, 2015 at 5:27 PM, Mike Lyon <mike.lyon at gmail.com <mailto:mike.lyon at gmail.com>> wrote:
> Hmmm,
> It shows as if it is getting there but look at that jump from 15ms to 85ms on hops 7 and 8:
>
> traceroute amazon.com <http://amazon.com/>
> traceroute: Warning: amazon.com <http://amazon.com/> has multiple addresses; using 176.32.103.205
>
> traceroute to amazon.com <http://amazon.com/> (176.32.103.205), 64 hops max, 52 byte packets
>
> 1 192.168.20.1 (192.168.20.1) 0.708 ms 0.395 ms 0.328 ms
>
> 2 162.250.208.141 (162.250.208.141) 0.671 ms 0.622 ms 0.588 ms
>
> 3 10.25.0.1 (10.25.0.1) 1.985 ms 4.165 ms 21.626 ms
>
> 4 10.20.0.1 (10.20.0.1) 7.852 ms 7.626 ms 5.214 ms
>
> 5 10.0.0.1 (10.0.0.1) 8.041 ms 17.839 ms 34.089 ms
>
> 6 69.36.226.193 (69.36.226.193) 5.477 ms 14.228 ms 20.203 ms
>
> 7 xe0-2-0-4.core4.scl.layer42.net <http://xe0-2-0-4.core4.scl.layer42.net/> (65.50.198.237) 15.153 ms 13.475 ms 8.706 ms
>
> 8 205.251.229.145 (205.251.229.145) 85.231 ms
>
> 54.240.242.108 (54.240.242.108) 96.157 ms
>
> 205.251.229.145 (205.251.229.145) 175.633 ms
>
> 9 54.240.242.141 (54.240.242.141) 100.007 ms 234.602 ms
>
> 54.240.242.121 (54.240.242.121) 94.263 ms
>
> 10 54.239.109.102 (54.239.109.102) 110.966 ms
>
> 54.239.109.136 (54.239.109.136) 123.322 ms
>
> 54.239.110.50 (54.239.110.50) 105.029 ms
>
> 11 54.239.109.143 (54.239.109.143) 84.239 ms
>
> 54.239.110.23 (54.239.110.23) 91.540 ms
>
> 54.239.110.57 (54.239.110.57) 86.620 ms
>
> 12 54.239.109.143 (54.239.109.143) 91.382 ms
>
> 205.251.245.168 (205.251.245.168) 87.930 ms
>
> 54.239.109.143 (54.239.109.143) 76.879 ms
>
> 13 * 205.251.245.168 (205.251.245.168) 90.780 ms 96.386 ms
>
>
> On Fri, Nov 20, 2015 at 5:24 PM, Grant Ridder <shortdudey123 at gmail.com <mailto:shortdudey123 at gmail.com>> wrote:
> Looks like they solved it by routing around layer42 and go via cogent. That in itself is a bad idea :/
> 4. AS46375 50.ae4.gw.pao1.sonic.net <http://50.ae4.gw.pao1.sonic.net/> 1.2% 81 3.5 12.0 2.6 91.0 16.2
> 5. AS174 te0-0-0-15.ccr21.sjc04.atlas.cogentco.com <http://te0-0-0-15.ccr21.sjc04.atlas.cogentco.com/> 6.2% 81 6.6 10.5 3.2 79.9 11.7
> 6. AS174 be2016.ccr22.sfo01.atlas.cogentco.com <http://be2016.ccr22.sfo01.atlas.cogentco.com/> 3.7% 81 15.3 14.6 4.2 98.9 16.3
> 7. AS174 be2133.ccr22.mci01.atlas.cogentco.com <http://be2133.ccr22.mci01.atlas.cogentco.com/> 4.9% 81 47.7 50.4 41.8 113.6 13.3
> 8. AS174 be2157.ccr42.ord01.atlas.cogentco.com <http://be2157.ccr42.ord01.atlas.cogentco.com/> 70.0% 81 54.1 67.2 53.9 140.2 21.3
> 9. AS174 be2217.ccr41.ord03.atlas.cogentco.com <http://be2217.ccr41.ord03.atlas.cogentco.com/> 33.3% 81 57.3 63.6 54.3 158.4 17.0
> 10. AS174 ntt.ord03.atlas.cogentco.com <http://ntt.ord03.atlas.cogentco.com/> 66.2% 81 56.9 72.0 56.5 126.8 20.6
> 11. AS2914 ae-3.amazon.chcgil09.us.bb.gin.ntt.net <http://ae-3.amazon.chcgil09.us.bb.gin.ntt.net/> 34.6% 81 61.5 61.8 55.9 89.7 8.1
>
> -Grant
>
> On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon <mike.lyon at gmail.com <mailto:mike.lyon at gmail.com>> wrote:
> Oh thank god! I was seeing it too but wasn't sure if was Layer42 or Amazon that was having the issue...
> Thanks,
> Mike
>
>
> On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages <outages at outages.org <mailto:outages at outages.org>> wrote:
> Saw that, thanks!
> -Grant
>
> On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <tomoc.field at sonic.com <mailto:tomoc.field at sonic.com>> wrote:
> Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. This was experienced via both layer42 and XO, but Amazon appears to have resolved their issues at this time.
>
> -Tomoc
>
>
> On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote:
> > Anyone else seeing packet log into us-east-1?
> >
> > 54.240.243.121 (AS16509) is Amazon
> >
> > MTR report excerpt
> > 4. AS46375 50.ae4.gw.pao1.sonic.net <http://50.ae4.gw.pao1.sonic.net/> 0.0% 10 2.5
> > 8.3 2.5 37.5 10.7 <tel:8.3%20%20%202.5%20%2037.5%20%2010.7>
> > 5. AS8121 199.127.107.189 0.0% 10 3.8
> > 7.9 3.4 34.9 9.5
> > 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net <http://xe0-0-0-7.core2.sv8.layer42.net/> 0.0% 10 3.9
> > 6.0 3.4 17.6 4.6
> > 7. AS16509 72.21.221.176 0.0% 10 4.6
> > 5.2 3.0 15.5 3.6
> > 8. AS??? 54.240.243.110 0.0% 10 98.6
> > 96.4 86.9 104.4 6.2
> > 9. AS??? 54.240.243.121 90.0% 10 85.7
> > 85.7 85.7 85.7 0.0
> > 10. AS16509 205.251.229.104 20.0% 10 87.5
> > 85.2 82.5 87.8 2.0
> > 11. AS??? 54.239.42.29 70.0% 10 85.0
> > 84.3 82.8 85.2 1.0
> > 12. AS16509 54.239.109.238 30.0% 10 87.4
> > 96.8 87.4 105.2 6.5
> > AS16509 54.239.110.16
> >
> >
> > -Grant
>
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org <mailto:Outages at outages.org>
> > https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
>
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org <mailto:Outages at outages.org>
> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
>
>
>
>
> --
> Mike Lyon
> 408-621-4826 <tel:408-621-4826>
> mike.lyon at gmail.com <mailto:mike.lyon at gmail.com>
> http://www.linkedin.com/in/mlyon <http://www.linkedin.com/in/mlyon>
>
>
>
>
>
>
>
> --
> Mike Lyon
> 408-621-4826 <tel:408-621-4826>
> mike.lyon at gmail.com <mailto:mike.lyon at gmail.com>
> http://www.linkedin.com/in/mlyon <http://www.linkedin.com/in/mlyon>
>
>
>
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20151123/f4954462/attachment.htm>
More information about the Outages
mailing list