<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p dir="ltr">Confirmed from Iceland </p>
<p dir="ltr">~ Gunnar</p>
<div class="quote">On Nov 23, 2015 19:55, Grant Ridder via Outages <outages@outages.org> wrote:<br type="attribution">
<blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="ltr">It is related for some since Slack is in us-east-1</div>
<div><br>
<div class="elided-text">On Mon, Nov 23, 2015 at 11:52 AM, Chris Swingler <span dir="ltr">
<<a href="mailto:chris@chrisswingler.com">chris@chrisswingler.com</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">I'm not sure these are related (I'm seeing zero packet loss to Slack from Chicago on Cogent) but Slack's definitely broken. Seemed to have been a cascading failure, a colleague of mine complained over IRC he got booted from
his, then I got booted from one Slack team, then another.
<div>
<div>
<div><br>
<div>
<blockquote>
<div>On Nov 23, 2015, at 1:49 PM, Justin Oeder via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:</div>
<br>
<div>
<div>
<div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:12pt">
<div>Slack, which is hosted with Amazon is currently down.<br>
</div>
<div><br>
</div>
<div><br>
</div>
<br>
<hr>
<div><b>From: </b>"Grant Ridder via Outages" <<a href="mailto:outages@outages.org">outages@outages.org</a>><br>
<b>To: </b>"Mike Lyon" <<a href="mailto:mike.lyon@gmail.com">mike.lyon@gmail.com</a>><br>
<b>Cc: </b><a href="mailto:outages@outages.org">outages@outages.org</a>, "Tomoc Field" <<a href="mailto:tomoc.field@sonic.com">tomoc.field@sonic.com</a>><br>
<b>Sent: </b>Monday, November 23, 2015 2:43:57 PM<br>
<b>Subject: </b>Re: [outages] packet loss to us-east-1<br>
</div>
<br>
<div>
<div dir="ltr">Seeing this issue again<br>
<div>
<div> 4. AS46375 <a href="http://50.ae4.gw.pao1.sonic.net/">50.ae4.gw.pao1.sonic.net</a> 0.0% 10 15.5 23.2 2.9 89.3 28.6</div>
<div> 5. AS8121 199.127.107.189 0.0% 10 64.2 26.6 3.7 85.7 28.2</div>
<div> 6. AS8121 <a href="http://xe0-0-0-7.core2.sv8.layer42.net/">xe0-0-0-7.core2.sv8.layer42.net</a> 0.0% 10 66.6 24.8 3.5 102.1 33.4</div>
<div> 7. AS16509 72.21.221.176 0.0% 10 64.2 26.9 3.0 99.1 32.4</div>
<div> 8. AS??? 54.240.243.50 0.0% 10 147.4 115.9 88.2 176.9 27.9</div>
<div> 9. AS??? 54.240.243.61 90.0% 10 78.4 78.4 78.4 78.4 0.0</div>
<div> 10. AS??? 54.239.42.38 10.0% 10 130.4 106.9 71.8 167.6 31.8</div>
<div> 11. AS16509 54.239.108.120 70.0% 10 178.9 125.2 90.0 178.9 47.2</div>
<div> 12. AS16509 54.239.111.235 20.0% 10 143.1 99.0 70.8 158.1 34.7</div>
<div> 13. AS16509 54.239.111.230 70.0% 10 89.5 93.9 88.7 103.5 8.3</div>
</div>
<br>
<div>-Grant</div>
</div>
<div><br>
<div class="elided-text">On Fri, Nov 20, 2015 at 5:27 PM, Mike Lyon <span dir="ltr">
<<a href="mailto:mike.lyon@gmail.com">mike.lyon@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hmmm,<br>
<div>It shows as if it is getting there but look at that jump from 15ms to 85ms on hops 7 and 8:</div>
<br>
<div>
<p>traceroute <a href="http://amazon.com/">amazon.com</a><br>
</p>
<p>traceroute: Warning: <a href="http://amazon.com/">amazon.com</a> has multiple addresses; using 176.32.103.205</p>
<p>traceroute to <a href="http://amazon.com/">amazon.com</a> (176.32.103.205), 64 hops max, 52 byte packets</p>
<p> 1 192.168.20.1 (192.168.20.1) 0.708 ms 0.395 ms 0.328 ms</p>
<p> 2 162.250.208.141 (162.250.208.141) 0.671 ms 0.622 ms 0.588 ms</p>
<p> 3 10.25.0.1 (10.25.0.1) 1.985 ms 4.165 ms 21.626 ms</p>
<p> 4 10.20.0.1 (10.20.0.1) 7.852 ms 7.626 ms 5.214 ms</p>
<p> 5 10.0.0.1 (10.0.0.1) 8.041 ms 17.839 ms 34.089 ms</p>
<p> 6 69.36.226.193 (69.36.226.193) 5.477 ms 14.228 ms 20.203 ms</p>
<p> 7 <a href="http://xe0-2-0-4.core4.scl.layer42.net/">xe0-2-0-4.core4.scl.layer42.net</a> (65.50.198.237) 15.153 ms 13.475 ms 8.706 ms</p>
<p> 8 205.251.229.145 (205.251.229.145) 85.231 ms</p>
<p> 54.240.242.108 (54.240.242.108) 96.157 ms</p>
<p> 205.251.229.145 (205.251.229.145) 175.633 ms</p>
<p> 9 54.240.242.141 (54.240.242.141) 100.007 ms 234.602 ms</p>
<p> 54.240.242.121 (54.240.242.121) 94.263 ms</p>
<p>10 54.239.109.102 (54.239.109.102) 110.966 ms</p>
<p> 54.239.109.136 (54.239.109.136) 123.322 ms</p>
<p> 54.239.110.50 (54.239.110.50) 105.029 ms</p>
<p>11 54.239.109.143 (54.239.109.143) 84.239 ms</p>
<p> 54.239.110.23 (54.239.110.23) 91.540 ms</p>
<p> 54.239.110.57 (54.239.110.57) 86.620 ms</p>
<p>12 54.239.109.143 (54.239.109.143) 91.382 ms</p>
<p> 205.251.245.168 (205.251.245.168) 87.930 ms</p>
<p> 54.239.109.143 (54.239.109.143) 76.879 ms</p>
<p>13 * 205.251.245.168 (205.251.245.168) 90.780 ms 96.386 ms</p>
</div>
</div>
<div>
<div>
<div><br>
<div class="elided-text">On Fri, Nov 20, 2015 at 5:24 PM, Grant Ridder <span dir="ltr">
<<a href="mailto:shortdudey123@gmail.com">shortdudey123@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Looks like they solved it by routing around layer42 and go via cogent. That in itself is a bad idea :/<br>
<div>
<div> 4. AS46375 <a href="http://50.ae4.gw.pao1.sonic.net/">50.ae4.gw.pao1.sonic.net</a> 1.2% 81 3.5 12.0 2.6 91.0 16.2</div>
<div> 5. AS174 <a href="http://te0-0-0-15.ccr21.sjc04.atlas.cogentco.com/">te0-0-0-15.ccr21.sjc04.atlas.cogentco.com</a> 6.2% 81 6.6 10.5 3.2 79.9 11.7</div>
<div> 6. AS174 <a href="http://be2016.ccr22.sfo01.atlas.cogentco.com/">be2016.ccr22.sfo01.atlas.cogentco.com</a> 3.7% 81 15.3 14.6 4.2 98.9 16.3</div>
<div> 7. AS174 <a href="http://be2133.ccr22.mci01.atlas.cogentco.com/">be2133.ccr22.mci01.atlas.cogentco.com</a> 4.9% 81 47.7 50.4 41.8 113.6 13.3</div>
<div> 8. AS174 <a href="http://be2157.ccr42.ord01.atlas.cogentco.com/">be2157.ccr42.ord01.atlas.cogentco.com</a> 70.0% 81 54.1 67.2 53.9 140.2 21.3</div>
<div> 9. AS174 <a href="http://be2217.ccr41.ord03.atlas.cogentco.com/">be2217.ccr41.ord03.atlas.cogentco.com</a> 33.3% 81 57.3 63.6 54.3 158.4 17.0</div>
<div>10. AS174 <a href="http://ntt.ord03.atlas.cogentco.com/">ntt.ord03.atlas.cogentco.com</a> 66.2% 81 56.9 72.0 56.5 126.8 20.6</div>
<div>11. AS2914 <a href="http://ae-3.amazon.chcgil09.us.bb.gin.ntt.net/">ae-3.amazon.chcgil09.us.bb.gin.ntt.net</a> 34.6% 81 61.5 61.8 55.9 89.7 8.1</div>
<span style="color:#888888"><br>
</span></div>
<span style="color:#888888"></span>
<div>-Grant</div>
</div>
<div>
<div>
<div><br>
<div class="elided-text">On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon <span dir="ltr">
<<a href="mailto:mike.lyon@gmail.com">mike.lyon@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Oh thank god! I was seeing it too but wasn't sure if was Layer42 or Amazon that was having the issue...<br>
<div>Thanks,</div>
<div>Mike</div>
<br>
</div>
<div>
<div>
<div><br>
<div class="elided-text">On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages
<span dir="ltr"><<a href="mailto:outages@outages.org">outages@outages.org</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Saw that, thanks!<br>
<div>-Grant</div>
</div>
<div><br>
<div class="elided-text">On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <span dir="ltr">
<<a href="mailto:tomoc.field@sonic.com">tomoc.field@sonic.com</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex">
Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. This was experienced via both layer42 and XO, but Amazon appears to have resolved their issues at this time.<br>
<br>
-Tomoc
<div>
<div><br>
<div>
<div><br>
On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote:<br>
> Anyone else seeing packet log into us-east-1?<br>
><br>
> 54.240.243.121 (AS16509) is Amazon<br>
><br>
> MTR report excerpt<br>
> 4. AS46375 <a href="http://50.ae4.gw.pao1.sonic.net/">50.ae4.gw.pao1.sonic.net</a> 0.0% 10 2.5<br>
> 8.3 2.5 37.5 10.7<br>
> 5. AS8121 199.127.107.189 0.0% 10 3.8<br>
> 7.9 3.4 34.9 9.5<br>
> 6. AS8121 <a href="http://xe0-0-0-7.core2.sv8.layer42.net/">xe0-0-0-7.core2.sv8.layer42.net</a> 0.0% 10 3.9<br>
> 6.0 3.4 17.6 4.6<br>
> 7. AS16509 72.21.221.176 0.0% 10 4.6<br>
> 5.2 3.0 15.5 3.6<br>
> 8. AS??? 54.240.243.110 0.0% 10 98.6<br>
> 96.4 86.9 104.4 6.2<br>
> 9. AS??? 54.240.243.121 90.0% 10 85.7<br>
> 85.7 85.7 85.7 0.0<br>
> 10. AS16509 205.251.229.104 20.0% 10 87.5<br>
> 85.2 82.5 87.8 2.0<br>
> 11. AS??? 54.239.42.29 70.0% 10 85.0<br>
> 84.3 82.8 85.2 1.0<br>
> 12. AS16509 54.239.109.238 30.0% 10 87.4<br>
> 96.8 87.4 105.2 6.5<br>
> AS16509 54.239.110.16<br>
><br>
><br>
> -Grant<br>
<br>
</div>
</div>
</div>
</div>
> _______________________________________________<br>
> Outages mailing list<br>
> <a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
> <a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
</div>
</div>
<span style="color:#888888">-- <br>
</span>
<div>
<div>
<div>Mike Lyon</div>
<div>408-621-4826 </div>
<div><a href="mailto:mike.lyon@gmail.com" style="color:rgb( 17 , 85 , 204 )">mike.lyon@gmail.com</a>
</div>
<br>
<div><a href="http://www.linkedin.com/in/mlyon" name="15135e7f47e598fd_15127a71bc7333d5_15127a44cdcad299_151279b4ff05187c_UNIQUE_ID_SafeHtmlFilter_SafeHtmlFilter_SafeHtmlFilter_134f26c37324f9ff_webProfileURL" style="color:rgb( 0 , 102 , 153 );padding-right:0px;padding-left:0px;border-bottom-width:0px;padding-top:0px;margin-bottom:0px;padding-bottom:0px;border-top-width:0px;vertical-align:baseline;line-height:15px;outline-style:none;margin-right:0px;margin-left:0px;border-right-width:0px;font-family:'arial' , 'helvetica' , 'nimbus sans l' , sans-serif;margin-top:0px;border-left-width:0px">http://www.linkedin.com/in/mlyon</a><br>
</div>
<br>
<br>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div>
<div>
<div>Mike Lyon</div>
<div>408-621-4826 </div>
<div><a href="mailto:mike.lyon@gmail.com" style="color:rgb( 17 , 85 , 204 )">mike.lyon@gmail.com</a>
</div>
<br>
<div><a href="http://www.linkedin.com/in/mlyon" name="15135e7f47e598fd_15127a71bc7333d5_UNIQUE_ID_SafeHtmlFilter_SafeHtmlFilter_SafeHtmlFilter_134f26c37324f9ff_webProfileURL" style="color:rgb( 0 , 102 , 153 );padding-right:0px;padding-left:0px;border-bottom-width:0px;padding-top:0px;margin-bottom:0px;padding-bottom:0px;border-top-width:0px;vertical-align:baseline;line-height:15px;outline-style:none;margin-right:0px;margin-left:0px;border-right-width:0px;font-family:'arial' , 'helvetica' , 'nimbus sans l' , sans-serif;margin-top:0px;border-left-width:0px">http://www.linkedin.com/in/mlyon</a><br>
</div>
<br>
<br>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a><br>
</div>
</div>
</div>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a><br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
</body>
</html>