<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Slack, which is hosted with Amazon is currently down.<br></div><div><br></div><div data-marker="__SIG_PRE__"><br></div><br><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>From: </b>"Grant Ridder via Outages" <outages@outages.org><br><b>To: </b>"Mike Lyon" <mike.lyon@gmail.com><br><b>Cc: </b>outages@outages.org, "Tomoc Field" <tomoc.field@sonic.com><br><b>Sent: </b>Monday, November 23, 2015 2:43:57 PM<br><b>Subject: </b>Re: [outages] packet loss to us-east-1<br></div><br><div data-marker="__QUOTED_TEXT__"><div dir="ltr">Seeing this issue again<br><div><div> 4. AS46375 <a href="http://50.ae4.gw.pao1.sonic.net" target="_blank">50.ae4.gw.pao1.sonic.net</a> 0.0% 10 15.5 23.2 2.9 89.3 28.6</div><div> 5. AS8121 199.127.107.189 0.0% 10 64.2 26.6 3.7 85.7 28.2</div><div> 6. AS8121 <a href="http://xe0-0-0-7.core2.sv8.layer42.net" target="_blank">xe0-0-0-7.core2.sv8.layer42.net</a> 0.0% 10 66.6 24.8 3.5 102.1 33.4</div><div> 7. AS16509 72.21.221.176 0.0% 10 64.2 26.9 3.0 99.1 32.4</div><div> 8. AS??? 54.240.243.50 0.0% 10 147.4 115.9 88.2 176.9 27.9</div><div> 9. AS??? 54.240.243.61 90.0% 10 78.4 78.4 78.4 78.4 0.0</div><div> 10. AS??? 54.239.42.38 10.0% 10 130.4 106.9 71.8 167.6 31.8</div><div> 11. AS16509 54.239.108.120 70.0% 10 178.9 125.2 90.0 178.9 47.2</div><div> 12. AS16509 54.239.111.235 20.0% 10 143.1 99.0 70.8 158.1 34.7</div><div> 13. AS16509 54.239.111.230 70.0% 10 89.5 93.9 88.7 103.5 8.3</div></div><br><div>-Grant</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 20, 2015 at 5:27 PM, Mike Lyon <span dir="ltr"><<a href="mailto:mike.lyon@gmail.com" target="_blank">mike.lyon@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hmmm,<br><div>It shows as if it is getting there but look at that jump from 15ms to 85ms on hops 7 and 8:</div><br><div><p class="MsoNormal">traceroute
<a href="http://amazon.com" target="_blank">amazon.com</a><br data-mce-bogus="1"></p>
<p class="MsoNormal">traceroute:
Warning: <a href="http://amazon.com" target="_blank">amazon.com</a> has multiple addresses;
using 176.32.103.205</p>
<p class="MsoNormal">traceroute
to <a href="http://amazon.com" target="_blank">amazon.com</a> (176.32.103.205), 64 hops max, 52
byte packets</p>
<p class="MsoNormal"> 1
192.168.20.1 (192.168.20.1) 0.708 ms 0.395 ms 0.328 ms</p>
<p class="MsoNormal"> 2
162.250.208.141 (162.250.208.141) 0.671 ms 0.622 ms
0.588 ms</p>
<p class="MsoNormal"> 3
10.25.0.1 (10.25.0.1) 1.985 ms 4.165 ms 21.626 ms</p>
<p class="MsoNormal"> 4
10.20.0.1 (10.20.0.1) 7.852 ms 7.626 ms 5.214 ms</p>
<p class="MsoNormal"> 5
10.0.0.1 (10.0.0.1) 8.041 ms 17.839 ms 34.089 ms</p>
<p class="MsoNormal"> 6
69.36.226.193 (69.36.226.193) 5.477 ms 14.228 ms 20.203
ms</p>
<p class="MsoNormal"> 7
<a href="http://xe0-2-0-4.core4.scl.layer42.net" target="_blank">xe0-2-0-4.core4.scl.layer42.net</a>
(65.50.198.237) 15.153 ms 13.475 ms 8.706 ms</p>
<p class="MsoNormal"> 8
205.251.229.145 (205.251.229.145) 85.231 ms</p>
<p class="MsoNormal">
54.240.242.108 (54.240.242.108) 96.157 ms</p>
<p class="MsoNormal">
205.251.229.145 (205.251.229.145) 175.633 ms</p>
<p class="MsoNormal"> 9
54.240.242.141 (54.240.242.141) 100.007 ms 234.602 ms</p>
<p class="MsoNormal">
54.240.242.121 (54.240.242.121) 94.263 ms</p>
<p class="MsoNormal">10
54.239.109.102 (54.239.109.102) 110.966 ms</p>
<p class="MsoNormal">
54.239.109.136 (54.239.109.136) 123.322 ms</p>
<p class="MsoNormal">
54.239.110.50 (54.239.110.50) 105.029 ms</p>
<p class="MsoNormal">11
54.239.109.143 (54.239.109.143) 84.239 ms</p>
<p class="MsoNormal">
54.239.110.23 (54.239.110.23) 91.540 ms</p>
<p class="MsoNormal">
54.239.110.57 (54.239.110.57) 86.620 ms</p>
<p class="MsoNormal">12
54.239.109.143 (54.239.109.143) 91.382 ms</p>
<p class="MsoNormal">
205.251.245.168 (205.251.245.168) 87.930 ms</p>
<p class="MsoNormal">
54.239.109.143 (54.239.109.143) 76.879 ms</p>
<p class="MsoNormal">13
* 205.251.245.168 (205.251.245.168) 90.780 ms 96.386 ms</p></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 20, 2015 at 5:24 PM, Grant Ridder <span dir="ltr"><<a href="mailto:shortdudey123@gmail.com" target="_blank">shortdudey123@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Looks like they solved it by routing around layer42 and go via cogent. That in itself is a bad idea :/<br><div><div> 4. AS46375 <a href="http://50.ae4.gw.pao1.sonic.net" target="_blank">50.ae4.gw.pao1.sonic.net</a> 1.2% 81 3.5 12.0 2.6 91.0 16.2</div><div> 5. AS174 <a href="http://te0-0-0-15.ccr21.sjc04.atlas.cogentco.com" target="_blank">te0-0-0-15.ccr21.sjc04.atlas.cogentco.com</a> 6.2% 81 6.6 10.5 3.2 79.9 11.7</div><div> 6. AS174 <a href="http://be2016.ccr22.sfo01.atlas.cogentco.com" target="_blank">be2016.ccr22.sfo01.atlas.cogentco.com</a> 3.7% 81 15.3 14.6 4.2 98.9 16.3</div><div> 7. AS174 <a href="http://be2133.ccr22.mci01.atlas.cogentco.com" target="_blank">be2133.ccr22.mci01.atlas.cogentco.com</a> 4.9% 81 47.7 50.4 41.8 113.6 13.3</div><div> 8. AS174 <a href="http://be2157.ccr42.ord01.atlas.cogentco.com" target="_blank">be2157.ccr42.ord01.atlas.cogentco.com</a> 70.0% 81 54.1 67.2 53.9 140.2 21.3</div><div> 9. AS174 <a href="http://be2217.ccr41.ord03.atlas.cogentco.com" target="_blank">be2217.ccr41.ord03.atlas.cogentco.com</a> 33.3% 81 57.3 63.6 54.3 158.4 17.0</div><div>10. AS174 <a href="http://ntt.ord03.atlas.cogentco.com" target="_blank">ntt.ord03.atlas.cogentco.com</a> 66.2% 81 56.9 72.0 56.5 126.8 20.6</div><div>11. AS2914 <a href="http://ae-3.amazon.chcgil09.us.bb.gin.ntt.net" target="_blank">ae-3.amazon.chcgil09.us.bb.gin.ntt.net</a> 34.6% 81 61.5 61.8 55.9 89.7 8.1</div><span><span style="color: #888888;" data-mce-style="color: #888888;" color="#888888"><br></span></span></div><span><span style="color: #888888;" data-mce-style="color: #888888;" color="#888888"><div>-Grant</div></span></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon <span dir="ltr"><<a href="mailto:mike.lyon@gmail.com" target="_blank">mike.lyon@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Oh thank god! I was seeing it too but wasn't sure if was Layer42 or Amazon that was having the issue...<br><div>Thanks,</div><div>Mike</div><br></div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Saw that, thanks!<br><div>-Grant</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <span dir="ltr"><<a href="mailto:tomoc.field@sonic.com" target="_blank">tomoc.field@sonic.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. This was experienced via both layer42 and XO, but Amazon appears to have resolved their issues at this time.<br>
<br>
-Tomoc<div><div><br>
<div><div><br>
On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote:<br>
> Anyone else seeing packet log into us-east-1?<br>
><br>
> 54.240.243.121 (AS16509) is Amazon<br>
><br>
> MTR report excerpt<br>
> 4. AS46375 <a href="http://50.ae4.gw.pao1.sonic.net" rel="noreferrer" target="_blank">50.ae4.gw.pao1.sonic.net</a> 0.0% 10 2.5<br>
> <a href="tel:8.3%20%20%202.5%20%2037.5%20%2010.7" target="_blank">8.3 2.5 37.5 10.7</a><br>
> 5. AS8121 199.127.107.189 0.0% 10 3.8<br>
> 7.9 3.4 34.9 9.5<br>
> 6. AS8121 <a href="http://xe0-0-0-7.core2.sv8.layer42.net" rel="noreferrer" target="_blank">xe0-0-0-7.core2.sv8.layer42.net</a> 0.0% 10 3.9<br>
> 6.0 3.4 17.6 4.6<br>
> 7. AS16509 72.21.221.176 0.0% 10 4.6<br>
> 5.2 3.0 15.5 3.6<br>
> 8. AS??? 54.240.243.110 0.0% 10 98.6<br>
> 96.4 86.9 104.4 6.2<br>
> 9. AS??? 54.240.243.121 90.0% 10 85.7<br>
> 85.7 85.7 85.7 0.0<br>
> 10. AS16509 205.251.229.104 20.0% 10 87.5<br>
> 85.2 82.5 87.8 2.0<br>
> 11. AS??? 54.239.42.29 70.0% 10 85.0<br>
> 84.3 82.8 85.2 1.0<br>
> 12. AS16509 54.239.109.238 30.0% 10 87.4<br>
> 96.8 87.4 105.2 6.5<br>
> AS16509 54.239.110.16<br>
><br>
><br>
> -Grant<br>
<br>
</div></div></div></div>> _______________________________________________<br>
> Outages mailing list<br>
> <a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
> <a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
</blockquote></div><br></div>
<br>_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br></blockquote></div><br><br clear="all"><br></div></div><span><span style="color: #888888;" data-mce-style="color: #888888;" color="#888888">-- <br><div><div><div>Mike Lyon</div><div><a href="tel:408-621-4826" target="_blank">408-621-4826</a>
</div><div><a href="mailto:mike.lyon@gmail.com" style="color:rgb(17,85,204)" target="_blank">mike.lyon@gmail.com</a>
</div><br><div><a href="http://www.linkedin.com/in/mlyon" title="View public profile" name="15127a71bc7333d5_15127a44cdcad299_151279b4ff05187c_UNIQUE_ID_SafeHtmlFilter_SafeHtmlFilter_SafeHtmlFilter_134f26c37324f9ff_webProfileURL" style="color:rgb(0,102,153);padding-right:0px;padding-left:0px;border-bottom-width:0px;padding-top:0px;outline-width:initial;border-style:initial;border-color:initial;margin-bottom:0px;padding-bottom:0px;border-top-width:0px;vertical-align:baseline;line-height:15px;outline-style:none;margin-right:0px;outline-color:initial;margin-left:0px;border-right-width:0px;font-family:Arial,Helvetica,'Nimbus Sans L',sans-serif;margin-top:0px;border-left-width:0px" target="_blank">http://www.linkedin.com/in/mlyon</a><br data-mce-bogus="1"></div><br><br><br></div></div>
</span></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div><div><div>Mike Lyon</div><div><a href="tel:408-621-4826" target="_blank">408-621-4826</a>
</div><div><a href="mailto:mike.lyon@gmail.com" style="color:rgb(17,85,204)" target="_blank">mike.lyon@gmail.com</a>
</div><br><div><a href="http://www.linkedin.com/in/mlyon" title="View public profile" name="15127a71bc7333d5_UNIQUE_ID_SafeHtmlFilter_SafeHtmlFilter_SafeHtmlFilter_134f26c37324f9ff_webProfileURL" style="color:rgb(0,102,153);padding-right:0px;padding-left:0px;border-bottom-width:0px;padding-top:0px;outline-width:initial;border-style:initial;border-color:initial;margin-bottom:0px;padding-bottom:0px;border-top-width:0px;vertical-align:baseline;line-height:15px;outline-style:none;margin-right:0px;outline-color:initial;margin-left:0px;border-right-width:0px;font-family:Arial,Helvetica,'Nimbus Sans L',sans-serif;margin-top:0px;border-left-width:0px" target="_blank">http://www.linkedin.com/in/mlyon</a><br data-mce-bogus="1"></div><br><br><br></div></div>
</div>
</div></div></blockquote></div><br></div>
<br>_______________________________________________<br>Outages mailing list<br>Outages@outages.org<br>https://puck.nether.net/mailman/listinfo/outages<br></div></div></body></html>