<span style="font-family: Arial; font-size: 13px;"><div><div></div><div dir="auto" style="">Here is a more detailed analysis of what happened:<div><span class="Apple-tab-span" style="white-space:pre;"> </span><a href="https://arstechnica.com/information-technology/2018/04/suspicious-event-hijacks-amazon-traffic-for-2-hours-steals-cryptocurrency/" target="_blank">https://arstechnica.com/information-technology/2018/04/suspicious-event-hijacks-amazon-traffic-for-2-hours-steals-cryptocurrency/</a></div><div><br></div><div><br><div class="AppleOriginalContents" style="direction:ltr;"><blockquote><div>On Apr 24, 2018, at 14:19 , Ryan McGinnis via Outages <outages@outages.org> wrote:</div><br class="Apple-interchange-newline"><div><div dir="ltr" class=""><span style="color:#222;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;background-color:#fff;float:none;display:inline;" class="">I suspect this was related to this issue (via ycombinator hacker news): </span><div style="color:#222;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;background-color:#fff;" class=""><br class=""></div><div style="color:#222;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;background-color:#fff;" class=""><a style="color:#15c;" class="" target="_blank" href="https://doublepulsar.com/hijack-of-amazons-internet-domain-service-used-to-reroute-web-traffic-for-two-hours-unnoticed-3a6f0dda6a6f" onclick="window.open('https://doublepulsar.com/hijack-of-amazons-internet-domain-service-used-to-reroute-web-traffic-for-two-hours-unnoticed-3a6f0dda6a6f');return false;">https://doublepulsar.com/hijack-of-amazons-internet-domain-service-used-to-reroute-web-traffic-for-two-hours-unnoticed-3a6f0dda6a6f</a></div><br class=""></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Tue, Apr 24, 2018 at 8:51 AM, Zach Hanna via Outages <span dir="ltr" class=""><<a class="">outages@outages.org</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="border-left:1px #ccc solid;padding-left:1ex;margin:0 0 0 .8ex;"><div class=""><div dir="auto" class="">Resolved here too..</div><div class=""><div class="h5"><br class=""><div class="gmail_quote"><div class="">On Tue, Apr 24, 2018 at 7:30 AM Phil Lavin via Outages <<a class="">outages@outages.org</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="border-left:1px #ccc solid;padding-left:1ex;margin:0 0 0 .8ex;">
<div class="" lang="EN-GB">
<div class="m_7432672595941775700m_4697775517823711469WordSection1"><p class="MsoNormal" style="margin-bottom:12pt;"><span class="">Those prefixes had been withdrawn now – traffic is flowing correctly again for us. If that was the cause, I suspect things are back to rights for everyone now?<u class=""></u><u class=""></u></span></p><p class="MsoNormal" style="margin-bottom:12pt;"><span class="">One wonders why HE doesn’t apply filters on a peer with 20 legit prefixes…<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span class=""><u class=""></u> <u class=""></u></span></p>
<div class="">
<div style="border:none;border-top:solid #e1e1e1 1pt;padding:3pt 0 0;" class=""><p class="MsoNormal"><b class=""><span class="" lang="EN-US">From:</span></b><span class="" lang="EN-US"> Outages <<a class="">outages-bounces@outages.org</a>>
<b class="">On Behalf Of </b>Joseph B via Outages<br class="">
<b class="">Sent:</b> 24 April 2018 13:56</span></p></div></div></div></div><div class="" lang="EN-GB"><div class="m_7432672595941775700m_4697775517823711469WordSection1"><div class=""><div style="border:none;border-top:solid #e1e1e1 1pt;padding:3pt 0 0;" class=""><p class="MsoNormal"><span class="" lang="EN-US"><br class="">
<b class="">To:</b> <a class="">outages@outages.org</a><br class="">
<b class="">Subject:</b> Re: [outages] Google 8.8.8.8 Resolution of Route53 domains<u class=""></u><u class=""></u></span></p></div></div></div></div><div class="" lang="EN-GB"><div class="m_7432672595941775700m_4697775517823711469WordSection1"><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
<div class=""><p class="MsoNormal"><span style="" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class="">Tue Apr 24 11:05:41 UTC onwards one of Hurricane Electric's peers AS1<span class="m_7432672595941775700m_4697775517823711469s1">0297</span> started advertising the following subnets via HE.<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class="">205.251.192.0<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class="">205.251.193.0<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class="">205.251.195.0<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class="">205.251.197.0<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class="">205.251.199.0<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class="">These are all Amazon subnets, usually originated as part of /23s and seemingly host a fair bit of AWS Route53.<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class="">If you (or your DNS resolver) are a HE transit customer you will be impacted the most.<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class="">Cheers,<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class="">Joseph<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
</div>
<div class=""><p class="MsoNormal">On Tue, Apr 24, 2018, at 9:50 PM, Phil Lavin via Outages wrote:<u class=""></u><u class=""></u></p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt;" class="">
<div class=""><p style="margin-bottom:12pt;" class="">
<span class="m_7432672595941775700m_4697775517823711469size">This doesn’t feel right, though I’ll admit I’ve never checked before. Our only route to <a class="" target="_blank" href="http://ns-163.awsdns-20.com/" onclick="window.open('http://ns-163.awsdns-20.com/');return false;">ns-163.awsdns-20.com</a> (205.251.192.163) is through HE:</span><u class=""></u><u class=""></u></p><p style="margin-bottom:12pt;" class="">
<span class="m_7432672595941775700m_4697775517823711469font"><span style="" class="">inet.0: 757581 destinations, 2107440 routes (757301 active, 0 holddown, 522 hidden)</span></span><u class=""></u><u class=""></u></p><p style="margin-bottom:12pt;" class="">
<span class="m_7432672595941775700m_4697775517823711469font"><span style="" class="">+ = Active Route, - = Last Active, * = Both</span></span><u class=""></u><u class=""></u></p><p style="margin-bottom:12pt;" class="">
<span class="m_7432672595941775700m_4697775517823711469font"><span style="" class=""><a class="" target="_blank" href="http://205.251.192.0/24" onclick="window.open('http://205.251.192.0/24');return false;">205.251.192.0/24</a> *[BGP/170] 01:12:08, localpref 70</span></span><u class=""></u><u class=""></u></p><p style="margin-bottom:12pt;" class="">
<span class="m_7432672595941775700m_4697775517823711469font"><span style="" class=""> AS path: 6939 10297 I, validation-state: unverified</span></span><u class=""></u><u class=""></u></p><p style="margin-bottom:12pt;" class="">
<span class="m_7432672595941775700m_4697775517823711469font"><span style="" class=""> > to 216.66.90.21 via ge-1/0/5.0</span></span><u class=""></u><u class=""></u></p><p style="margin-bottom:12pt;" class="">
<span class="m_7432672595941775700m_4697775517823711469size">AS10297 is eNET inc. Is this expected?</span><u class=""></u><u class=""></u></p><p style="margin-bottom:12pt;" class="">
<span class="m_7432672595941775700m_4697775517823711469size"> </span><u class=""></u><u class=""></u></p>
<div class="">
<div style="border:none;border-top:solid #e1e1e1 1pt;border-right-width:initial;border-bottom-width:initial;border-left-width:initial;border-right-color:initial;border-bottom-color:initial;border-left-color:initial;padding:3pt 0 0;" class=""><div style="margin:0 0 .0001pt;" class=""><span class="m_7432672595941775700m_4697775517823711469size"><b class=""><span class="" lang="EN-US">From:</span></b></span><span class="m_7432672595941775700m_4697775517823711469size"><span class="" lang="EN-US"> Outages <<a class="">outages-bounces@outages.org</a>>
<b class="">On Behalf Of </b>Phil Lavin via Outages</span></span><span class="" lang="EN-US"><br class="">
<span class="m_7432672595941775700m_4697775517823711469size"><b class="">Sent:</b> 24 April 2018 13:04</span><br class="">
<span class="m_7432672595941775700m_4697775517823711469size"><b class="">To:</b> <a class="">outages@outages.org</a></span><br class="">
<span class="m_7432672595941775700m_4697775517823711469size"><b class="">Subject:</b> Re: [outages] Google 8.8.8.8 Resolution of Route53 domains</span></span><u class=""></u><u class=""></u></div>
</div>
</div><div style="margin:0 0 .0001pt;" class=""><span class="m_7432672595941775700m_4697775517823711469size"> </span><u class=""></u><u class=""></u></div><p style="margin-bottom:12pt;" class="">
<span class="m_7432672595941775700m_4697775517823711469size">Looks more specific to AWS than it does to Google+AWS. Can’t resolve against some of AWS’s NS directly:</span><u class=""></u><u class=""></u></p><p style="margin-bottom:12pt;" class="">
<span class="m_7432672595941775700m_4697775517823711469size"><span style="font-size:10pt;font-family:'Courier New';" class="">phil@phil-debian:~$ dig <a class="" target="_blank" href="http://cloudcall.com/" onclick="window.open('http://cloudcall.com/');return false;">cloudcall.com</a> IN A @<a class="" target="_blank" href="http://ns-163.awsdns-20.com/" onclick="window.open('http://ns-163.awsdns-20.com/');return false;">ns-163.awsdns-20.com</a></span></span><u class=""></u><u class=""></u></p><p style="margin-bottom:12pt;" class="">
<span class="m_7432672595941775700m_4697775517823711469size"><span style="font-size:10pt;font-family:'Courier New';" class="">; <<>> DiG 9.10.3-P4-Debian <<>> <a class="" target="_blank" href="http://cloudcall.com/" onclick="window.open('http://cloudcall.com/');return false;">cloudcall.com</a> IN A @<a class="" target="_blank" href="http://ns-163.awsdns-20.com/" onclick="window.open('http://ns-163.awsdns-20.com/');return false;">ns-163.awsdns-20.com</a></span></span><u class=""></u><u class=""></u></p><p style="margin-bottom:12pt;" class="">
<span class="m_7432672595941775700m_4697775517823711469size"><span style="font-size:10pt;font-family:'Courier New';" class="">;; global options: +cmd</span></span><u class=""></u><u class=""></u></p><p style="margin-bottom:12pt;" class="">
<span class="m_7432672595941775700m_4697775517823711469size"><span style="font-size:10pt;font-family:'Courier New';" class="">;; connection timed out; no servers could be reached</span></span><u class=""></u><u class=""></u></p><div style="margin:0 0 .0001pt;" class=""><span class="m_7432672595941775700m_4697775517823711469size"> </span><u class=""></u><u class=""></u></div>
<div class="">
<div style="border:none;border-top:solid #e1e1e1 1pt;border-right-width:initial;border-bottom-width:initial;border-left-width:initial;border-right-color:initial;border-bottom-color:initial;border-left-color:initial;padding:3pt 0 0;" class=""><div style="margin:0 0 .0001pt;" class=""><span class="m_7432672595941775700m_4697775517823711469size"><b class=""><span class="" lang="EN-US">From:</span></b></span><span class="m_7432672595941775700m_4697775517823711469size"><span class="" lang="EN-US"> Outages <<a class=""><span style="color:#0563c1;" class="">outages-bounces@outages.org</span></a>>
<b class="">On Behalf Of </b>Phil Lavin via Outages</span></span><span class="" lang="EN-US"><br class="">
<span class="m_7432672595941775700m_4697775517823711469size"><b class="">Sent:</b> 24 April 2018 12:56</span><br class="">
<span class="m_7432672595941775700m_4697775517823711469size"><b class="">To:</b> <a class=""><span style="color:#0563c1;" class="">outages@outages.org</span></a></span><br class="">
<span class="m_7432672595941775700m_4697775517823711469size"><b class="">Subject:</b> Re: [outages] Google 8.8.8.8 Resolution of Route53 domains</span></span><u class=""></u><u class=""></u></div>
</div>
</div><div style="margin:0 0 .0001pt;" class=""><span class="m_7432672595941775700m_4697775517823711469size"> </span><u class=""></u><u class=""></u></div><p style="margin-bottom:12pt;" class="">
<span class="m_7432672595941775700m_4697775517823711469size">Yeh. Still digging into it.</span><u class=""></u><u class=""></u></p><div style="margin:0 0 .0001pt;" class=""><span class="m_7432672595941775700m_4697775517823711469size"> </span><u class=""></u><u class=""></u></div><div style="margin:0 0 .0001pt;" class=""><span class="m_7432672595941775700m_4697775517823711469size"><b class=""><span class="" lang="EN-US">From:</span></b></span><span class="m_7432672595941775700m_4697775517823711469size"><span class="" lang="EN-US"> Outages <<a class=""><span style="color:#0563c1;" class="">outages-bounces@outages.org</span></a>>
<b class="">On Behalf Of </b>Zach Hanna via Outages</span></span><span class="" lang="EN-US"><br class="">
<span class="m_7432672595941775700m_4697775517823711469size"><b class="">Sent:</b> 24 April 2018 12:54</span><br class="">
<span class="m_7432672595941775700m_4697775517823711469size"><b class="">To:</b> <a class=""><span style="color:#0563c1;" class="">outages@outages.org</span></a></span><br class="">
<span class="m_7432672595941775700m_4697775517823711469size"><b class="">Subject:</b> [outages] Google 8.8.8.8 Resolution of Route53 domains</span></span><u class=""></u><u class=""></u></div><div style="margin:0 0 .0001pt;" class=""><span class="m_7432672595941775700m_4697775517823711469size"> </span><u class=""></u><u class=""></u></div>
<div class=""><div style="margin:0 0 .0001pt;" class=""><span class="m_7432672595941775700m_4697775517823711469size">Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?</span><u class=""></u><u class=""></u></div>
</div>
</div>
<div class=""><p class="MsoNormal"><u class="">_______________________________________________</u><u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal">Outages mailing list<u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal"><a class=""><span style="color:#0563c1;" class="">Outages@outages.org</span></a><u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal"><a class="" target="_blank" href="https://puck.nether.net/mailman/listinfo/outages" onclick="window.open('https://puck.nether.net/mailman/listinfo/outages');return false;"><span style="color:#0563c1;" class="">https://puck.nether.net/mailman/listinfo/outages</span></a><u class=""></u><u class=""></u></p>
</div>
</blockquote>
<div class=""><p class="MsoNormal"><span style="" class=""><u class=""></u> <u class=""></u></span></p>
</div>
</div></div>
_______________________________________________<br class="">
Outages mailing list<br class="">
<a class="">Outages@outages.org</a><br class="">
<a class="" target="_blank" href="https://puck.nether.net/mailman/listinfo/outages" onclick="window.open('https://puck.nether.net/mailman/listinfo/outages');return false;">https://puck.nether.net/mailman/listinfo/outages</a><br class="">
</blockquote></div></div></div></div>
<br class="">_______________________________________________<br class="">
Outages mailing list<br class="">
<a class="">Outages@outages.org</a><br class="">
<a class="" target="_blank" href="https://puck.nether.net/mailman/listinfo/outages" onclick="window.open('https://puck.nether.net/mailman/listinfo/outages');return false;">https://puck.nether.net/mailman/listinfo/outages</a><br class="">
<br class=""></blockquote></div><br class=""><br class="" clear="all"><div class=""><br class=""></div>-- <br class=""><div class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><span style="font-size:12.8px;" class="">-Ryan McGinnis</span><br style="font-size:12.8px;" class=""><span style="font-size:12.8px;" class="">Platte Valley Communications</span><br style="font-size:12.8px;" class=""><a style="color:#15c;font-size:12.8px;" class="">308-237-9512</a><br style="font-size:12.8px;" class=""><span style="font-size:12.8px;" class="">PGP: 62E39BC1</span><br class=""></div><div class=""><blockquote style="font-size:12.8px;" class=""><span class=""></span></blockquote></div></div></div></div></div>
</div>
_______________________________________________<br class="">Outages mailing list<br class="">Outages@outages.org<br class=""><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br class=""></div></blockquote></div><br></div></div></div></span>