<div dir="ltr"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">I suspect this was related to this issue (via ycombinator hacker news): </span><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><a href="https://doublepulsar.com/hijack-of-amazons-internet-domain-service-used-to-reroute-web-traffic-for-two-hours-unnoticed-3a6f0dda6a6f" target="_blank" style="color:rgb(17,85,204)">https://doublepulsar.com/<wbr>hijack-of-amazons-internet-<wbr>domain-service-used-to-<wbr>reroute-web-traffic-for-two-<wbr>hours-unnoticed-3a6f0dda6a6f</a></div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 24, 2018 at 8:51 AM, Zach Hanna via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div dir="auto">Resolved here too..</div><div><div class="h5"><br><div class="gmail_quote"><div>On Tue, Apr 24, 2018 at 7:30 AM Phil Lavin via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-GB" link="blue" vlink="purple">
<div class="m_7432672595941775700m_4697775517823711469WordSection1">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span>Those prefixes had been withdrawn now – traffic is flowing correctly again for us. If that was the cause, I suspect things are back to rights for everyone now?<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span>One wonders why HE doesn’t apply filters on a peer with 20 legit prefixes…<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Outages <<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>>
<b>On Behalf Of </b>Joseph B via Outages<br>
<b>Sent:</b> 24 April 2018 13:56</span></p></div></div></div></div><div lang="EN-GB" link="blue" vlink="purple"><div class="m_7432672595941775700m_4697775517823711469WordSection1"><div><div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><span lang="EN-US"><br>
<b>To:</b> <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] Google 8.8.8.8 Resolution of Route53 domains<u></u><u></u></span></p></div></div></div></div><div lang="EN-GB" link="blue" vlink="purple"><div class="m_7432672595941775700m_4697775517823711469WordSection1">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Tue Apr 24 11:05:41 UTC onwards one of Hurricane Electric's peers AS1<span class="m_7432672595941775700m_4697775517823711469s1">0297</span> started advertising the following subnets via HE.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">205.251.192.0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">205.251.193.0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">205.251.195.0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">205.251.197.0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">205.251.199.0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">These are all Amazon subnets, usually originated as part of /23s and seemingly host a fair bit of AWS Route53.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">If you (or your DNS resolver) are a HE transit customer you will be impacted the most.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Cheers,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Joseph<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">On Tue, Apr 24, 2018, at 9:50 PM, Phil Lavin via Outages wrote:<u></u><u></u></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p style="margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span class="m_7432672595941775700m_4697775517823711469size">This doesn’t feel right, though I’ll admit I’ve never checked before. Our only route to <a href="http://ns-163.awsdns-20.com" target="_blank">ns-163.awsdns-20.com</a> (205.251.192.163) is through HE:</span><u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span class="m_7432672595941775700m_4697775517823711469font"><span style="font-family:"Courier New"">inet.0: 757581 destinations, 2107440 routes (757301 active, 0 holddown, 522 hidden)</span></span><u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span class="m_7432672595941775700m_4697775517823711469font"><span style="font-family:"Courier New"">+ = Active Route, - = Last Active, * = Both</span></span><u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span class="m_7432672595941775700m_4697775517823711469font"><span style="font-family:"Courier New""><a href="http://205.251.192.0/24" target="_blank">205.251.192.0/24</a> *[BGP/170] 01:12:08, localpref 70</span></span><u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span class="m_7432672595941775700m_4697775517823711469font"><span style="font-family:"Courier New""> AS path: 6939 10297 I, validation-state: unverified</span></span><u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span class="m_7432672595941775700m_4697775517823711469font"><span style="font-family:"Courier New""> > to 216.66.90.21 via ge-1/0/5.0</span></span><u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span class="m_7432672595941775700m_4697775517823711469size">AS10297 is eNET inc. Is this expected?</span><u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span class="m_7432672595941775700m_4697775517823711469size"> </span><u></u><u></u></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm;border-right-width:initial;border-bottom-width:initial;border-left-width:initial;border-right-color:initial;border-bottom-color:initial;border-left-color:initial">
<p style="margin:0cm;margin-bottom:.0001pt"><span class="m_7432672595941775700m_4697775517823711469size"><b><span lang="EN-US">From:</span></b></span><span class="m_7432672595941775700m_4697775517823711469size"><span lang="EN-US"> Outages <<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>>
<b>On Behalf Of </b>Phil Lavin via Outages</span></span><span lang="EN-US"><br>
<span class="m_7432672595941775700m_4697775517823711469size"><b>Sent:</b> 24 April 2018 13:04</span><br>
<span class="m_7432672595941775700m_4697775517823711469size"><b>To:</b> <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a></span><br>
<span class="m_7432672595941775700m_4697775517823711469size"><b>Subject:</b> Re: [outages] Google 8.8.8.8 Resolution of Route53 domains</span></span><u></u><u></u></p>
</div>
</div>
<p style="margin:0cm;margin-bottom:.0001pt"><span class="m_7432672595941775700m_4697775517823711469size"> </span><u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span class="m_7432672595941775700m_4697775517823711469size">Looks more specific to AWS than it does to Google+AWS. Can’t resolve against some of AWS’s NS directly:</span><u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span class="m_7432672595941775700m_4697775517823711469size"><span style="font-size:10.0pt;font-family:"Courier New";color:black">phil@phil-debian:~$ dig <a href="http://cloudcall.com" target="_blank">cloudcall.com</a> IN A @<a href="http://ns-163.awsdns-20.com" target="_blank">ns-163.awsdns-20.com</a></span></span><u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span class="m_7432672595941775700m_4697775517823711469size"><span style="font-size:10.0pt;font-family:"Courier New";color:black">; <<>> DiG 9.10.3-P4-Debian <<>> <a href="http://cloudcall.com" target="_blank">cloudcall.com</a> IN A @<a href="http://ns-163.awsdns-20.com" target="_blank">ns-163.awsdns-20.com</a></span></span><u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span class="m_7432672595941775700m_4697775517823711469size"><span style="font-size:10.0pt;font-family:"Courier New";color:black">;; global options: +cmd</span></span><u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span class="m_7432672595941775700m_4697775517823711469size"><span style="font-size:10.0pt;font-family:"Courier New";color:black">;; connection timed out; no servers could be reached</span></span><u></u><u></u></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span class="m_7432672595941775700m_4697775517823711469size"> </span><u></u><u></u></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm;border-right-width:initial;border-bottom-width:initial;border-left-width:initial;border-right-color:initial;border-bottom-color:initial;border-left-color:initial">
<p style="margin:0cm;margin-bottom:.0001pt"><span class="m_7432672595941775700m_4697775517823711469size"><b><span lang="EN-US">From:</span></b></span><span class="m_7432672595941775700m_4697775517823711469size"><span lang="EN-US"> Outages <<a href="mailto:outages-bounces@outages.org" target="_blank"><span style="color:#0563c1">outages-bounces@outages.org</span></a>>
<b>On Behalf Of </b>Phil Lavin via Outages</span></span><span lang="EN-US"><br>
<span class="m_7432672595941775700m_4697775517823711469size"><b>Sent:</b> 24 April 2018 12:56</span><br>
<span class="m_7432672595941775700m_4697775517823711469size"><b>To:</b> <a href="mailto:outages@outages.org" target="_blank"><span style="color:#0563c1">outages@outages.org</span></a></span><br>
<span class="m_7432672595941775700m_4697775517823711469size"><b>Subject:</b> Re: [outages] Google 8.8.8.8 Resolution of Route53 domains</span></span><u></u><u></u></p>
</div>
</div>
<p style="margin:0cm;margin-bottom:.0001pt"><span class="m_7432672595941775700m_4697775517823711469size"> </span><u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span class="m_7432672595941775700m_4697775517823711469size">Yeh. Still digging into it.</span><u></u><u></u></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span class="m_7432672595941775700m_4697775517823711469size"> </span><u></u><u></u></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span class="m_7432672595941775700m_4697775517823711469size"><b><span lang="EN-US">From:</span></b></span><span class="m_7432672595941775700m_4697775517823711469size"><span lang="EN-US"> Outages <<a href="mailto:outages-bounces@outages.org" target="_blank"><span style="color:#0563c1">outages-bounces@outages.org</span></a>>
<b>On Behalf Of </b>Zach Hanna via Outages</span></span><span lang="EN-US"><br>
<span class="m_7432672595941775700m_4697775517823711469size"><b>Sent:</b> 24 April 2018 12:54</span><br>
<span class="m_7432672595941775700m_4697775517823711469size"><b>To:</b> <a href="mailto:outages@outages.org" target="_blank"><span style="color:#0563c1">outages@outages.org</span></a></span><br>
<span class="m_7432672595941775700m_4697775517823711469size"><b>Subject:</b> [outages] Google 8.8.8.8 Resolution of Route53 domains</span></span><u></u><u></u></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span class="m_7432672595941775700m_4697775517823711469size"> </span><u></u><u></u></p>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><span class="m_7432672595941775700m_4697775517823711469size">Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u>______________________________<wbr>_________________</u><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Outages mailing list<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="mailto:Outages@outages.org" target="_blank"><span style="color:#0563c1">Outages@outages.org</span></a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank"><span style="color:#0563c1">https://puck.nether.net/<wbr>mailman/listinfo/outages</span></a><u></u><u></u></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
</div>
</div></div>
______________________________<wbr>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/<wbr>mailman/listinfo/outages</a><br>
</blockquote></div></div></div></div>
<br>______________________________<wbr>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/<wbr>mailman/listinfo/outages</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><span style="font-size:12.8px">-Ryan McGinnis</span><br style="font-size:12.8px"><span style="font-size:12.8px">Platte Valley Communications</span><br style="font-size:12.8px"><a href="tel:308-237-9512" dir="ltr" style="color:rgb(17,85,204);font-size:12.8px" target="_blank">308-237-9512</a><br style="font-size:12.8px"><span style="font-size:12.8px">PGP: 62E39BC1</span><br></div><div><blockquote type="cite" style="font-size:12.8px"><span></span></blockquote></div></div></div></div></div>
</div>