<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal>According to CableOne’s Twitter posting at 11:22 pm Central services started restoring:<o:p></o:p></p><p class=MsoNormal style='text-indent:.5in'><a href="https://twitter.com/CableONEBiz/status/845491069898784771">https://twitter.com/CableONEBiz/status/845491069898784771</a><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Checking the same traceroute this morning, I now see a Phoenix hop and the same 4.28.80.134, which suggests that the device with the IP of 4.28.80.134 is in Phoenix, which further indicates that the issue last night was some kind of network related failure. That CableONE asked customers <i>not</i> to reboot their modems suggests that access to their cable modem provision systems was also affected, and they didn’t want those systems to be slammed as soon as connectivity was restored.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>BTW, if you run a national network, don’t put your authoritative name servers behind the same network point of failure. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>root@nagios:/tmp# tcptraceroute ns1.cableone.net<o:p></o:p></p><p class=MsoNormal>Selected device eth0.3, address 96.31.0.5, port 32877 for outgoing packets<o:p></o:p></p><p class=MsoNormal>Tracing the path to ns1.cableone.net (24.116.0.206) on TCP port 80 (www), 30 hops max<o:p></o:p></p><p class=MsoNormal> 1 router-core-inside.mtcnet.net (96.31.0.254) 0.247 ms 0.341 ms 0.248 ms<o:p></o:p></p><p class=MsoNormal> 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.273 ms 0.175 ms 0.150 ms<o:p></o:p></p><p class=MsoNormal> 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 1.474 ms 1.411 ms 1.411 ms<o:p></o:p></p><p class=MsoNormal> 4 ins-db4-te-0-6-0-4.219.desm.netins.net (167.142.60.157) 7.770 ms 7.526 ms 7.715 ms<o:p></o:p></p><p class=MsoNormal> 5 ins-dc1-po200.desm.netins.net (167.142.67.193) 8.737 ms 8.812 ms 7.557 ms<o:p></o:p></p><p class=MsoNormal> 6 216.55.25.121 18.964 ms 18.983 ms 18.950 ms<o:p></o:p></p><p class=MsoNormal> 7 216.156.16.186.ptr.us.xo.net (216.156.16.186) 24.925 ms 24.917 ms 24.914 ms<o:p></o:p></p><p class=MsoNormal> 8 207.88.12.162.ptr.us.xo.net (207.88.12.162) 25.259 ms 26.128 ms 24.827 ms<o:p></o:p></p><p class=MsoNormal> 9 207.88.13.135.ptr.us.xo.net (207.88.13.135) 25.002 ms 24.706 ms 24.686 ms<o:p></o:p></p><p class=MsoNormal>10 lag-13.ear4.Chicago2.Level3.net (4.68.127.245) 25.165 ms 25.123 ms 32.023 ms<o:p></o:p></p><p class=MsoNormal>11 * * ae-1-11.bear1.Phoenix1.Level3.net (4.69.210.157) 241.215 ms<o:p></o:p></p><p class=MsoNormal>12 4.28.80.134 67.458 ms 67.522 ms 67.398 ms<o:p></o:p></p><p class=MsoNormal>13 * * *<o:p></o:p></p><p class=MsoNormal>14 ns1.cableone.net (24.116.0.206) [closed] 67.536 ms 67.638 ms 67.521 ms<o:p></o:p></p><p class=MsoNormal>root@nagios:/tmp#<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>root@nagios:/tmp# tcptraceroute ns2.cableone.net<o:p></o:p></p><p class=MsoNormal>Selected device eth0.3, address 96.31.0.5, port 35762 for outgoing packets<o:p></o:p></p><p class=MsoNormal>Tracing the path to ns2.cableone.net (24.119.5.34) on TCP port 80 (www), 30 hops max<o:p></o:p></p><p class=MsoNormal> 1 router-core-inside.mtcnet.net (96.31.0.254) 0.260 ms 0.208 ms 0.269 ms<o:p></o:p></p><p class=MsoNormal> 2 sxct.spnc.mtcnet.net (167.142.156.194) 10.723 ms 0.196 ms 0.160 ms<o:p></o:p></p><p class=MsoNormal> 3 premier.spnc-mlx.fbnt.netins.net (173.215.60.1) 4.862 ms 4.790 ms 15.821 ms<o:p></o:p></p><p class=MsoNormal> 4 ins-kb1-te-12-2-3031.kmrr.netins.net (167.142.64.253) 8.509 ms 8.492 ms 8.366 ms<o:p></o:p></p><p class=MsoNormal> 5 ins-kc2-et-po101.kmrr.netins.net (167.142.67.49) 8.499 ms 8.654 ms 8.679 ms<o:p></o:p></p><p class=MsoNormal> 6 10ge7-3.core1.oma1.he.net (184.105.18.169) 19.224 ms 21.610 ms 25.048 ms<o:p></o:p></p><p class=MsoNormal> 7 10ge12-1.core1.blp1.he.net (184.105.64.98) 20.218 ms 20.544 ms 13.523 ms<o:p></o:p></p><p class=MsoNormal> 8 10ge5-9.core1.msp1.he.net (184.105.64.93) 18.398 ms 23.572 ms 23.583 ms<o:p></o:p></p><p class=MsoNormal> 9 100ge13-1.core2.chi1.he.net (184.105.223.177) 30.769 ms 21.350 ms 21.371 ms<o:p></o:p></p><p class=MsoNormal>10 level3-as3356.100gigabitethernet12-1.core2.chi1.he.net (184.105.36.158) 26.706 ms 26.758 ms 26.646 ms<o:p></o:p></p><p class=MsoNormal>11 * * *<o:p></o:p></p><p class=MsoNormal>12 4.28.80.134 69.043 ms 69.020 ms 69.054 ms<o:p></o:p></p><p class=MsoNormal>13 * * *<o:p></o:p></p><p class=MsoNormal>14 ns2.cableone.net (24.119.5.34) [closed] 69.209 ms 69.115 ms 69.115 ms<o:p></o:p></p><p class=MsoNormal>root@nagios:/tmp#<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Frank<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> Outages-discussion [mailto:outages-discussion-bounces@outages.org] <b>On Behalf Of </b>frnkblk@iname.com<br><b>Sent:</b> Friday, March 24, 2017 9:46 PM<br><b>To:</b> outages-discussion@outages.org<br><b>Subject:</b> Re: [Outages-discussion] [outages] CableOne DNS issue?<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Traceroutes to the zone’s A record [24.116.124.161] and ns1.cableone.net [24.116.0.206] seem to get stuck somewhere in Chicago … 4.28.80.134 (Level 3 IP) is the last visible hop.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>10 level3-as3356.100gigabitethernet12-1.core2.chi1.he.net (184.105.36.158) 31.519 ms 27.337 ms 26.194 ms<o:p></o:p></p><p class=MsoNormal>11 * * *<o:p></o:p></p><p class=MsoNormal>12 4.28.80.134 73.404 ms 68.450 ms 68.521 ms<o:p></o:p></p><p class=MsoNormal>13 * * *<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>10 lag-13.ear4.Chicago2.Level3.net (4.68.127.245) 25.255 ms 25.331 ms 26.475 ms<o:p></o:p></p><p class=MsoNormal>11 * * *<o:p></o:p></p><p class=MsoNormal>12 4.28.80.134 74.048 ms 67.425 ms 67.541 ms<o:p></o:p></p><p class=MsoNormal>13 * * *<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Frank<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> Outages [<a href="mailto:outages-bounces@outages.org">mailto:outages-bounces@outages.org</a>] <b>On Behalf Of </b>frnkblk--- via Outages<br><b>Sent:</b> Friday, March 24, 2017 9:27 PM<br><b>To:</b> <a href="mailto:outages@outages.org">outages@outages.org</a><br><b>Subject:</b> [outages] CableOne DNS issue?<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Was just alerted that our outbound email queues have a few messages to cableone.net queued up, because there’s no MX record (even Google and OpenDNS don’t have an MX record cached).<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>DNSInspect (<a href="http://www.dnsinspect.com/cableone.net/10059114">http://www.dnsinspect.com/cableone.net/10059114</a>) is showing that their two nameservers aren’t resolving their A record.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>DownDetector is reporting issues: <a href="http://downdetector.com/status/cable-one">http://downdetector.com/status/cable-one</a><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>There’s also few posts on Twitter, too: <o:p></o:p></p><p class=MsoNormal><a href="https://twitter.com/lucky_boym/status/845441576973295616">https://twitter.com/lucky_boym/status/845441576973295616</a><o:p></o:p></p><p class=MsoNormal><a href="https://twitter.com/AnonySerb/status/845441954599092224">https://twitter.com/AnonySerb/status/845441954599092224</a><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>And just now posted, from CableONE:<o:p></o:p></p><p class=MsoNormal><a href="https://twitter.com/cableONE/status/845460682560688128">https://twitter.com/cableONE/status/845460682560688128</a><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Frank<o:p></o:p></p></div></body></html>