<div dir="ltr"><a href="http://arstechnica.com/business/2015/06/comcast-issuing-5-credits-after-internet-outage-caused-by-dns-failure/">http://arstechnica.com/business/2015/06/comcast-issuing-5-credits-after-internet-outage-caused-by-dns-failure/</a><br><br><div>What happened</div><div><br></div><div>A piece of hardware in our backbone network failed yesterday morning. Hardware issues of this sort are fairly commonplace and redundancy in our network addressed any immediate concern.</div><div><br></div><div>Most backbone networks, including Comcast’s, are designed to heal themselves and route traffic along alternate paths much like a detour would route automobile traffic around a street closed for construction.</div><div><br></div><div>One type of traffic that gets automatically rerouted is Domain Name System (DNS) traffic. Unfortunately, some of that traffic shifted in an unexpected way and overloaded local DNS server capacity causing many customers to experience service interruptions.</div><div><br></div><div>What we are doing about it</div><div><br></div><div>Since the incident, we have been working non-stop to fully understand the problem and bring additional DNS capacity online in the affected areas, so we can prevent it from happening again.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jun 2, 2015 at 11:00 AM Sajal Kayan <<a href="mailto:sajal83@gmail.com">sajal83@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Seems to work now.... only occasionally dropping queries...</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 1, 2015 at 8:12 PM, Frank Bulk <span dir="ltr"><<a href="mailto:frnkblk@iname.com" target="_blank">frnkblk@iname.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I don’t see anything on <a href="http://dns.comcast.net" target="_blank">dns.comcast.net</a>.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I will forward to my Comcast contacts.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Frank<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Outages [mailto:<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>] <b>On Behalf Of </b>Sajal Kayan via Outages<br><b>Sent:</b> Monday, June 01, 2015 10:06 PM<br><b>To:</b> OUTAGES Mailing List<br><b>Subject:</b> [outages] Comcast DNS broken?<u></u><u></u></span></p><div><div><p class="MsoNormal"><u></u> <u></u></p><div><p class="MsoNormal">Hi,<u></u><u></u></p><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">Comcast DNS servers (75.75.75.75, 75.75.76.76) seem to be down. Web pages were suck at dns resolution phase, switching to Google DNS solved the problem immediately.<u></u><u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">Tested from San Francisco.<u></u><u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><div><p class="MsoNormal"><span style="font-family:"Courier New"">sajal@sajal-lappy:~$ mtr --report 75.75.75.75</span><u></u><u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New"">Start: Mon Jun 1 20:03:02 2015</span><u></u><u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New"">HOST: sajal-lappy Loss% Snt Last Avg Best Wrst StDev</span><u></u><u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New""> 1.|-- 10.0.1.1 0.0% 10 1.2 3.1 1.0 5.9 1.2</span><u></u><u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New""> 2.|-- 50.174.142.1 0.0% 10 15.9 20.7 14.6 49.7 10.4</span><u></u><u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New""> 3.|-- te-0-7-0-5-sur03.sffolsom 0.0% 10 16.9 24.1 16.4 54.6 11.6</span><u></u><u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New""> 4.|-- <a href="http://be-209-ar01.santaclara.ca" target="_blank">be-209-ar01.santaclara.ca</a> 0.0% 10 18.2 33.4 17.0 146.1 39.9</span><u></u><u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New""> 5.|-- he-0-14-0-0-ar01.sfsutro. 0.0% 10 22.4 26.9 20.0 42.9 6.8</span><u></u><u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New""> 6.|-- <a href="http://te-9-4-ur01-d.sanjose.ca" target="_blank">te-9-4-ur01-d.sanjose.ca</a>. 0.0% 10 22.9 23.1 20.6 28.4 2.0</span><u></u><u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New""> 7.|-- <a href="http://te-8-2-ur01-d.sanjose.ca" target="_blank">te-8-2-ur01-d.sanjose.ca</a>. 0.0% 10 21.5 23.4 20.6 29.2 3.1</span><u></u><u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New""> 8.|-- <a href="http://cdns01.comcast.net" target="_blank">cdns01.comcast.net</a> 0.0% 10 32.2 47.3 20.5 89.7 29.0</span><u></u><u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New"">sajal@sajal-lappy:~$ dig <a href="http://google.com" target="_blank">google.com</a> @<a href="http://75.75.75.75" target="_blank">75.75.75.75</a></span><u></u><u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New"">; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> <a href="http://google.com" target="_blank">google.com</a> @<a href="http://75.75.75.75" target="_blank">75.75.75.75</a></span><u></u><u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New"">;; global options: +cmd</span><u></u><u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New"">;; connection timed out; no servers could be reached</span><u></u><u></u></p></div><div><p class="MsoNormal"><span style="font-family:"Courier New"">sajal@sajal-lappy:~$ </span><u></u><u></u></p></div></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal"><a href="https://pulse.turbobytes.com/results/556d1b32ecbe4023eb0000ab/" target="_blank">https://pulse.turbobytes.com/results/556d1b32ecbe4023eb0000ab/</a> - Using our tool we see timeout from Seattle and Mountain View. Is this a national level outage or something?<u></u><u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">-Sajal <u></u><u></u></p></div></div></div></div></div></div></blockquote></div><br></div>
</blockquote></div>