[outages] Comcast DNS broken?

[Sajal Kayan]

http://arstechnica.com/business/2015/06/comcast-issuing-5-credits-after-internet-outage-caused-by-dns-failure/

What happened

A piece of hardware in our backbone network failed yesterday morning.
Hardware issues of this sort are fairly commonplace and redundancy in our
network addressed any immediate concern.

Most backbone networks, including Comcast’s, are designed to heal
themselves and route traffic along alternate paths much like a detour would
route automobile traffic around a street closed for construction.

One type of traffic that gets automatically rerouted is Domain Name System
(DNS) traffic. Unfortunately, some of that traffic shifted in an unexpected
way and overloaded local DNS server capacity causing many customers to
experience service interruptions.

What we are doing about it

Since the incident, we have been working non-stop to fully understand the
problem and bring additional DNS capacity online in the affected areas, so
we can prevent it from happening again.


On Tue, Jun 2, 2015 at 11:00 AM Sajal Kayan <sajal83 at gmail.com> wrote:

> Seems to work now.... only occasionally dropping queries...
>
> On Mon, Jun 1, 2015 at 8:12 PM, Frank Bulk <frnkblk at iname.com> wrote:
>
>> I don’t see anything on dns.comcast.net.
>>
>>
>>
>> I will forward to my Comcast contacts.
>>
>>
>>
>> Frank
>>
>>
>>
>> *From:* Outages [mailto:outages-bounces at outages.org] *On Behalf Of *Sajal
>> Kayan via Outages
>> *Sent:* Monday, June 01, 2015 10:06 PM
>> *To:* OUTAGES Mailing List
>> *Subject:* [outages] Comcast DNS broken?
>>
>>
>>
>> Hi,
>>
>>
>>
>> Comcast DNS servers (75.75.75.75, 75.75.76.76) seem to be down. Web pages
>> were suck at dns resolution phase, switching to Google DNS solved the
>> problem immediately.
>>
>>
>>
>> Tested from San Francisco.
>>
>>
>>
>> sajal at sajal-lappy:~$ mtr --report 75.75.75.75
>>
>> Start: Mon Jun  1 20:03:02 2015
>>
>> HOST: sajal-lappy                 Loss%   Snt   Last   Avg  Best  Wrst
>> StDev
>>
>>   1.|-- 10.0.1.1                   0.0%    10    1.2   3.1   1.0   5.9
>> 1.2
>>
>>   2.|-- 50.174.142.1               0.0%    10   15.9  20.7  14.6  49.7
>>  10.4
>>
>>   3.|-- te-0-7-0-5-sur03.sffolsom  0.0%    10   16.9  24.1  16.4  54.6
>>  11.6
>>
>>   4.|-- be-209-ar01.santaclara.ca  0.0%    10   18.2  33.4  17.0 146.1
>>  39.9
>>
>>   5.|-- he-0-14-0-0-ar01.sfsutro.  0.0%    10   22.4  26.9  20.0  42.9
>> 6.8
>>
>>   6.|-- te-9-4-ur01-d.sanjose.ca.  0.0%    10   22.9  23.1  20.6  28.4
>> 2.0
>>
>>   7.|-- te-8-2-ur01-d.sanjose.ca.  0.0%    10   21.5  23.4  20.6  29.2
>> 3.1
>>
>>   8.|-- cdns01.comcast.net         0.0%    10   32.2  47.3  20.5  89.7
>>  29.0
>>
>> sajal at sajal-lappy:~$ dig google.com @75.75.75.75
>>
>>
>>
>> ; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> google.com @75.75.75.75
>>
>> ;; global options: +cmd
>>
>> ;; connection timed out; no servers could be reached
>>
>> sajal at sajal-lappy:~$
>>
>>
>>
>>
>>
>> https://pulse.turbobytes.com/results/556d1b32ecbe4023eb0000ab/ - Using
>> our tool we see timeout from Seattle and Mountain View. Is this a national
>> level outage or something?
>>
>>
>>
>> -Sajal
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20150605/685d3c92/attachment.htm>