[outages] comcast dns issues today?

Wed Jul 28 14:21:59 EDT 2010

relayed to lead at Comcast.  Thanks, -ren

On Wed, Jul 28, 2010 at 1:58 PM, Robin Pimentel <robin.pimentel at gmail.com>wrote:

> I agree with Jared.  Looks like broken Anycast. It seems that the anycast
> loopback is properly configured on the machines, but I suspect the
> bind/nameserver config is borked and causing it to bind to the wrong
> interface.
>
> On Wed, Jul 28, 2010 at 10:37 AM, Jared Mauch <jared at puck.nether.net>wrote:
>
>> Yeah, what I'm seeing is if you query for AAAA the 77 or 72.130 ips, you
>> get back the ServFail from the .131 or .132 IP instead of the .130 IP.
>>
>> *sigh*
>>
>> Thanks for looking.
>>
>> - Jared
>>
>> On Jul 28, 2010, at 1:31 PM, Josh Luthman wrote:
>>
>> > Note that 68.87.72.130 took several seconds to respond.
>> >
>> > C:\Users\jluthman>dig www.thruway.ny.gov @68.87.77.130
>> >
>> > ; <<>> DiG 9.3.2 <<>> www.thruway.ny.gov @68.87.77.130
>> > ; (1 server found)
>> > ;; global options:  printcmd
>> > ;; Got answer:
>> > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1343
>> > ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
>> >
>> > ;; QUESTION SECTION:
>> > ;www.thruway.ny.gov.            IN      A
>> >
>> > ;; ANSWER SECTION:
>> > www.thruway.ny.gov.     38400   IN      CNAME   www.wip.thruway.ny.gov.
>> > www.wip.thruway.ny.gov. 30      IN      A       208.105.158.48
>> >
>> > ;; Query time: 274 msec
>> > ;; SERVER: 68.87.77.130#53(68.87.77.130)
>> > ;; WHEN: Wed Jul 28 13:30:53 2010
>> > ;; MSG SIZE  rcvd: 74
>> >
>> >
>> > C:\Users\jluthman>dig www.thruway.ny.gov @68.87.72.130
>> >
>> > ; <<>> DiG 9.3.2 <<>> www.thruway.ny.gov @68.87.72.130
>> > ; (1 server found)
>> > ;; global options:  printcmd
>> > ;; Got answer:
>> > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1500
>> > ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
>> >
>> > ;; QUESTION SECTION:
>> > ;www.thruway.ny.gov.            IN      A
>> >
>> > ;; ANSWER SECTION:
>> > www.thruway.ny.gov.     37594   IN      CNAME   www.wip.thruway.ny.gov.
>> > www.wip.thruway.ny.gov. 30      IN      A       161.11.122.48
>> >
>> > ;; Query time: 209 msec
>> > ;; SERVER: 68.87.72.130#53(68.87.72.130)
>> > ;; WHEN: Wed Jul 28 13:31:23 2010
>> > ;; MSG SIZE  rcvd: 74
>> >
>> > Josh Luthman
>> > Office: 937-552-2340
>> > Direct: 937-552-2343
>> > 1100 Wayne St
>> > Suite 1337
>> > Troy, OH 45373
>> >
>> >
>> >
>> > On Wed, Jul 28, 2010 at 1:29 PM, Jared Mauch <jared at puck.nether.net>
>> wrote:
>> >> I'm finding certain dns queries result in them responding from the
>> "wrong" address on their (apparently) anycasted load-balancer nodes.
>> >>
>> >> take for example: www.thruway.ny.gov
>> >>
>> >> - Jared
>> >>
>> >> On Jul 28, 2010, at 1:27 PM, Josh Luthman wrote:
>> >>
>> >>> Works for me.  Coming from 74.218.88.134
>> >>>
>> >>> C:\Users\jluthman>dig google.com @68.87.72.130
>> >>>
>> >>> ; <<>> DiG 9.3.2 <<>> google.com @68.87.72.130
>> >>> ; (1 server found)
>> >>> ;; global options:  printcmd
>> >>> ;; Got answer:
>> >>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1358
>> >>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0
>> >>>
>> >>> ;; QUESTION SECTION:
>> >>> ;google.com.                    IN      A
>> >>>
>> >>> ;; ANSWER SECTION:
>> >>> google.com.             178     IN      A       74.125.95.99
>> >>> google.com.             178     IN      A       74.125.95.147
>> >>> google.com.             178     IN      A       74.125.95.104
>> >>> google.com.             178     IN      A       74.125.95.106
>> >>> google.com.             178     IN      A       74.125.95.105
>> >>> google.com.             178     IN      A       74.125.95.103
>> >>>
>> >>> ;; Query time: 25 msec
>> >>> ;; SERVER: 68.87.72.130#53(68.87.72.130)
>> >>> ;; WHEN: Wed Jul 28 13:26:20 2010
>> >>> ;; MSG SIZE  rcvd: 124
>> >>>
>> >>>
>> >>> C:\Users\jluthman>dig google.com @68.87.7.130
>> >>> ^C
>> >>> C:\Users\jluthman>dig google.com @68.87.77.130
>> >>>
>> >>> ; <<>> DiG 9.3.2 <<>> google.com @68.87.77.130
>> >>> ; (1 server found)
>> >>> ;; global options:  printcmd
>> >>> ;; Got answer:
>> >>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1226
>> >>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0
>> >>>
>> >>> ;; QUESTION SECTION:
>> >>> ;google.com.                    IN      A
>> >>>
>> >>> ;; ANSWER SECTION:
>> >>> google.com.             218     IN      A       209.85.225.103
>> >>> google.com.             218     IN      A       209.85.225.99
>> >>> google.com.             218     IN      A       209.85.225.104
>> >>> google.com.             218     IN      A       209.85.225.147
>> >>> google.com.             218     IN      A       209.85.225.106
>> >>> google.com.             218     IN      A       209.85.225.105
>> >>>
>> >>> Josh Luthman
>> >>> Office: 937-552-2340
>> >>> Direct: 937-552-2343
>> >>> 1100 Wayne St
>> >>> Suite 1337
>> >>> Troy, OH 45373
>> >>>
>> >>>
>> >>> On Wed, Jul 28, 2010 at 12:53 PM, Jared Mauch <jared at puck.nether.net>
>> wrote:
>> >>>>
>> >>>> Anyone else notice issues with comcast dns servers today?  It seems
>> in the past 20 minutes their 'anycast' instances have started returning
>> packets from the wrong source ip when querying the following:
>> >>>>
>> >>>> nameserver 68.87.72.130
>> >>>> nameserver 68.87.77.130
>> >>>>
>> >>>> 2010/07/28 12:41:47| WARNING: Reply from unknown nameserver
>> 68.87.72.132:53
>> >>>> 2010/07/28 12:41:53| WARNING: Reply from unknown nameserver
>> 68.87.77.131:53 (retrying...6<=60)
>> >>>> 2010/07/28 12:41:58| WARNING: Reply from unknown nameserver
>> 68.87.72.131:53 (retrying...11<=60)
>> >>>> 2010/07/28 12:42:09| WARNING: Reply from unknown nameserver
>> 68.87.77.132:53 (retrying...22<=60)
>> >>>> 2010/07/28 12:42:17| WARNING: Reply from unknown nameserver
>> 68.87.72.132:53 (retrying...30<=60)
>> >>>> 2010/07/28 12:42:39| WARNING: Reply from unknown nameserver
>> 68.87.77.131:53 (retrying...52<=60)
>> >>>> 2010/07/28 12:42:57| WARNING: Reply from unknown nameserver
>> 68.87.72.131:53
>> >>>> 2010/07/28 12:43:21| WARNING: Reply from unknown nameserver
>> 68.87.72.132:53 (retrying...24<=60)
>> >>>> 2010/07/28 12:43:27| WARNING: Reply from unknown nameserver
>> 68.87.77.132:53 (retrying...30<=60)
>> >>>> 2010/07/28 12:43:30| WARNING: Reply from unknown nameserver
>> 68.87.72.132:53 (retrying...33<=60)
>> >>>> 2010/07/28 12:43:39| WARNING: Reply from unknown nameserver
>> 68.87.77.131:53 (retrying...42<=60)
>> >>>> 2010/07/28 12:43:42| WARNING: Reply from unknown nameserver
>> 68.87.77.131:53 (retrying...45<=60)
>> >>>> 2010/07/28 12:43:52| WARNING: Reply from unknown nameserver
>> 68.87.72.131:53 (retrying...55<=60)
>> >>>> 2010/07/28 12:44:11| WARNING: Reply from unknown nameserver
>> 68.87.77.131:53
>> >>>> 2010/07/28 12:44:31| WARNING: Reply from unknown nameserver
>> 68.87.72.131:53 (retrying...20<=60)
>> >>>> 2010/07/28 12:45:12| WARNING: Reply from unknown nameserver
>> 68.87.77.131:53
>> >>>> 2010/07/28 12:45:57| WARNING: Reply from unknown nameserver
>> 68.87.72.132:53 (retrying...45<=60)
>> >>>> 2010/07/28 12:46:02| WARNING: Reply from unknown nameserver
>> 68.87.77.132:53 (retrying...50<=60)
>> >>>> 2010/07/28 12:46:07| WARNING: Reply from unknown nameserver
>> 68.87.72.132:53 (retrying...55<=60)
>> >>>> 2010/07/28 12:46:16| WARNING: Reply from unknown nameserver
>> 68.87.77.132:53
>> >>>> 2010/07/28 12:46:27| WARNING: Reply from unknown nameserver
>> 68.87.72.131:53 (retrying...11<=60)
>> >>>> 2010/07/28 12:46:47| WARNING: Reply from unknown nameserver
>> 68.87.77.131:53 (retrying...31<=60)
>> >>>> 2010/07/28 12:46:49| WARNING: Reply from unknown nameserver
>> 68.87.72.131:53 (retrying...33<=60)
>> >>>> 2010/07/28 12:46:55| WARNING: Reply from unknown nameserver
>> 68.87.77.132:53 (retrying...39<=60)
>> >>>> 2010/07/28 12:47:00| WARNING: Reply from unknown nameserver
>> 68.87.72.132:53 (retrying...44<=60)
>> >>>> 2010/07/28 12:47:07| WARNING: Reply from unknown nameserver
>> 68.87.72.131:53 (retrying...51<=60)
>> >>>> 2010/07/28 12:47:10| WARNING: Reply from unknown nameserver
>> 68.87.77.132:53 (retrying...54<=60)
>> >>>> 2010/07/28 12:47:20| WARNING: Reply from unknown nameserver
>> 68.87.72.131:53
>> >>>> 2010/07/28 12:47:40| WARNING: Reply from unknown nameserver
>> 68.87.77.131:53 (retrying...20<=60)
>> >>>> 2010/07/28 12:47:47| WARNING: Reply from unknown nameserver
>> 68.87.77.131:53 (retrying...27<=60)
>> >>>> 2010/07/28 12:48:00| WARNING: Reply from unknown nameserver
>> 68.87.72.132:53 (retrying...40<=60)
>> >>>> 2010/07/28 12:48:40| WARNING: Reply from unknown nameserver
>> 68.87.77.131:53
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Outages mailing list
>> >>>> Outages at outages.org
>> >>>> https://puck.nether.net/mailman/listinfo/outages
>> >>
>> >>
>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100728/c599c130/attachment.htm>