[outages] comcast dns issues today?

Wed Jul 28 13:58:51 EDT 2010

I agree with Jared.  Looks like broken Anycast. It seems that the anycast
loopback is properly configured on the machines, but I suspect the
bind/nameserver config is borked and causing it to bind to the wrong
interface.

On Wed, Jul 28, 2010 at 10:37 AM, Jared Mauch <jared at puck.nether.net> wrote:

> Yeah, what I'm seeing is if you query for AAAA the 77 or 72.130 ips, you
> get back the ServFail from the .131 or .132 IP instead of the .130 IP.
>
> *sigh*
>
> Thanks for looking.
>
> - Jared
>
> On Jul 28, 2010, at 1:31 PM, Josh Luthman wrote:
>
> > Note that 68.87.72.130 took several seconds to respond.
> >
> > C:\Users\jluthman>dig www.thruway.ny.gov @68.87.77.130
> >
> > ; <<>> DiG 9.3.2 <<>> www.thruway.ny.gov @68.87.77.130
> > ; (1 server found)
> > ;; global options:  printcmd
> > ;; Got answer:
> > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1343
> > ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
> >
> > ;; QUESTION SECTION:
> > ;www.thruway.ny.gov.            IN      A
> >
> > ;; ANSWER SECTION:
> > www.thruway.ny.gov.     38400   IN      CNAME   www.wip.thruway.ny.gov.
> > www.wip.thruway.ny.gov. 30      IN      A       208.105.158.48
> >
> > ;; Query time: 274 msec
> > ;; SERVER: 68.87.77.130#53(68.87.77.130)
> > ;; WHEN: Wed Jul 28 13:30:53 2010
> > ;; MSG SIZE  rcvd: 74
> >
> >
> > C:\Users\jluthman>dig www.thruway.ny.gov @68.87.72.130
> >
> > ; <<>> DiG 9.3.2 <<>> www.thruway.ny.gov @68.87.72.130
> > ; (1 server found)
> > ;; global options:  printcmd
> > ;; Got answer:
> > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1500
> > ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
> >
> > ;; QUESTION SECTION:
> > ;www.thruway.ny.gov.            IN      A
> >
> > ;; ANSWER SECTION:
> > www.thruway.ny.gov.     37594   IN      CNAME   www.wip.thruway.ny.gov.
> > www.wip.thruway.ny.gov. 30      IN      A       161.11.122.48
> >
> > ;; Query time: 209 msec
> > ;; SERVER: 68.87.72.130#53(68.87.72.130)
> > ;; WHEN: Wed Jul 28 13:31:23 2010
> > ;; MSG SIZE  rcvd: 74
> >
> > Josh Luthman
> > Office: 937-552-2340
> > Direct: 937-552-2343
> > 1100 Wayne St
> > Suite 1337
> > Troy, OH 45373
> >
> >
> >
> > On Wed, Jul 28, 2010 at 1:29 PM, Jared Mauch <jared at puck.nether.net>
> wrote:
> >> I'm finding certain dns queries result in them responding from the
> "wrong" address on their (apparently) anycasted load-balancer nodes.
> >>
> >> take for example: www.thruway.ny.gov
> >>
> >> - Jared
> >>
> >> On Jul 28, 2010, at 1:27 PM, Josh Luthman wrote:
> >>
> >>> Works for me.  Coming from 74.218.88.134
> >>>
> >>> C:\Users\jluthman>dig google.com @68.87.72.130
> >>>
> >>> ; <<>> DiG 9.3.2 <<>> google.com @68.87.72.130
> >>> ; (1 server found)
> >>> ;; global options:  printcmd
> >>> ;; Got answer:
> >>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1358
> >>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0
> >>>
> >>> ;; QUESTION SECTION:
> >>> ;google.com.                    IN      A
> >>>
> >>> ;; ANSWER SECTION:
> >>> google.com.             178     IN      A       74.125.95.99
> >>> google.com.             178     IN      A       74.125.95.147
> >>> google.com.             178     IN      A       74.125.95.104
> >>> google.com.             178     IN      A       74.125.95.106
> >>> google.com.             178     IN      A       74.125.95.105
> >>> google.com.             178     IN      A       74.125.95.103
> >>>
> >>> ;; Query time: 25 msec
> >>> ;; SERVER: 68.87.72.130#53(68.87.72.130)
> >>> ;; WHEN: Wed Jul 28 13:26:20 2010
> >>> ;; MSG SIZE  rcvd: 124
> >>>
> >>>
> >>> C:\Users\jluthman>dig google.com @68.87.7.130
> >>> ^C
> >>> C:\Users\jluthman>dig google.com @68.87.77.130
> >>>
> >>> ; <<>> DiG 9.3.2 <<>> google.com @68.87.77.130
> >>> ; (1 server found)
> >>> ;; global options:  printcmd
> >>> ;; Got answer:
> >>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1226
> >>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0
> >>>
> >>> ;; QUESTION SECTION:
> >>> ;google.com.                    IN      A
> >>>
> >>> ;; ANSWER SECTION:
> >>> google.com.             218     IN      A       209.85.225.103
> >>> google.com.             218     IN      A       209.85.225.99
> >>> google.com.             218     IN      A       209.85.225.104
> >>> google.com.             218     IN      A       209.85.225.147
> >>> google.com.             218     IN      A       209.85.225.106
> >>> google.com.             218     IN      A       209.85.225.105
> >>>
> >>> Josh Luthman
> >>> Office: 937-552-2340
> >>> Direct: 937-552-2343
> >>> 1100 Wayne St
> >>> Suite 1337
> >>> Troy, OH 45373
> >>>
> >>>
> >>> On Wed, Jul 28, 2010 at 12:53 PM, Jared Mauch <jared at puck.nether.net>
> wrote:
> >>>>
> >>>> Anyone else notice issues with comcast dns servers today?  It seems in
> the past 20 minutes their 'anycast' instances have started returning packets
> from the wrong source ip when querying the following:
> >>>>
> >>>> nameserver 68.87.72.130
> >>>> nameserver 68.87.77.130
> >>>>
> >>>> 2010/07/28 12:41:47| WARNING: Reply from unknown nameserver
> 68.87.72.132:53
> >>>> 2010/07/28 12:41:53| WARNING: Reply from unknown nameserver
> 68.87.77.131:53 (retrying...6<=60)
> >>>> 2010/07/28 12:41:58| WARNING: Reply from unknown nameserver
> 68.87.72.131:53 (retrying...11<=60)
> >>>> 2010/07/28 12:42:09| WARNING: Reply from unknown nameserver
> 68.87.77.132:53 (retrying...22<=60)
> >>>> 2010/07/28 12:42:17| WARNING: Reply from unknown nameserver
> 68.87.72.132:53 (retrying...30<=60)
> >>>> 2010/07/28 12:42:39| WARNING: Reply from unknown nameserver
> 68.87.77.131:53 (retrying...52<=60)
> >>>> 2010/07/28 12:42:57| WARNING: Reply from unknown nameserver
> 68.87.72.131:53
> >>>> 2010/07/28 12:43:21| WARNING: Reply from unknown nameserver
> 68.87.72.132:53 (retrying...24<=60)
> >>>> 2010/07/28 12:43:27| WARNING: Reply from unknown nameserver
> 68.87.77.132:53 (retrying...30<=60)
> >>>> 2010/07/28 12:43:30| WARNING: Reply from unknown nameserver
> 68.87.72.132:53 (retrying...33<=60)
> >>>> 2010/07/28 12:43:39| WARNING: Reply from unknown nameserver
> 68.87.77.131:53 (retrying...42<=60)
> >>>> 2010/07/28 12:43:42| WARNING: Reply from unknown nameserver
> 68.87.77.131:53 (retrying...45<=60)
> >>>> 2010/07/28 12:43:52| WARNING: Reply from unknown nameserver
> 68.87.72.131:53 (retrying...55<=60)
> >>>> 2010/07/28 12:44:11| WARNING: Reply from unknown nameserver
> 68.87.77.131:53
> >>>> 2010/07/28 12:44:31| WARNING: Reply from unknown nameserver
> 68.87.72.131:53 (retrying...20<=60)
> >>>> 2010/07/28 12:45:12| WARNING: Reply from unknown nameserver
> 68.87.77.131:53
> >>>> 2010/07/28 12:45:57| WARNING: Reply from unknown nameserver
> 68.87.72.132:53 (retrying...45<=60)
> >>>> 2010/07/28 12:46:02| WARNING: Reply from unknown nameserver
> 68.87.77.132:53 (retrying...50<=60)
> >>>> 2010/07/28 12:46:07| WARNING: Reply from unknown nameserver
> 68.87.72.132:53 (retrying...55<=60)
> >>>> 2010/07/28 12:46:16| WARNING: Reply from unknown nameserver
> 68.87.77.132:53
> >>>> 2010/07/28 12:46:27| WARNING: Reply from unknown nameserver
> 68.87.72.131:53 (retrying...11<=60)
> >>>> 2010/07/28 12:46:47| WARNING: Reply from unknown nameserver
> 68.87.77.131:53 (retrying...31<=60)
> >>>> 2010/07/28 12:46:49| WARNING: Reply from unknown nameserver
> 68.87.72.131:53 (retrying...33<=60)
> >>>> 2010/07/28 12:46:55| WARNING: Reply from unknown nameserver
> 68.87.77.132:53 (retrying...39<=60)
> >>>> 2010/07/28 12:47:00| WARNING: Reply from unknown nameserver
> 68.87.72.132:53 (retrying...44<=60)
> >>>> 2010/07/28 12:47:07| WARNING: Reply from unknown nameserver
> 68.87.72.131:53 (retrying...51<=60)
> >>>> 2010/07/28 12:47:10| WARNING: Reply from unknown nameserver
> 68.87.77.132:53 (retrying...54<=60)
> >>>> 2010/07/28 12:47:20| WARNING: Reply from unknown nameserver
> 68.87.72.131:53
> >>>> 2010/07/28 12:47:40| WARNING: Reply from unknown nameserver
> 68.87.77.131:53 (retrying...20<=60)
> >>>> 2010/07/28 12:47:47| WARNING: Reply from unknown nameserver
> 68.87.77.131:53 (retrying...27<=60)
> >>>> 2010/07/28 12:48:00| WARNING: Reply from unknown nameserver
> 68.87.72.132:53 (retrying...40<=60)
> >>>> 2010/07/28 12:48:40| WARNING: Reply from unknown nameserver
> 68.87.77.131:53
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Outages mailing list
> >>>> Outages at outages.org
> >>>> https://puck.nether.net/mailman/listinfo/outages
> >>
> >>
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100728/8c86a2b1/attachment.htm>