I agree with Jared. Looks like broken Anycast. It seems that the anycast loopback is properly configured on the machines, but I suspect the bind/nameserver config is borked and causing it to bind to the wrong interface.<div>
<br><div class="gmail_quote">On Wed, Jul 28, 2010 at 10:37 AM, Jared Mauch <span dir="ltr"><<a href="mailto:jared@puck.nether.net">jared@puck.nether.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Yeah, what I'm seeing is if you query for AAAA the 77 or 72.130 ips, you get back the ServFail from the .131 or .132 IP instead of the .130 IP.<br>
<br>
*sigh*<br>
<br>
Thanks for looking.<br>
<font color="#888888"><br>
- Jared<br>
</font><div><div></div><div class="h5"><br>
On Jul 28, 2010, at 1:31 PM, Josh Luthman wrote:<br>
<br>
> Note that 68.87.72.130 took several seconds to respond.<br>
><br>
> C:\Users\jluthman>dig <a href="http://www.thruway.ny.gov" target="_blank">www.thruway.ny.gov</a> @<a href="http://68.87.77.130" target="_blank">68.87.77.130</a><br>
><br>
> ; <<>> DiG 9.3.2 <<>> <a href="http://www.thruway.ny.gov" target="_blank">www.thruway.ny.gov</a> @<a href="http://68.87.77.130" target="_blank">68.87.77.130</a><br>
> ; (1 server found)<br>
> ;; global options: printcmd<br>
> ;; Got answer:<br>
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1343<br>
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0<br>
><br>
> ;; QUESTION SECTION:<br>
> ;<a href="http://www.thruway.ny.gov" target="_blank">www.thruway.ny.gov</a>. IN A<br>
><br>
> ;; ANSWER SECTION:<br>
> <a href="http://www.thruway.ny.gov" target="_blank">www.thruway.ny.gov</a>. 38400 IN CNAME <a href="http://www.wip.thruway.ny.gov" target="_blank">www.wip.thruway.ny.gov</a>.<br>
> <a href="http://www.wip.thruway.ny.gov" target="_blank">www.wip.thruway.ny.gov</a>. 30 IN A 208.105.158.48<br>
><br>
> ;; Query time: 274 msec<br>
> ;; SERVER: 68.87.77.130#53(68.87.77.130)<br>
> ;; WHEN: Wed Jul 28 13:30:53 2010<br>
> ;; MSG SIZE rcvd: 74<br>
><br>
><br>
> C:\Users\jluthman>dig <a href="http://www.thruway.ny.gov" target="_blank">www.thruway.ny.gov</a> @<a href="http://68.87.72.130" target="_blank">68.87.72.130</a><br>
><br>
> ; <<>> DiG 9.3.2 <<>> <a href="http://www.thruway.ny.gov" target="_blank">www.thruway.ny.gov</a> @<a href="http://68.87.72.130" target="_blank">68.87.72.130</a><br>
> ; (1 server found)<br>
> ;; global options: printcmd<br>
> ;; Got answer:<br>
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1500<br>
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0<br>
><br>
> ;; QUESTION SECTION:<br>
> ;<a href="http://www.thruway.ny.gov" target="_blank">www.thruway.ny.gov</a>. IN A<br>
><br>
> ;; ANSWER SECTION:<br>
> <a href="http://www.thruway.ny.gov" target="_blank">www.thruway.ny.gov</a>. 37594 IN CNAME <a href="http://www.wip.thruway.ny.gov" target="_blank">www.wip.thruway.ny.gov</a>.<br>
> <a href="http://www.wip.thruway.ny.gov" target="_blank">www.wip.thruway.ny.gov</a>. 30 IN A 161.11.122.48<br>
><br>
> ;; Query time: 209 msec<br>
> ;; SERVER: 68.87.72.130#53(68.87.72.130)<br>
> ;; WHEN: Wed Jul 28 13:31:23 2010<br>
> ;; MSG SIZE rcvd: 74<br>
><br>
> Josh Luthman<br>
> Office: 937-552-2340<br>
> Direct: 937-552-2343<br>
> 1100 Wayne St<br>
> Suite 1337<br>
> Troy, OH 45373<br>
><br>
><br>
><br>
> On Wed, Jul 28, 2010 at 1:29 PM, Jared Mauch <<a href="mailto:jared@puck.nether.net">jared@puck.nether.net</a>> wrote:<br>
>> I'm finding certain dns queries result in them responding from the "wrong" address on their (apparently) anycasted load-balancer nodes.<br>
>><br>
>> take for example: <a href="http://www.thruway.ny.gov" target="_blank">www.thruway.ny.gov</a><br>
>><br>
>> - Jared<br>
>><br>
>> On Jul 28, 2010, at 1:27 PM, Josh Luthman wrote:<br>
>><br>
>>> Works for me. Coming from 74.218.88.134<br>
>>><br>
>>> C:\Users\jluthman>dig <a href="http://google.com" target="_blank">google.com</a> @<a href="http://68.87.72.130" target="_blank">68.87.72.130</a><br>
>>><br>
>>> ; <<>> DiG 9.3.2 <<>> <a href="http://google.com" target="_blank">google.com</a> @<a href="http://68.87.72.130" target="_blank">68.87.72.130</a><br>
>>> ; (1 server found)<br>
>>> ;; global options: printcmd<br>
>>> ;; Got answer:<br>
>>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1358<br>
>>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0<br>
>>><br>
>>> ;; QUESTION SECTION:<br>
>>> ;<a href="http://google.com" target="_blank">google.com</a>. IN A<br>
>>><br>
>>> ;; ANSWER SECTION:<br>
>>> <a href="http://google.com" target="_blank">google.com</a>. 178 IN A 74.125.95.99<br>
>>> <a href="http://google.com" target="_blank">google.com</a>. 178 IN A 74.125.95.147<br>
>>> <a href="http://google.com" target="_blank">google.com</a>. 178 IN A 74.125.95.104<br>
>>> <a href="http://google.com" target="_blank">google.com</a>. 178 IN A 74.125.95.106<br>
>>> <a href="http://google.com" target="_blank">google.com</a>. 178 IN A 74.125.95.105<br>
>>> <a href="http://google.com" target="_blank">google.com</a>. 178 IN A 74.125.95.103<br>
>>><br>
>>> ;; Query time: 25 msec<br>
>>> ;; SERVER: 68.87.72.130#53(68.87.72.130)<br>
>>> ;; WHEN: Wed Jul 28 13:26:20 2010<br>
>>> ;; MSG SIZE rcvd: 124<br>
>>><br>
>>><br>
>>> C:\Users\jluthman>dig <a href="http://google.com" target="_blank">google.com</a> @<a href="http://68.87.7.130" target="_blank">68.87.7.130</a><br>
>>> ^C<br>
>>> C:\Users\jluthman>dig <a href="http://google.com" target="_blank">google.com</a> @<a href="http://68.87.77.130" target="_blank">68.87.77.130</a><br>
>>><br>
>>> ; <<>> DiG 9.3.2 <<>> <a href="http://google.com" target="_blank">google.com</a> @<a href="http://68.87.77.130" target="_blank">68.87.77.130</a><br>
>>> ; (1 server found)<br>
>>> ;; global options: printcmd<br>
>>> ;; Got answer:<br>
>>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1226<br>
>>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0<br>
>>><br>
>>> ;; QUESTION SECTION:<br>
>>> ;<a href="http://google.com" target="_blank">google.com</a>. IN A<br>
>>><br>
>>> ;; ANSWER SECTION:<br>
>>> <a href="http://google.com" target="_blank">google.com</a>. 218 IN A 209.85.225.103<br>
>>> <a href="http://google.com" target="_blank">google.com</a>. 218 IN A 209.85.225.99<br>
>>> <a href="http://google.com" target="_blank">google.com</a>. 218 IN A 209.85.225.104<br>
>>> <a href="http://google.com" target="_blank">google.com</a>. 218 IN A 209.85.225.147<br>
>>> <a href="http://google.com" target="_blank">google.com</a>. 218 IN A 209.85.225.106<br>
>>> <a href="http://google.com" target="_blank">google.com</a>. 218 IN A 209.85.225.105<br>
>>><br>
>>> Josh Luthman<br>
>>> Office: 937-552-2340<br>
>>> Direct: 937-552-2343<br>
>>> 1100 Wayne St<br>
>>> Suite 1337<br>
>>> Troy, OH 45373<br>
>>><br>
>>><br>
>>> On Wed, Jul 28, 2010 at 12:53 PM, Jared Mauch <<a href="mailto:jared@puck.nether.net">jared@puck.nether.net</a>> wrote:<br>
>>>><br>
>>>> Anyone else notice issues with comcast dns servers today? It seems in the past 20 minutes their 'anycast' instances have started returning packets from the wrong source ip when querying the following:<br>
>>>><br>
>>>> nameserver 68.87.72.130<br>
>>>> nameserver 68.87.77.130<br>
>>>><br>
>>>> 2010/07/28 12:41:47| WARNING: Reply from unknown nameserver <a href="http://68.87.72.132:53" target="_blank">68.87.72.132:53</a><br>
>>>> 2010/07/28 12:41:53| WARNING: Reply from unknown nameserver <a href="http://68.87.77.131:53" target="_blank">68.87.77.131:53</a> (retrying...6<=60)<br>
>>>> 2010/07/28 12:41:58| WARNING: Reply from unknown nameserver <a href="http://68.87.72.131:53" target="_blank">68.87.72.131:53</a> (retrying...11<=60)<br>
>>>> 2010/07/28 12:42:09| WARNING: Reply from unknown nameserver <a href="http://68.87.77.132:53" target="_blank">68.87.77.132:53</a> (retrying...22<=60)<br>
>>>> 2010/07/28 12:42:17| WARNING: Reply from unknown nameserver <a href="http://68.87.72.132:53" target="_blank">68.87.72.132:53</a> (retrying...30<=60)<br>
>>>> 2010/07/28 12:42:39| WARNING: Reply from unknown nameserver <a href="http://68.87.77.131:53" target="_blank">68.87.77.131:53</a> (retrying...52<=60)<br>
>>>> 2010/07/28 12:42:57| WARNING: Reply from unknown nameserver <a href="http://68.87.72.131:53" target="_blank">68.87.72.131:53</a><br>
>>>> 2010/07/28 12:43:21| WARNING: Reply from unknown nameserver <a href="http://68.87.72.132:53" target="_blank">68.87.72.132:53</a> (retrying...24<=60)<br>
>>>> 2010/07/28 12:43:27| WARNING: Reply from unknown nameserver <a href="http://68.87.77.132:53" target="_blank">68.87.77.132:53</a> (retrying...30<=60)<br>
>>>> 2010/07/28 12:43:30| WARNING: Reply from unknown nameserver <a href="http://68.87.72.132:53" target="_blank">68.87.72.132:53</a> (retrying...33<=60)<br>
>>>> 2010/07/28 12:43:39| WARNING: Reply from unknown nameserver <a href="http://68.87.77.131:53" target="_blank">68.87.77.131:53</a> (retrying...42<=60)<br>
>>>> 2010/07/28 12:43:42| WARNING: Reply from unknown nameserver <a href="http://68.87.77.131:53" target="_blank">68.87.77.131:53</a> (retrying...45<=60)<br>
>>>> 2010/07/28 12:43:52| WARNING: Reply from unknown nameserver <a href="http://68.87.72.131:53" target="_blank">68.87.72.131:53</a> (retrying...55<=60)<br>
>>>> 2010/07/28 12:44:11| WARNING: Reply from unknown nameserver <a href="http://68.87.77.131:53" target="_blank">68.87.77.131:53</a><br>
>>>> 2010/07/28 12:44:31| WARNING: Reply from unknown nameserver <a href="http://68.87.72.131:53" target="_blank">68.87.72.131:53</a> (retrying...20<=60)<br>
>>>> 2010/07/28 12:45:12| WARNING: Reply from unknown nameserver <a href="http://68.87.77.131:53" target="_blank">68.87.77.131:53</a><br>
>>>> 2010/07/28 12:45:57| WARNING: Reply from unknown nameserver <a href="http://68.87.72.132:53" target="_blank">68.87.72.132:53</a> (retrying...45<=60)<br>
>>>> 2010/07/28 12:46:02| WARNING: Reply from unknown nameserver <a href="http://68.87.77.132:53" target="_blank">68.87.77.132:53</a> (retrying...50<=60)<br>
>>>> 2010/07/28 12:46:07| WARNING: Reply from unknown nameserver <a href="http://68.87.72.132:53" target="_blank">68.87.72.132:53</a> (retrying...55<=60)<br>
>>>> 2010/07/28 12:46:16| WARNING: Reply from unknown nameserver <a href="http://68.87.77.132:53" target="_blank">68.87.77.132:53</a><br>
>>>> 2010/07/28 12:46:27| WARNING: Reply from unknown nameserver <a href="http://68.87.72.131:53" target="_blank">68.87.72.131:53</a> (retrying...11<=60)<br>
>>>> 2010/07/28 12:46:47| WARNING: Reply from unknown nameserver <a href="http://68.87.77.131:53" target="_blank">68.87.77.131:53</a> (retrying...31<=60)<br>
>>>> 2010/07/28 12:46:49| WARNING: Reply from unknown nameserver <a href="http://68.87.72.131:53" target="_blank">68.87.72.131:53</a> (retrying...33<=60)<br>
>>>> 2010/07/28 12:46:55| WARNING: Reply from unknown nameserver <a href="http://68.87.77.132:53" target="_blank">68.87.77.132:53</a> (retrying...39<=60)<br>
>>>> 2010/07/28 12:47:00| WARNING: Reply from unknown nameserver <a href="http://68.87.72.132:53" target="_blank">68.87.72.132:53</a> (retrying...44<=60)<br>
>>>> 2010/07/28 12:47:07| WARNING: Reply from unknown nameserver <a href="http://68.87.72.131:53" target="_blank">68.87.72.131:53</a> (retrying...51<=60)<br>
>>>> 2010/07/28 12:47:10| WARNING: Reply from unknown nameserver <a href="http://68.87.77.132:53" target="_blank">68.87.77.132:53</a> (retrying...54<=60)<br>
>>>> 2010/07/28 12:47:20| WARNING: Reply from unknown nameserver <a href="http://68.87.72.131:53" target="_blank">68.87.72.131:53</a><br>
>>>> 2010/07/28 12:47:40| WARNING: Reply from unknown nameserver <a href="http://68.87.77.131:53" target="_blank">68.87.77.131:53</a> (retrying...20<=60)<br>
>>>> 2010/07/28 12:47:47| WARNING: Reply from unknown nameserver <a href="http://68.87.77.131:53" target="_blank">68.87.77.131:53</a> (retrying...27<=60)<br>
>>>> 2010/07/28 12:48:00| WARNING: Reply from unknown nameserver <a href="http://68.87.72.132:53" target="_blank">68.87.72.132:53</a> (retrying...40<=60)<br>
>>>> 2010/07/28 12:48:40| WARNING: Reply from unknown nameserver <a href="http://68.87.77.131:53" target="_blank">68.87.77.131:53</a><br>
>>>><br>
>>>><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> Outages mailing list<br>
>>>> <a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
>>>> <a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
>><br>
>><br>
<br>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</div></div></blockquote></div><br></div>