[j-nsp] Bizaare bug of the year award

[Richard A Steenbergen]

Ok, I just found a Juniper bug that is so strange I can't even bring 
myself to open a case for it. Not only because I can just imagine trying 
to explain it to JTAC, but because it is so freaking bizaare and funny 
that I just have to share it with others. :)

It seems that in 9.1+ code there is a bug in the traceroute application, 
which causes it to query the incorrect reverse DNS for certain IP 
addresses. The nameservers and the resolv lib on the router itself are 
fine, you can do a "host x.x.x.x" and get correct results, it is only 
the traceroute application which is broken. JUNOS 9.0 and earlier is 
confirmed to be working correctly, JUNOS 9.1/9.2 is confirmed to be 
broken on all versions.

Here is an example:

traceroute to ytti.fi (62.236.255.178), 30 hops max, 40 byte packets
...
 7  te0-2-0-0-10G.alb2nqh1.ip.tele.dk (83.88.20.25)  148.334 ms  126.819 ms  126.299 ms
 8  pos0-0-0-0-10G.stkm3nqh1.ip.tele.dk (83.88.27.6)  156.377 ms  127.819 ms  126.253 ms
 9  Wimax-Cali-190-0-0-61.orbitel.net.co (62.236.181.189)  126.271 ms  126.310 ms  126.189 ms
10  Wimax-Cali-190-0-0-16.orbitel.net.co (62.236.181.16)  130.327 ms  126.542 ms  126.455 ms
11  Wimax-Cali-190-0-0-29.orbitel.net.co (62.236.182.29)  126.178 ms  126.272 ms  126.346 ms

As you can see, it seems to be querying the incorrect IP for hops 9, 10, 
and 11. Here is a tcpdump from the router itself showing that the 
traceroute app is actually querying for the wrong ptr:

IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto: UDP (17),
length: 119) ns1.mynameserverhere.net.domain >
lo0.myrouterhere.net.56079: [udp sum ok] 11052 q: PTR? 
29.0.0.190.in-addr.arpa. 1/0/0 29.0.0.190.in-addr.arpa. PTR
Wimax-Cali-190-0-0-29.orbitel.net.co. (91)

As you can see, hops 10 and 11 are directly mapping of the last octet to
a query of 190.0.0.x, but hop 9 is not. However, since 189-61=128, you
can clearly see that hop 9 is actually zeroing out the top bit (or bits)
of the last octet.

Here is another example of the brokenness:

traceroute to www.telekom.de (217.6.10.34), 30 hops max, 40 byte packets
...
 6  Wimax-Cali-190-0-0-42.orbitel.net.co (62.154.16.234)  13.428 ms 
    Wimax-Cali-190-0-0-46.orbitel.net.co (62.154.16.238)  15.618 ms 
    Wimax-Cali-190-0-0-42.orbitel.net.co (62.154.16.234)  15.574 ms

Only thing in common here is the 62 first octet, though you can now see
that its zeroing out the top 2 bits when doing its lookup. No clue if
there are any other patterns of IPs where this happens also, if all of
62.0.0.0/8 is broken, etc, but its definitely been confirmed on many
routers.

Bonus points to any jnpr developer who can tell me how you guys actually
managed to break this, because this is an award winner in my book. :)

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)