[c-nsp] "ip icmp rate-limit unreachables DF" broken in IOS 12.2(28)SB6

[Vinny Abello]

Hello Cisco fans,

I've been troubleshooting a PMTU issue on my network that involves DS3's at both ends and couple of redundant FastEthernet links in the middle. The MTU sizes are defaults: 4470 for the DS3 serial interfaces and 1500 for the FastEthernet. All routers are 7206VXR's running the same 12.2(28)SB6 code. For the sake of simplicity, here is a basic diagram:

            DS3                  Fe                  DS3
Router A ---------- Router B ---------- Router C ---------- Router D
MTU:       4470                 1500                4470

I have "ip tcp path-mtu-discovery" enabled on both routers A and D. This is to make BGP more efficient in the size of packets it sends from the routers. I found that an iBGP between loopbacks of Router A and Router D over this path breaks when doing this after resetting it to renegotiate using PMTU discovery. It will hang which is typical of an MTU issue. I confirmed this by doing the following: 

I took a look at the neighbor detail and see they are using the max data size of I think 4410 if my memory is serving me correctly. It should negotiate at 1440 based on my other 1500 byte interfaces in the network with BGP running over them. I tried pinging with large sizes through the path with DF set and I didn't get any icmp error packets back at all indicating fragmentation is needed. I set the MTU on my DS3 interfaces to 1500 temporarily and everything worked as far as BGP goes. I set them back to the default after that and did more investigation. I suspected a router in the middle to be at fault so I took a look at the router that is at the other end of the DS3 on one side with the next interface being a FastEthernet interface. That router should be sending back the unreachable packets. I turned on debug ip icmp to look for anything when I sent oversized ICMP DF packets but didn't see any corresponding results. I verified all interfaces in the path did not have "no ip 
unreachables" configured as I have read that this breaks PMTU discovery but still no luck (CAN ANYONE CONFIRM THAT TO BE TRUE?? My tests don't seem to confirm that so far). Finally, I tried tinkering with a setting for rate-limiting unreachables with the DF bit set. I have a setting of:

ip icmp rate-limit unreachables DF 2000

in my configuration on router B. This seems to be a common value used by many ISP's and security guides and seemed sensible to me as well. I fully understand what it is supposed to do per the Cisco documentation. I would expect to see at least one or two icmp responses to my oversized pings with DF set but was not. I kept playing with the value and even lowered it all the way to 1 millisecond and still no change. I finally removed it completely and PMTU started working and I saw the ICMP packets in the debug! I don't understand why as I've read the default rate limiting is set to 500 milliseconds and I had clearly set the value much lower. I also tried rate-limiting unreachables without the DF bit set and that did not seem to break anything as far as PMTU so it only seems to be specifically the above command with any value that totally disables ICMP type 3 subtype 4 packets from originating from the router.

This is a ping from Router A with ip icmp rate-limit unreachables DF 2000 enabled on Router B:

Type escape sequence to abort.
Sending 5, 4470-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
Packet sent with the DF bit set
.....
Success rate is 0 percent (0/5)

and this is with is disabled:

Type escape sequence to abort.
Sending 5, 4470-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
Packet sent with the DF bit set
MMMMM
Success rate is 0 percent (0/5)

Has anyone seen this behavior? Is this a known bug and is it fixed in a later release? It would appear that you completely break PMTU discovery when using this command with any value configured. This would be very easy to reproduce in a lab. I'm curious as to what the Cisco employees on the list have to say and if they can confirm this. I can't be the only one who has run into this issue.


Thanks in advance!

-- 

Vinny Abello
Network Engineer
vinny at tellurian.com
(973)940-6100 (NOC)
PGP Key Fingerprint: 3BC5 9A48 FC78 03D3 82E0  E935 5325 FBCB 0100 977A

Tellurian Networks - The Ultimate Internet Connection
http://www.tellurian.com (888)TELLURIAN

"There is no objective reality. Only that which is measured exists.
We construct reality, and only in the moment of measurement or observation." -- Niels Bohr