[nsp] Latency across Cat4k SUP3

[Tony Mucker]

I have two Cat4ks who are HSRPing for my default gateway on my backend
admin network.  We tried making one the active router today, and it
didn't go so well.  We ended up having the flip the HSRP back.

Testing reveals that one of them is fine, and the other causes latency
when you cross it's supervisor module.  For example, if I take a host
and set it's default GW to the problem child, and try to ping something
on another vlan, it looks like this (mind you these are on the same
switches):


> tmucker at neta01 tmucker $ ping 10.13.199.4
> PING 10.13.199.4 (10.13.199.4) from 10.13.255.101 : 56(84) bytes of data.
> 64 bytes from 10.13.199.4: icmp_seq=0 ttl=254 time=96.748 msec
> 64 bytes from 10.13.199.4: icmp_seq=1 ttl=254 time=37.394 msec
> 64 bytes from 10.13.199.4: icmp_seq=2 ttl=254 time=48.522 msec
> 64 bytes from 10.13.199.4: icmp_seq=3 ttl=254 time=60.912 msec
> 64 bytes from 10.13.199.4: icmp_seq=4 ttl=254 time=18.307 msec


Trace route looks like this (msfc1 is the Cat4k, lb-s100a is F5 BIG-IP
external IP, third hop is the internal IP for the BIG-IP):


> tmucker at neta01 tmucker $ /usr/sbin/traceroute 10.13.199.4
> traceroute to 10.13.199.4 (10.13.199.4), 30 hops max, 38 byte packets
>  1  msfc1 (10.13.255.2)  0.855 ms  0.797 ms  0.893 ms
>  2  lb-s100a (some external IP here)  196.474 ms  221.150 ms  176.251 ms
>  3  lb-s199a-199 (10.13.199.4)  161.389 ms  105.360 ms  176.249 ms


The two Cat4ks contain two routes, a default gateway (our FWSMs on the 6509s), 
and the route to our office via a backend T1 router.  The Cat4ks themselves are 
directly connected to the 10.13.255 network, as is the T1 router to our office.

Pinging the T1 router from a host with the problem Cat4k as it's default GW 
shows no problem.  The host and the T1 router are on the same VLAN (10.13.255.X
network).  When we ping a machine in the office from a host in the cage, we get 
the same latency issues.  This rules out a loadbalancer issue.  

What this means is that we have a host on the 255 network with it's default 
gateway as the problem Cat4k.  It can ping the other machines on that 255 
network with no problem.  When it traverses a VLAN, it gets latency.  However,
when it has a route applied against it (in the case of the route that points 
to our office), it also incurs latency.  Even if the next hop is on that same
255 network.

Any ideas?