[c-nsp] 6500 SUP720 High Latency and Jitter issues

[Dan Benson]

Hello all, I am in dire need of some assistance with a new problem that 
has started occurring in one of my networks.  We are seeing very high 
Latency and Jitter traversing one of our 6509's that has a sup720 with 
the PFC3BXL and a GIG or Ram.  This period occur every 10 to 15 seconds, 
and last for about 2 seconds.  This is causing a massive amount of 
troubles for our customers and I am at a loss as to what could be the 
cause. 

Hardware:

 Cisco 6509, Fan2, Sup720,PFC3BXL, 1GIGe of Ram, 1x8 port GBIC gige, 
1x48 port FE.

Config:

 IOS Version 12.2(18)SXD3, running 2 full BGP routing tables to ISP's 
VIA FE and GIGe.  OSPF running for core network with 5 neighbors.  2 
VLans with ports assigned to them.  6 GRE tunnels for private access to 
other offsite POPs.  NAT running with overload to a public VLAN 
interface.  Router CPU average load is 50% per 24 hours, peaks around 
60%, lows around 20%.  Mem usage is at 20%.

Issue: 

Every 5 to 15 seconds we are seeing pings like this traversing the box:

64 bytes from 147.135.0.16: icmp_seq=2099 ttl=63 time=1.000 ms
64 bytes from 147.135.0.16: icmp_seq=2100 ttl=63 time=1.000 ms
64 bytes from 147.135.0.16: icmp_seq=2101 ttl=63 time=1.033 ms
64 bytes from 147.135.0.16: icmp_seq=2102 ttl=63 time=1.037 ms
64 bytes from 147.135.0.16: icmp_seq=2103 ttl=63 time=1.001 ms
64 bytes from 147.135.0.16: icmp_seq=2104 ttl=63 time=86.345 ms
64 bytes from 147.135.0.16: icmp_seq=2105 ttl=63 time=179.171 ms
64 bytes from 147.135.0.16: icmp_seq=2106 ttl=63 time=178.301 ms
64 bytes from 147.135.0.16: icmp_seq=2107 ttl=63 time=108.800 ms
64 bytes from 147.135.0.16: icmp_seq=2108 ttl=63 time=33.387 ms
64 bytes from 147.135.0.16: icmp_seq=2109 ttl=63 time=1.018 ms
64 bytes from 147.135.0.16: icmp_seq=2110 ttl=63 time=1.014 ms
64 bytes from 147.135.0.16: icmp_seq=2111 ttl=63 time=1.064 ms
64 bytes from 147.135.0.16: icmp_seq=2112 ttl=63 time=1.023 ms
64 bytes from 147.135.0.16: icmp_seq=2113 ttl=63 time=1.042 ms

I am used to the 60 sec. ICMP de-pri BGP update CPU load pinging the box 
directly, but I have never seen this effect devices on the far side of 
the router.  Any input on this is greatly appreciated.  //db