[c-nsp] 2651xm performance experience

[barney gumbo]

I have a pair of 2651xm which are running high CPU at ~10 min intervals. The
CPU runs up to about 90% for ~90 seconds then recovers.  The bandwidth being
consumed when this happens is less then 700 kbps total through the router.
I believe the 2651xm is rated at 20 mbps, when there are no fancy
configurations/features etc.  Both routers are currently running 12.4 latest
releast, base IP image.

I know the traffic flow which occurs when the CPU runs up, and that flow is
normal expected traffic, as noted only consuming ~700 kbps.  The 2651xm have
performed well for me in the past as a simple LAN router, routing between
two fast-e networks, and we definitely got alot more then 700kbps, typically
around 15 Mbps, w/o any issues.

Each 2651xm is configured with 4 pt-pt T1 links and a single fast-e link to
a LAN.  The routing protocol is EIGRP.  There is a QOS policy which matches
some VOIP traffic and assigns it to a priority queue which reserves 50% of
the interface bandwidth, everythig else is fair-queue.  There are no special
features enabled on the router, no NAT, no interface access-groups, no
policy-routing, very simple configuration.

I've tried using MLPP for the 4x T1 and I've also tried this with each T1
being in it's own /30.  In both scenarios, traffic is being balanced well
across the T1's in both directions.  High CPU happens with MLPP or each T1
having it's own /30.  Each T1 is performing fairly well, other then slightly
high latency (no jitter however) they take the same physical transport path.

As best as I can tell, the router isn't doing any IP fragmentation or
generating/processing alot of ICMP traffic.  IP Cef is enabled.

I've tried downgrading to 12.3 latest release, same CPU consumption.

Can anyone provide any insight to why this might be happening?
Any suggestions on how to troubleshoot this further?