[c-nsp] 6500 high cpu usage

[Bill Desjardins]

hi,

I searched the cisco docs and archives and couldnt find anything much to
answer this.

over the weekend I just replaced 2 7507 core routers with 2 6500's. each
6500 has dual sup 1's. one with msfc2/512MB and second with msfc1/128MB. my
network is very simple with one 6500 having a gig uplink doing bgp with my
provider taking only a default route. the 2nd 6500 connects to my other
location via a FE cross-connect. off of the 2 6500's, hang 2 5500's each
with gig uplinks to each 6500. all customer vlans are on each 6500 using
hsrp between the 2 core 6500's. the 6500's are connected via a 4gig
portchannel. the 6500's are running native IOS Version 12.1(26)E3.

the network is running fine and I see no packetloss or errors on any
interfaces. as for traffic, at peak times, it will hit 90Mbit with about
30Mbit of that being voip traffic. pps max is around 40k each in & out on
the gig uplink. now at that traffic rate, I am seeing anywhere from 35-40%
cpu usage which seems really hi for this amt of traffic. the previous
7507's with vip2-50's & vip4-80 were only hitting about 75% during peak
traffic using dCEF. I am running only a single input ACL on the primary
6500 Gig uplink to my provider with about 200 entries and none with a log
statement. internal routing is via ospf and there are only 3 routers in the
entire ospf area.

I think I may have some mls issue that is causing much of my traffic to be
process switched instead of hardware switched. the output from: sh mls stat
shows only about 3-4% of packets being shortcut:

SR1-06#sh mls stat
  Number of ip unicast shortcuts    : 260
  Number of ip multicast shortcuts  : 0
  Number of ipx shortcuts           : 0
  Number of l2 shortcuts            : 0
  Total packets counted             : 22617337844
  Total ip packets shortcut         : 790816353
  Total ipx packets shortcut        : 0
  Total ip packets with TOS changed : 621266
  Total ip packets with COS changed : 4145992881
  Total non ip packets COS changed  : 0
  Total packets dropped by access-lists     : 1168711
  Total packets dropped by traffic policing : 0

  Errors

  MAC/IP length inconsistencies             : 0
  Short IP packets received                 : 3631
  IP header checksum errors                 : 0
  MAC/IPX length inconsistencies             : 0
  Short IPX packets received                : 0


on the provider gig interface (actual interface, not a vlan confg), I am
using route-cache flow, all other interfaces are using the default
route-cache config. I am using the default mls configuration as well with
no changes.

I am at a loss on why only so little traffic is shortcut vi mls or why
exactly I am using so much cpu. I have upcoming projects that are going to
easily double my traffic and possibly higher and this cpu usage I see wont
get me to far along.

any help tracking this down is appreciated. let me know what further
information or config's I can provide to help provide more info.

Thanks,

Bill