[c-nsp] High CPU on Cisco 7606 (SUP2/MSFC2/PFC2) w GRE tunnels

[Rich Davies]

Hello,

I am doing some troubleshooting on a Cisco 7606 in regards to high CPU
(between 50 and 75%).   When I do a "show proc cpu" I do not show any
processes that are using up the CPU.  The one thing we are doing
differently on this router is terminating about 4 GRE tunnels.  In addition
when I do a "show mls stats" I am showing the router is currently
processing about 500,000 PPS:

#show mls statistics

Statistics for Earl in Module 1

L2 Forwarding Engine
  Total packets Switched                : 943744507055

L3 Forwarding Engine
  Total Packets L3 Switched             : 943744507055 @ 520113 pps

  Total Packets Bridged                 : 159661113984
  Total Packets FIB Switched            : 783029776276
  Total Packets ACL Routed              : 0
  Total Packets Netflow Switched        : 0
  Total Mcast Packets Switched/Routed   : 884048717
  Total ip packets with TOS changed     : 0
  Total ip packets with COS changed     : 0
  Total non ip packets COS changed      : 0
  Total packets dropped by ACL          : 779576
  Total packets dropped by Policing     : 0
  Total Unicast RPF failed packets      : 0

Errors
  MAC/IP length inconsistencies         : 1
  Short IP packets received             : 0
  IP header checksum errors             : 0
  MAC/IPX length inconsistencies        : 0
  Short IPX packets received            : 0

Total packets L3 Switched by all Modules: 943744507055 @ 520113 PPS



I recently read a Cisco forum document that stated GRE tunnels can push up
CPU if they "share the same source interface" and suggestion was to use 1
loopback interface per GRE tunnel:

https://supportforums.cisco.com/discussion/10920621/high-cpu-utilization-7600-rsp720-and-gre-tunnel


I have performed re-configuration of our GRE tunnels to use a Loopback
interface as source however I am still seeing CPU in range of 50-75%.

Should GRE tunnels cause this much of a CPU increase?

I guess my other question is in relation to software switched versus
hardware switched.   I am showing that my GRE interfaces are using FastCEF
so I would expect performance not to be an issue however the GRE tunnel
(and Loopback) are indeed software interfaces so does CEF even come into
play regarding GRE tunnel interfaces?


#show mod
Mod Ports Card Type                              Model              Serial
No.
--- ----- -------------------------------------- ------------------
-----------
  1    2  Catalyst 6000 supervisor 2 (Active)    WS-X6K-SUP2-2GE
 SAL08486LSL
  3    8  8 port 1000mb GBIC Enhanced QoS        WS-X6408A-GBIC
SAL050630Y0

Mod MAC addresses                       Hw    Fw           Sw
Status
--- ---------------------------------- ------ ------------ ------------
-------
  1  0012.0150.4ad0 to 0012.0150.4ad1   5.1   7.1(1)       12.1(27b)E1  Ok
  3  0001.9724.0e54 to 0001.9724.0e5b   1.3   5.4(2)       8.5(0.23)COS Ok

Mod Sub-Module                  Model           Serial           Hw
Status
--- --------------------------- --------------- --------------- -------
-------
  1 Policy Feature Card 2       WS-F6K-PFC2     SAL084764AX      3.5    Ok
  1 Cat6k MSFC 2 daughterboard  WS-F6K-MSFC2    SAL08517DM7      2.8    Ok

Mod Online Diag Status
--- -------------------
  1 Pass
  3 Pass


I have been referring to this document quite a bit to reference PPS:

http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf

With the MSFC2/PFC2 I would expect to have a capacity of 15/30 million PPS
on this 7606 however I suspicion I am hitting bottleneck of a software
switched interface (500,000 for software-switched CEF).


Anyone have any pointers or tips?    I know many will be laughing that
MSFC2/PFC2 is still in play but it is what it is.  Should I consider
offloading these tunnels to another device?

The nature of the high PPS traffic is RTP/VOIP (so many PPS of smaller
packets).



Thanks,

Rich