[c-nsp] High CPU increase on upgrade to 12.2SB

[David Freedman]

Hiya.

Have the following box:

7206VXR + NPE-G1 + PA-MC-STM-1SMIx3

Box is currently pushing around 160Mbps at around 60kpps

Box has some simple CBWFQ configured on some ethernet and serial 
subinterfaces.

Box is doing some VRF and basic EoMPLS xconnects.

Box does not run any netflow of any kind.

Box was upgraded from 12.3M to 12.2(31)SB6, to provide support for some 
SB features.

Since then, Interrupt CPU utilisation has jumped from around 50% average 
to around 80%

I note that CEF CSSR is present in this train so can produce more useful 
output from commands:


router#show cef not-cef-switched
% Command accepted but obsolete, see 'show (ip|ipv6) cef switching 
statistics [feature]'

IPv4 CEF Packets passed on to next switching layer
Slot  No_adj No_encap Unsupp'ted Redirect  Receive  Options   Access 
  Frag
RP      5430       0      113034        0    93188      626        0 
     0

This shows an unsupported feature

router#show ip cef switching stat

Path   Reason                          Drop       Punt  Punt2Host
RP LES No route                           0          0         94
RP LES Packet destined for us             0      47284          0
RP LES Total                              0      47284         94

RP PAS No route                        4564          0       8946
RP PAS Packet destined for us             0      47284          0
RP PAS No adjacency                    1194          0       5500
RP PAS TTL expired                        0          0     107449
RP PAS IP options set                     0          0        634
RP PAS Routed to Null0                  346          0          0
RP PAS Total                           6104      47284     122529

All    Total                           6104      94568     122623


This shows it to be TTL expiry messages?


router#show ip cef switching stat feat
IPv4 CEF input features:
Path   Feature                Drop    Consume       Punt  Punt2Host Gave 
route
Total                            0          0          0          0 
      0

IPv4 CEF output features:
Path   Feature                Drop    Consume       Punt  Punt2Host 
New i/f
Total                            0          0          0          0 
      0

IPv4 CEF post-encap features:
Path   Feature                Drop    Consume       Punt  Punt2Host 
New i/f
Total                            0          0          0          0 
      0

IPv4 CEF for us features:
Path   Feature                Drop    Consume       Punt  Punt2Host 
New i/f
Total                            0          0          0          0 
      0


This all empty (perhaps not supported?)


On the uplink ports, we get this:

router#show int g0/1 switching
GigabitEthernet0/1
           Throttle count          0
         Drops         RP          0         SP          0
   SPD Flushes       Fast       3189        SSE          0
   SPD Aggress       Fast          0
  SPD Priority     Inputs      12615      Drops          0

      Protocol       Path    Pkts In   Chars In   Pkts Out  Chars Out
         Other    Process          0          0      13248     794880
             Cache misses          0
                     Fast          0          0          0          0
                Auton/SSE          0          0          0          0
            IP    Process    1106051  153315242     105255    7231157
             Cache misses          0
                     Fast  770751156 2994272118    3772580 1821732081
                Auton/SSE          0          0          0          0
          CLNS    Process     201612  240174881      16850   23234124
             Cache misses          0
                     Fast          0          0          0          0
                Auton/SSE          0          0          0          0
           ARP    Process       3379     202740         70       4200
             Cache misses          0
                     Fast          0          0          0          0
                Auton/SSE          0          0          0          0
           CDP    Process       8863    3580717       2211     915354
             Cache misses          0
                     Fast          0          0          0          0
                Auton/SSE          0          0          0          0
          MPLS    Process       2339     235495    3426680  274358967
             Cache misses          0
                     Fast 3074439385 3269202725 3569375313 2811091314
                Auton/SSE          0          0          0          0

router#show int g0/2 switching
GigabitEthernet0/2
           Throttle count          0
         Drops         RP          0         SP          0
   SPD Flushes       Fast         60        SSE          0
   SPD Aggress       Fast          0
  SPD Priority     Inputs      13039      Drops          0

      Protocol       Path    Pkts In   Chars In   Pkts Out  Chars Out
         Other    Process          0          0      13252     795120
             Cache misses          0
                     Fast          0          0          0          0
                Auton/SSE          0          0          0          0
            IP    Process     471259   57793662      78908    5443081
             Cache misses          0
                     Fast  184085302 1818412512    7761754 1497075822
                Auton/SSE          0          0          0          0
          CLNS    Process     201982  240258389      17116   23240837
             Cache misses          0
                     Fast          0          0          0          0
                Auton/SSE          0          0          0          0
           ARP    Process       3303     198180         69       4140
             Cache misses          0
                     Fast          0          0          0          0
                Auton/SSE          0          0          0          0
           CDP    Process      11085    4498405       2211     915354
             Cache misses          0
                     Fast          0          0          0          0
                Auton/SSE          0          0          0          0
          MPLS    Process       9024    8008512      35310   10556524
             Cache misses          0
                     Fast  302274149 1229580627  589234146 1635228633
                Auton/SSE          0          0          0          0




I'm unsure as to how to proceed, the traffic profile does not appear to 
have changed, we are not running netflow on here and all interfaces 
apeear to be CEF switching (i.e its not disabled),
the interfaces with the CBWFQ do not appear to be the major offenders.

Any advice appreciated.


Dave.