[c-nsp] Huge SP CPU usage spikes 100%

james list jameslist72 at gmail.com
Fri Mar 2 08:07:49 EST 2018


Hi
all the protocol mentioned are stable, we've less than 10 vlans on the gear.

Here the output taken 3 times after 30 secs.

thanks

xxx#sh mls statistics
Statistics for Earl in Module 1

L2 Forwarding Engine
  Total packets Switched                : 1126244033503

L3 Forwarding Engine
  Total packets Processed               : 421505439125 @ 14674 pps
  Total packets L3 Switched             : 204107242703 @ 8069 pps
  Total Packets Bridged                 : 3318438612
  Total Packets FIB Switched            : 204107242702
  Total Packets ACL Routed              : 0
  Total Packets Netflow Switched        : 1
  Total Mcast Packets Switched/Routed   : 124896376575
  Total ip packets with TOS changed     : 2
  Total ip packets with COS changed     : 2
  Total non ip packets COS changed      : 0
  Total packets dropped by ACL          : 3022
  Total packets dropped by Policing     : 0
  Total packets exceeding CIR           : 0
  Total packets exceeding PIR           : 0

Errors
  MAC/IP length inconsistencies         : 1
  Short IP packets received             : 0
  IP header checksum errors             : 0
  TTL failures                          : 103
  MTU failures                          : 0

Statistics for Earl in Module 2

L2 Forwarding Engine
  Total packets Switched                : 288056446772

L3 Forwarding Engine
  Total packets Processed               : 140160213188 @ 8038 pps
  Total packets L3 Switched             : 34932291771 @ 1199 pps
  Total Packets Bridged                 : 656015670
  Total Packets FIB Switched            : 34932291770
  Total Packets ACL Routed              : 0
  Total Packets Netflow Switched        : 1
  Total Mcast Packets Switched/Routed   : 484819383
  Total ip packets with TOS changed     : 2
  Total ip packets with COS changed     : 2
  Total non ip packets COS changed      : 0
  Total packets dropped by ACL          : 0
  Total packets dropped by Policing     : 0
  Total packets exceeding CIR           : 0
  Total packets exceeding PIR           : 0

Errors
  MAC/IP length inconsistencies         : 0
  Short IP packets received             : 0
  IP header checksum errors             : 0
  TTL failures                          : 31737163
  MTU failures                          : 0

Statistics for Earl in Module 3

L2 Forwarding Engine
  Total packets Switched                : 16262292535

L3 Forwarding Engine
  Total packets Processed               : 6051789421 @ 189 pps
  Total packets L3 Switched             : 3336100064 @ 105 pps
  Total Packets Bridged                 : 18843968
  Total Packets FIB Switched            : 3336100063
  Total Packets ACL Routed              : 0
  Total Packets Netflow Switched        : 1
  Total Mcast Packets Switched/Routed   : 2696739936
  Total ip packets with TOS changed     : 2
  Total ip packets with COS changed     : 2
  Total non ip packets COS changed      : 0
  Total packets dropped by ACL          : 0
  Total packets dropped by Policing     : 0
  Total packets exceeding CIR           : 0
  Total packets exceeding PIR           : 0

Errors
  MAC/IP length inconsistencies         : 0
  Short IP packets received             : 0
  IP header checksum errors             : 0
  TTL failures                          : 0
  MTU failures                          : 0

Statistics for Earl in Module 5

L2 Forwarding Engine
  Total packets Switched                : 2610065106448

L3 Forwarding Engine
  Total packets Processed               : 1142795976405 @ 32745 pps
  Total packets L3 Switched             : 508643600024 @ 19294 pps
  Total Packets Bridged                 : 17347781752
  Total Packets FIB Switched            : 508643600023
  Total Packets ACL Routed              : 0
  Total Packets Netflow Switched        : 1
  Total Mcast Packets Switched/Routed   : 181041209538
  Total ip packets with TOS changed     : 2
  Total ip packets with COS changed     : 2
  Total non ip packets COS changed      : 0
  Total packets dropped by ACL          : 2276418
  Total packets dropped by Policing     : 0
  Total packets exceeding CIR           : 0
  Total packets exceeding PIR           : 0

Errors
  MAC/IP length inconsistencies         : 1881
  Short IP packets received             : 0
  IP header checksum errors             : 0
  TTL failures                          : 8688774
  MTU failures                          : 0
Total packets L3 Processed by all Modules: 1710513418139 @ 55646 pps

xxx#sh mls statistics
Statistics for Earl in Module 1

L2 Forwarding Engine
  Total packets Switched                : 1126244939688

L3 Forwarding Engine
  Total packets Processed               : 421505780728 @ 14792 pps
  Total packets L3 Switched             : 204107434350 @ 8375 pps
  Total Packets Bridged                 : 3318439356
  Total Packets FIB Switched            : 204107434349
  Total Packets ACL Routed              : 0
  Total Packets Netflow Switched        : 1
  Total Mcast Packets Switched/Routed   : 124896452309
  Total ip packets with TOS changed     : 2
  Total ip packets with COS changed     : 2
  Total non ip packets COS changed      : 0
  Total packets dropped by ACL          : 3022
  Total packets dropped by Policing     : 0
  Total packets exceeding CIR           : 0
  Total packets exceeding PIR           : 0

Errors
  MAC/IP length inconsistencies         : 1
  Short IP packets received             : 0
  IP header checksum errors             : 0
  TTL failures                          : 103
  MTU failures                          : 0

Statistics for Earl in Module 2

L2 Forwarding Engine
  Total packets Switched                : 288056821850

L3 Forwarding Engine
  Total packets Processed               : 140160389722 @ 7701 pps
  Total packets L3 Switched             : 34932321387 @ 1222 pps
  Total Packets Bridged                 : 656015769
  Total Packets FIB Switched            : 34932321386
  Total Packets ACL Routed              : 0
  Total Packets Netflow Switched        : 1
  Total Mcast Packets Switched/Routed   : 484819394
  Total ip packets with TOS changed     : 2
  Total ip packets with COS changed     : 2
  Total non ip packets COS changed      : 0
  Total packets dropped by ACL          : 0
  Total packets dropped by Policing     : 0
  Total packets exceeding CIR           : 0
  Total packets exceeding PIR           : 0

Errors
  MAC/IP length inconsistencies         : 0
  Short IP packets received             : 0
  IP header checksum errors             : 0
  TTL failures                          : 31737163
  MTU failures                          : 0

Statistics for Earl in Module 3

L2 Forwarding Engine
  Total packets Switched                : 16262304705

L3 Forwarding Engine
  Total packets Processed               : 6051793996 @ 190 pps
  Total packets L3 Switched             : 3336102677 @ 111 pps
  Total Packets Bridged                 : 18843981
  Total Packets FIB Switched            : 3336102676
  Total Packets ACL Routed              : 0
  Total Packets Netflow Switched        : 1
  Total Mcast Packets Switched/Routed   : 2696741885
  Total ip packets with TOS changed     : 2
  Total ip packets with COS changed     : 2
  Total non ip packets COS changed      : 0
  Total packets dropped by ACL          : 0
  Total packets dropped by Policing     : 0
  Total packets exceeding CIR           : 0
  Total packets exceeding PIR           : 0

Errors
  MAC/IP length inconsistencies         : 0
  Short IP packets received             : 0
  IP header checksum errors             : 0
  TTL failures                          : 0
  MTU failures                          : 0

Statistics for Earl in Module 5

L2 Forwarding Engine
  Total packets Switched                : 2610066755272

L3 Forwarding Engine
  Total packets Processed               : 1142796726231 @ 32470 pps
  Total packets L3 Switched             : 508644042257 @ 19228 pps
  Total Packets Bridged                 : 17347783264
  Total Packets FIB Switched            : 508644042256
  Total Packets ACL Routed              : 0
  Total Packets Netflow Switched        : 1
  Total Mcast Packets Switched/Routed   : 181041275389
  Total ip packets with TOS changed     : 2
  Total ip packets with COS changed     : 2
  Total non ip packets COS changed      : 0
  Total packets dropped by ACL          : 2276418
  Total packets dropped by Policing     : 0
  Total packets exceeding CIR           : 0
  Total packets exceeding PIR           : 0

Errors
  MAC/IP length inconsistencies         : 1881
  Short IP packets received             : 0
  IP header checksum errors             : 0
  TTL failures                          : 8688776
  MTU failures                          : 0
Total packets L3 Processed by all Modules: 1710514690677 @ 55153 pps


xxx#sh mls statistics
Statistics for Earl in Module 1

L2 Forwarding Engine
  Total packets Switched                : 1126246359524

L3 Forwarding Engine
  Total packets Processed               : 421506362768 @ 17285 pps
  Total packets L3 Switched             : 204107756570 @ 9475 pps
  Total Packets Bridged                 : 3318440478
  Total Packets FIB Switched            : 204107756569
  Total Packets ACL Routed              : 0
  Total Packets Netflow Switched        : 1
  Total Mcast Packets Switched/Routed   : 124896586736
  Total ip packets with TOS changed     : 2
  Total ip packets with COS changed     : 2
  Total non ip packets COS changed      : 0
  Total packets dropped by ACL          : 3022
  Total packets dropped by Policing     : 0
  Total packets exceeding CIR           : 0
  Total packets exceeding PIR           : 0

Errors
  MAC/IP length inconsistencies         : 1
  Short IP packets received             : 0
  IP header checksum errors             : 0
  TTL failures                          : 103
  MTU failures                          : 0

Statistics for Earl in Module 2

L2 Forwarding Engine
  Total packets Switched                : 288057472476

L3 Forwarding Engine
  Total packets Processed               : 140160704868 @ 9682 pps
  Total packets L3 Switched             : 34932368706 @ 1232 pps
  Total Packets Bridged                 : 656016702
  Total Packets FIB Switched            : 34932368705
  Total Packets ACL Routed              : 0
  Total Packets Netflow Switched        : 1
  Total Mcast Packets Switched/Routed   : 484819408
  Total ip packets with TOS changed     : 2
  Total ip packets with COS changed     : 2
  Total non ip packets COS changed      : 0
  Total packets dropped by ACL          : 0
  Total packets dropped by Policing     : 0
  Total packets exceeding CIR           : 0
  Total packets exceeding PIR           : 0

Errors
  MAC/IP length inconsistencies         : 0
  Short IP packets received             : 0
  IP header checksum errors             : 0
  TTL failures                          : 31737163
  MTU failures                          : 0

Statistics for Earl in Module 3

L2 Forwarding Engine
  Total packets Switched                : 16262339592

L3 Forwarding Engine
  Total packets Processed               : 6051808557 @ 283 pps
  Total packets L3 Switched             : 3336113840 @ 133 pps
  Total Packets Bridged                 : 18844001
  Total Packets FIB Switched            : 3336113839
  Total Packets ACL Routed              : 0
  Total Packets Netflow Switched        : 1
  Total Mcast Packets Switched/Routed   : 2696745262
  Total ip packets with TOS changed     : 2
  Total ip packets with COS changed     : 2
  Total non ip packets COS changed      : 0
  Total packets dropped by ACL          : 0
  Total packets dropped by Policing     : 0
  Total packets exceeding CIR           : 0
  Total packets exceeding PIR           : 0

Errors
  MAC/IP length inconsistencies         : 0
  Short IP packets received             : 0
  IP header checksum errors             : 0
  TTL failures                          : 0
  MTU failures                          : 0

Statistics for Earl in Module 5

L2 Forwarding Engine
  Total packets Switched                : 2610069540467

L3 Forwarding Engine
  Total packets Processed               : 1142797974401 @ 30466 pps
  Total packets L3 Switched             : 508644771724 @ 18373 pps
  Total Packets Bridged                 : 17347786423
  Total Packets FIB Switched            : 508644771723
  Total Packets ACL Routed              : 0
  Total Packets Netflow Switched        : 1
  Total Mcast Packets Switched/Routed   : 181041389292
  Total ip packets with TOS changed     : 2
  Total ip packets with COS changed     : 2
  Total non ip packets COS changed      : 0
  Total packets dropped by ACL          : 2276418
  Total packets dropped by Policing     : 0
  Total packets exceeding CIR           : 0
  Total packets exceeding PIR           : 0

Errors
  MAC/IP length inconsistencies         : 1881
  Short IP packets received             : 0
  IP header checksum errors             : 0
  TTL failures                          : 8688776
  MTU failures                          : 0
Total packets L3 Processed by all Modules: 1710516850594 @ 57716 pps

2018-03-02 11:22 GMT+01:00 James Bensley <jwbensley at gmail.com>:

> On 1 March 2018 at 09:53, james list <jameslist72 at gmail.com> wrote:
> > xxx#show ibc
> > Interface information:
> >         5 minute rx rate 944000 bits/sec, 793 packets/sec
> >         5 minute tx rate 25000 bits/sec, 37 packets/sec
> ...
> >         2467023087 Packets out of 554699386 CEF Switched, 0 Packets out
> of 0
> > Tag CEF Switched
> >         3916625157 Packets Fast Switched
> ...
> >         Potential/Actual paks copied to process level 228808364/225833216
> > (2975148 dropped, 265 spd drops)
> ...
> > MISTRAL ERROR COUNTERS
> ...
> >         2974883 total packets dropped on throttled interfaces (2954630
> low,
> > 16704 medium, 3549 high)
>
>
> >> On 1 March 2018 at 08:29, james list <jameslist72 at gmail.com> wrote:
> >> > Dear experts,
> >> > has anybody experienced a 100% SP CPU usage on C6500-Sup720
> >> > (12.2(33)SXI5)
> >> > with a lot of interrupts ?
> >> > The main process is Heartbeat.
> >> >
> >> > Cisco TAC is struggling in having an idea to sorting out the issue,
> they
> >> > are working since 3 days on it..
> >> >
> >> > STP is stable, no mac moving, no real issue found… maybe somebody
> >> > experienced the same due to something in particular?
>
> I've compared to a similar box I have, it has less control-plane
> traffic than yours it would seem. You have a decent amount of dropped
> packets which I guess to be expected if you have sustained 100% SP CPU
> utilisation.
>
> Do you have a lot of spanning-tree instances, HSRP/VRRP, multicast
> (various other control-plane stuff) running on this box?
>
> Have you confirmed there are no layer 2 loops?
>
> If you run "show mls statistics" several times do you see any error
> stats climbing very quickly?
>
> I would also run a netdr capture and check if the traffic in their is
> normal for your network.
>
> Cheers,
> James.
>


More information about the cisco-nsp mailing list