[c-nsp] Huge SP CPU usage spikes 100%
james list
jameslist72 at gmail.com
Fri Mar 2 08:07:49 EST 2018
Hi
all the protocol mentioned are stable, we've less than 10 vlans on the gear.
Here the output taken 3 times after 30 secs.
thanks
xxx#sh mls statistics
Statistics for Earl in Module 1
L2 Forwarding Engine
Total packets Switched : 1126244033503
L3 Forwarding Engine
Total packets Processed : 421505439125 @ 14674 pps
Total packets L3 Switched : 204107242703 @ 8069 pps
Total Packets Bridged : 3318438612
Total Packets FIB Switched : 204107242702
Total Packets ACL Routed : 0
Total Packets Netflow Switched : 1
Total Mcast Packets Switched/Routed : 124896376575
Total ip packets with TOS changed : 2
Total ip packets with COS changed : 2
Total non ip packets COS changed : 0
Total packets dropped by ACL : 3022
Total packets dropped by Policing : 0
Total packets exceeding CIR : 0
Total packets exceeding PIR : 0
Errors
MAC/IP length inconsistencies : 1
Short IP packets received : 0
IP header checksum errors : 0
TTL failures : 103
MTU failures : 0
Statistics for Earl in Module 2
L2 Forwarding Engine
Total packets Switched : 288056446772
L3 Forwarding Engine
Total packets Processed : 140160213188 @ 8038 pps
Total packets L3 Switched : 34932291771 @ 1199 pps
Total Packets Bridged : 656015670
Total Packets FIB Switched : 34932291770
Total Packets ACL Routed : 0
Total Packets Netflow Switched : 1
Total Mcast Packets Switched/Routed : 484819383
Total ip packets with TOS changed : 2
Total ip packets with COS changed : 2
Total non ip packets COS changed : 0
Total packets dropped by ACL : 0
Total packets dropped by Policing : 0
Total packets exceeding CIR : 0
Total packets exceeding PIR : 0
Errors
MAC/IP length inconsistencies : 0
Short IP packets received : 0
IP header checksum errors : 0
TTL failures : 31737163
MTU failures : 0
Statistics for Earl in Module 3
L2 Forwarding Engine
Total packets Switched : 16262292535
L3 Forwarding Engine
Total packets Processed : 6051789421 @ 189 pps
Total packets L3 Switched : 3336100064 @ 105 pps
Total Packets Bridged : 18843968
Total Packets FIB Switched : 3336100063
Total Packets ACL Routed : 0
Total Packets Netflow Switched : 1
Total Mcast Packets Switched/Routed : 2696739936
Total ip packets with TOS changed : 2
Total ip packets with COS changed : 2
Total non ip packets COS changed : 0
Total packets dropped by ACL : 0
Total packets dropped by Policing : 0
Total packets exceeding CIR : 0
Total packets exceeding PIR : 0
Errors
MAC/IP length inconsistencies : 0
Short IP packets received : 0
IP header checksum errors : 0
TTL failures : 0
MTU failures : 0
Statistics for Earl in Module 5
L2 Forwarding Engine
Total packets Switched : 2610065106448
L3 Forwarding Engine
Total packets Processed : 1142795976405 @ 32745 pps
Total packets L3 Switched : 508643600024 @ 19294 pps
Total Packets Bridged : 17347781752
Total Packets FIB Switched : 508643600023
Total Packets ACL Routed : 0
Total Packets Netflow Switched : 1
Total Mcast Packets Switched/Routed : 181041209538
Total ip packets with TOS changed : 2
Total ip packets with COS changed : 2
Total non ip packets COS changed : 0
Total packets dropped by ACL : 2276418
Total packets dropped by Policing : 0
Total packets exceeding CIR : 0
Total packets exceeding PIR : 0
Errors
MAC/IP length inconsistencies : 1881
Short IP packets received : 0
IP header checksum errors : 0
TTL failures : 8688774
MTU failures : 0
Total packets L3 Processed by all Modules: 1710513418139 @ 55646 pps
xxx#sh mls statistics
Statistics for Earl in Module 1
L2 Forwarding Engine
Total packets Switched : 1126244939688
L3 Forwarding Engine
Total packets Processed : 421505780728 @ 14792 pps
Total packets L3 Switched : 204107434350 @ 8375 pps
Total Packets Bridged : 3318439356
Total Packets FIB Switched : 204107434349
Total Packets ACL Routed : 0
Total Packets Netflow Switched : 1
Total Mcast Packets Switched/Routed : 124896452309
Total ip packets with TOS changed : 2
Total ip packets with COS changed : 2
Total non ip packets COS changed : 0
Total packets dropped by ACL : 3022
Total packets dropped by Policing : 0
Total packets exceeding CIR : 0
Total packets exceeding PIR : 0
Errors
MAC/IP length inconsistencies : 1
Short IP packets received : 0
IP header checksum errors : 0
TTL failures : 103
MTU failures : 0
Statistics for Earl in Module 2
L2 Forwarding Engine
Total packets Switched : 288056821850
L3 Forwarding Engine
Total packets Processed : 140160389722 @ 7701 pps
Total packets L3 Switched : 34932321387 @ 1222 pps
Total Packets Bridged : 656015769
Total Packets FIB Switched : 34932321386
Total Packets ACL Routed : 0
Total Packets Netflow Switched : 1
Total Mcast Packets Switched/Routed : 484819394
Total ip packets with TOS changed : 2
Total ip packets with COS changed : 2
Total non ip packets COS changed : 0
Total packets dropped by ACL : 0
Total packets dropped by Policing : 0
Total packets exceeding CIR : 0
Total packets exceeding PIR : 0
Errors
MAC/IP length inconsistencies : 0
Short IP packets received : 0
IP header checksum errors : 0
TTL failures : 31737163
MTU failures : 0
Statistics for Earl in Module 3
L2 Forwarding Engine
Total packets Switched : 16262304705
L3 Forwarding Engine
Total packets Processed : 6051793996 @ 190 pps
Total packets L3 Switched : 3336102677 @ 111 pps
Total Packets Bridged : 18843981
Total Packets FIB Switched : 3336102676
Total Packets ACL Routed : 0
Total Packets Netflow Switched : 1
Total Mcast Packets Switched/Routed : 2696741885
Total ip packets with TOS changed : 2
Total ip packets with COS changed : 2
Total non ip packets COS changed : 0
Total packets dropped by ACL : 0
Total packets dropped by Policing : 0
Total packets exceeding CIR : 0
Total packets exceeding PIR : 0
Errors
MAC/IP length inconsistencies : 0
Short IP packets received : 0
IP header checksum errors : 0
TTL failures : 0
MTU failures : 0
Statistics for Earl in Module 5
L2 Forwarding Engine
Total packets Switched : 2610066755272
L3 Forwarding Engine
Total packets Processed : 1142796726231 @ 32470 pps
Total packets L3 Switched : 508644042257 @ 19228 pps
Total Packets Bridged : 17347783264
Total Packets FIB Switched : 508644042256
Total Packets ACL Routed : 0
Total Packets Netflow Switched : 1
Total Mcast Packets Switched/Routed : 181041275389
Total ip packets with TOS changed : 2
Total ip packets with COS changed : 2
Total non ip packets COS changed : 0
Total packets dropped by ACL : 2276418
Total packets dropped by Policing : 0
Total packets exceeding CIR : 0
Total packets exceeding PIR : 0
Errors
MAC/IP length inconsistencies : 1881
Short IP packets received : 0
IP header checksum errors : 0
TTL failures : 8688776
MTU failures : 0
Total packets L3 Processed by all Modules: 1710514690677 @ 55153 pps
xxx#sh mls statistics
Statistics for Earl in Module 1
L2 Forwarding Engine
Total packets Switched : 1126246359524
L3 Forwarding Engine
Total packets Processed : 421506362768 @ 17285 pps
Total packets L3 Switched : 204107756570 @ 9475 pps
Total Packets Bridged : 3318440478
Total Packets FIB Switched : 204107756569
Total Packets ACL Routed : 0
Total Packets Netflow Switched : 1
Total Mcast Packets Switched/Routed : 124896586736
Total ip packets with TOS changed : 2
Total ip packets with COS changed : 2
Total non ip packets COS changed : 0
Total packets dropped by ACL : 3022
Total packets dropped by Policing : 0
Total packets exceeding CIR : 0
Total packets exceeding PIR : 0
Errors
MAC/IP length inconsistencies : 1
Short IP packets received : 0
IP header checksum errors : 0
TTL failures : 103
MTU failures : 0
Statistics for Earl in Module 2
L2 Forwarding Engine
Total packets Switched : 288057472476
L3 Forwarding Engine
Total packets Processed : 140160704868 @ 9682 pps
Total packets L3 Switched : 34932368706 @ 1232 pps
Total Packets Bridged : 656016702
Total Packets FIB Switched : 34932368705
Total Packets ACL Routed : 0
Total Packets Netflow Switched : 1
Total Mcast Packets Switched/Routed : 484819408
Total ip packets with TOS changed : 2
Total ip packets with COS changed : 2
Total non ip packets COS changed : 0
Total packets dropped by ACL : 0
Total packets dropped by Policing : 0
Total packets exceeding CIR : 0
Total packets exceeding PIR : 0
Errors
MAC/IP length inconsistencies : 0
Short IP packets received : 0
IP header checksum errors : 0
TTL failures : 31737163
MTU failures : 0
Statistics for Earl in Module 3
L2 Forwarding Engine
Total packets Switched : 16262339592
L3 Forwarding Engine
Total packets Processed : 6051808557 @ 283 pps
Total packets L3 Switched : 3336113840 @ 133 pps
Total Packets Bridged : 18844001
Total Packets FIB Switched : 3336113839
Total Packets ACL Routed : 0
Total Packets Netflow Switched : 1
Total Mcast Packets Switched/Routed : 2696745262
Total ip packets with TOS changed : 2
Total ip packets with COS changed : 2
Total non ip packets COS changed : 0
Total packets dropped by ACL : 0
Total packets dropped by Policing : 0
Total packets exceeding CIR : 0
Total packets exceeding PIR : 0
Errors
MAC/IP length inconsistencies : 0
Short IP packets received : 0
IP header checksum errors : 0
TTL failures : 0
MTU failures : 0
Statistics for Earl in Module 5
L2 Forwarding Engine
Total packets Switched : 2610069540467
L3 Forwarding Engine
Total packets Processed : 1142797974401 @ 30466 pps
Total packets L3 Switched : 508644771724 @ 18373 pps
Total Packets Bridged : 17347786423
Total Packets FIB Switched : 508644771723
Total Packets ACL Routed : 0
Total Packets Netflow Switched : 1
Total Mcast Packets Switched/Routed : 181041389292
Total ip packets with TOS changed : 2
Total ip packets with COS changed : 2
Total non ip packets COS changed : 0
Total packets dropped by ACL : 2276418
Total packets dropped by Policing : 0
Total packets exceeding CIR : 0
Total packets exceeding PIR : 0
Errors
MAC/IP length inconsistencies : 1881
Short IP packets received : 0
IP header checksum errors : 0
TTL failures : 8688776
MTU failures : 0
Total packets L3 Processed by all Modules: 1710516850594 @ 57716 pps
2018-03-02 11:22 GMT+01:00 James Bensley <jwbensley at gmail.com>:
> On 1 March 2018 at 09:53, james list <jameslist72 at gmail.com> wrote:
> > xxx#show ibc
> > Interface information:
> > 5 minute rx rate 944000 bits/sec, 793 packets/sec
> > 5 minute tx rate 25000 bits/sec, 37 packets/sec
> ...
> > 2467023087 Packets out of 554699386 CEF Switched, 0 Packets out
> of 0
> > Tag CEF Switched
> > 3916625157 Packets Fast Switched
> ...
> > Potential/Actual paks copied to process level 228808364/225833216
> > (2975148 dropped, 265 spd drops)
> ...
> > MISTRAL ERROR COUNTERS
> ...
> > 2974883 total packets dropped on throttled interfaces (2954630
> low,
> > 16704 medium, 3549 high)
>
>
> >> On 1 March 2018 at 08:29, james list <jameslist72 at gmail.com> wrote:
> >> > Dear experts,
> >> > has anybody experienced a 100% SP CPU usage on C6500-Sup720
> >> > (12.2(33)SXI5)
> >> > with a lot of interrupts ?
> >> > The main process is Heartbeat.
> >> >
> >> > Cisco TAC is struggling in having an idea to sorting out the issue,
> they
> >> > are working since 3 days on it..
> >> >
> >> > STP is stable, no mac moving, no real issue found… maybe somebody
> >> > experienced the same due to something in particular?
>
> I've compared to a similar box I have, it has less control-plane
> traffic than yours it would seem. You have a decent amount of dropped
> packets which I guess to be expected if you have sustained 100% SP CPU
> utilisation.
>
> Do you have a lot of spanning-tree instances, HSRP/VRRP, multicast
> (various other control-plane stuff) running on this box?
>
> Have you confirmed there are no layer 2 loops?
>
> If you run "show mls statistics" several times do you see any error
> stats climbing very quickly?
>
> I would also run a netdr capture and check if the traffic in their is
> normal for your network.
>
> Cheers,
> James.
>
More information about the cisco-nsp
mailing list