[f-nsp] FCX High CPU Diagnostics

[Hunter Fuller]

I don't know FCX, but on the ICX we have the command "dm cpu" that
gives more details. Lists the 5 most busy tasks on the system. Run it
two times 1 minute apart, if the first time gave you bogus info.

SSH at LAB-7450STACK#dm cpu
appl-task pid 1322 tid 0xa5270460 cpu usage 84%
console-task pid 1369 tid 0xa4a42460 cpu usage 0%
flash-task pid 1281 tid 0xb5fd6460 cpu usage 0%
sys-tmr-task pid 1280 tid 0xb5ff6460 cpu usage 0%
itc-task pid 1285 tid 0xb5f76460 cpu usage 0%
loop cnt: 567307594 569243809 -> 1936215
read cnt: 136318039 136318293 -> 254
loop rate = 1936215/270 = 7171
rcode_old = 100 - (100*7171/3900) = -83
rcode_log2= 100-(100*log2(7171)/log2(3900))=-7
1/100 busy, from 27 sec ago


--
Hunter Fuller
Router Jockey
VBH Annex B-5
+1 256 824 5331

Office of Information Technology
The University of Alabama in Huntsville
Network Engineering


On Wed, Apr 3, 2019 at 2:28 PM Tim Warnock <timoid at timoid.org> wrote:
>
> Are you trying to put more routes than can go into TCAM?
>
> Does 'show ip bgp sum' show any of your BGP sessions doing FIB programming?
>
> I don't know the FCX but a command from mlx/ce[s-r] is 'rcon lp', then run 'show task'. That will give you an idea if the LP is busy, if the command exists.
>
> > -----Original Message-----
> > From: foundry-nsp [mailto:foundry-nsp-bounces at puck.nether.net] On
> > Behalf Of Derek Maxwell
> > Sent: Thursday, 4 April 2019 5:14 AM
> > To: Jörg Kost <jk at ip-clear.de>; foundry-nsp <foundry-
> > nsp at puck.nether.net>
> > Subject: Re: [f-nsp] FCX High CPU Diagnostics
> >
> > So, something doesn't add up here:
> >
> > During a high CPU event, this is the output of show processes cpu:
> >
> > show processes cpu
> > Process Name   5Sec(%)   1Min(%)   5Min(%)   15Min(%)   Runtime(ms)
> > ARP              0.00      0.00      0.00      0.00           1989
> > BGP              0.00      0.00      0.00      0.00           2809
> > DOT1X            0.00      0.00      0.00      0.00              0
> > GVRP             0.00      0.00      0.00      0.00              0
> > ICMP             0.00      0.00      0.00      0.00           1690
> > IP               1.10      1.37      1.21      0.58         128381
> > OSPF             0.00      0.00      0.00      0.00              0
> > RIP              0.00      0.00      0.00      0.00             20
> > STP              0.00      0.00      0.00      0.00              0
> > VRRP             0.00      0.00      0.00      0.00              0
> > Process Name   5Sec(%)   1Min(%)   5Min(%)   15Min(%)   Runtime(ms)
> > IPv6             0.00      0.00      0.00      0.01          33559
> > ICMP6            0.00      0.01      0.01      0.01          31180
> > ND6              0.00      0.00      0.00      0.00           1461
> > RIPng            0.00      0.00      0.00      0.00             41
> > OSPFv3           0.00      0.00      0.00      0.00              0
> > IPV6_RX          0.00      0.00      0.00      0.00            181
> >
> > IP is the only high CPU process, but it's only at 1.1-1.37%.
> >
> >
> > This is the output of show cpu:
> > show cpu
> > 99 percent busy, from 289 sec ago
> > 1   sec avg: 99 percent busy
> > 5   sec avg: 99 percent busy
> > 60  sec avg: 99 percent busy
> > 300 sec avg: 99 percent busy
> >
> > It's 99% busy and the control plane is barely responding.
> >
> > Any thoughts on a hidden process that may be monopolizing the system?
> >
> >
> > On 4/3/19, 11:36 AM, "foundry-nsp on behalf of Derek Maxwell" <foundry-
> > nsp-bounces at puck.nether.net on behalf of
> > derek.maxwell at chosentechgroup.com> wrote:
> >
> >     Yes and that's exactly what I was looking for. Thanks!
> >
> >     Will continue to debug from here when it pops up again and see what I
> > find.
> >
> >     -----Original Message-----
> >     From: Jörg Kost <jk at ip-clear.de>
> >     Sent: Wednesday, April 3, 2019 11:26 AM
> >     To: Derek Maxwell <derek.maxwell at chosentechgroup.com>; foundry-nsp
> > <foundry-nsp at puck.nether.net>
> >     Subject: Re: [f-nsp] FCX High CPU Diagnostics
> >
> >     Does
> >     show processes cpu
> >     work on the FCX?
> >
> >     On 3 Apr 2019, at 19:51, Derek Maxwell wrote:
> >
> >     > Anyone have any thoughts on how to diagnose intermittent 99% CPU
> >     > utilization on a FCX624? We're running Layer 3 code and high CPU usage
> >     > seems to be causing BGP sessions to drop.
> >     >
> >     _______________________________________________
> >     foundry-nsp mailing list
> >     foundry-nsp at puck.nether.net
> >     http://puck.nether.net/mailman/listinfo/foundry-nsp
> >
> >
> > _______________________________________________
> > foundry-nsp mailing list
> > foundry-nsp at puck.nether.net
> > http://puck.nether.net/mailman/listinfo/foundry-nsp
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp at puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp