[f-nsp] FCX High CPU Diagnostics
Derek Maxwell
derek.maxwell at chosentechgroup.com
Wed Apr 3 15:49:26 EDT 2019
Output of dm cpu:
idle pid 0 id 536918812 cpu 0 usage 0
main pid 1 id 536918812 cpu 0 usage 0
flash pid 0 id 536918812 cpu 0 usage 0
arrays at 2238d894 and 2238dd6c
from entry 78 to entry 35
loop cnt: 3395 3626 -> 231
read cnt: 57670 61592 -> 3922
loop rate = 231/430 = 0
rcode = 100 - (100*0/2550) = 100
99/100 busy, from 43 sec ago
sho ip bgp sum does not show any FIB programming
We have fewer than 10 routes in so we're definitely not exceeding any limits.
We have icmp redirects disabled
On 4/3/19, 12:35 PM, "foundry-nsp on behalf of Hunter Fuller" <foundry-nsp-bounces at puck.nether.net on behalf of hf0002 at uah.edu> wrote:
I don't know FCX, but on the ICX we have the command "dm cpu" that
gives more details. Lists the 5 most busy tasks on the system. Run it
two times 1 minute apart, if the first time gave you bogus info.
SSH at LAB-7450STACK#dm cpu
appl-task pid 1322 tid 0xa5270460 cpu usage 84%
console-task pid 1369 tid 0xa4a42460 cpu usage 0%
flash-task pid 1281 tid 0xb5fd6460 cpu usage 0%
sys-tmr-task pid 1280 tid 0xb5ff6460 cpu usage 0%
itc-task pid 1285 tid 0xb5f76460 cpu usage 0%
loop cnt: 567307594 569243809 -> 1936215
read cnt: 136318039 136318293 -> 254
loop rate = 1936215/270 = 7171
rcode_old = 100 - (100*7171/3900) = -83
rcode_log2= 100-(100*log2(7171)/log2(3900))=-7
1/100 busy, from 27 sec ago
--
Hunter Fuller
Router Jockey
VBH Annex B-5
+1 256 824 5331
Office of Information Technology
The University of Alabama in Huntsville
Network Engineering
On Wed, Apr 3, 2019 at 2:28 PM Tim Warnock <timoid at timoid.org> wrote:
>
> Are you trying to put more routes than can go into TCAM?
>
> Does 'show ip bgp sum' show any of your BGP sessions doing FIB programming?
>
> I don't know the FCX but a command from mlx/ce[s-r] is 'rcon lp', then run 'show task'. That will give you an idea if the LP is busy, if the command exists.
>
> > -----Original Message-----
> > From: foundry-nsp [mailto:foundry-nsp-bounces at puck.nether.net] On
> > Behalf Of Derek Maxwell
> > Sent: Thursday, 4 April 2019 5:14 AM
> > To: Jörg Kost <jk at ip-clear.de>; foundry-nsp <foundry-
> > nsp at puck.nether.net>
> > Subject: Re: [f-nsp] FCX High CPU Diagnostics
> >
> > So, something doesn't add up here:
> >
> > During a high CPU event, this is the output of show processes cpu:
> >
> > show processes cpu
> > Process Name 5Sec(%) 1Min(%) 5Min(%) 15Min(%) Runtime(ms)
> > ARP 0.00 0.00 0.00 0.00 1989
> > BGP 0.00 0.00 0.00 0.00 2809
> > DOT1X 0.00 0.00 0.00 0.00 0
> > GVRP 0.00 0.00 0.00 0.00 0
> > ICMP 0.00 0.00 0.00 0.00 1690
> > IP 1.10 1.37 1.21 0.58 128381
> > OSPF 0.00 0.00 0.00 0.00 0
> > RIP 0.00 0.00 0.00 0.00 20
> > STP 0.00 0.00 0.00 0.00 0
> > VRRP 0.00 0.00 0.00 0.00 0
> > Process Name 5Sec(%) 1Min(%) 5Min(%) 15Min(%) Runtime(ms)
> > IPv6 0.00 0.00 0.00 0.01 33559
> > ICMP6 0.00 0.01 0.01 0.01 31180
> > ND6 0.00 0.00 0.00 0.00 1461
> > RIPng 0.00 0.00 0.00 0.00 41
> > OSPFv3 0.00 0.00 0.00 0.00 0
> > IPV6_RX 0.00 0.00 0.00 0.00 181
> >
> > IP is the only high CPU process, but it's only at 1.1-1.37%.
> >
> >
> > This is the output of show cpu:
> > show cpu
> > 99 percent busy, from 289 sec ago
> > 1 sec avg: 99 percent busy
> > 5 sec avg: 99 percent busy
> > 60 sec avg: 99 percent busy
> > 300 sec avg: 99 percent busy
> >
> > It's 99% busy and the control plane is barely responding.
> >
> > Any thoughts on a hidden process that may be monopolizing the system?
> >
> >
> > On 4/3/19, 11:36 AM, "foundry-nsp on behalf of Derek Maxwell" <foundry-
> > nsp-bounces at puck.nether.net on behalf of
> > derek.maxwell at chosentechgroup.com> wrote:
> >
> > Yes and that's exactly what I was looking for. Thanks!
> >
> > Will continue to debug from here when it pops up again and see what I
> > find.
> >
> > -----Original Message-----
> > From: Jörg Kost <jk at ip-clear.de>
> > Sent: Wednesday, April 3, 2019 11:26 AM
> > To: Derek Maxwell <derek.maxwell at chosentechgroup.com>; foundry-nsp
> > <foundry-nsp at puck.nether.net>
> > Subject: Re: [f-nsp] FCX High CPU Diagnostics
> >
> > Does
> > show processes cpu
> > work on the FCX?
> >
> > On 3 Apr 2019, at 19:51, Derek Maxwell wrote:
> >
> > > Anyone have any thoughts on how to diagnose intermittent 99% CPU
> > > utilization on a FCX624? We're running Layer 3 code and high CPU usage
> > > seems to be causing BGP sessions to drop.
> > >
> > _______________________________________________
> > foundry-nsp mailing list
> > foundry-nsp at puck.nether.net
> > http://puck.nether.net/mailman/listinfo/foundry-nsp
> >
> >
> > _______________________________________________
> > foundry-nsp mailing list
> > foundry-nsp at puck.nether.net
> > http://puck.nether.net/mailman/listinfo/foundry-nsp
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp at puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
_______________________________________________
foundry-nsp mailing list
foundry-nsp at puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp
More information about the foundry-nsp
mailing list