[j-nsp] Howto debug RTPERF_CPU_THRESHOLD_EXCEEDED

[Tom Eichhorn]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

I am currently fighting RTPERF_CPU_THRESHOLD_EXCEEDED messages in all
of my branch SRX devices, and I need some tip how to debug further:

As far as I understood, this messages are generated by the fwdd,
and so I thought I could find out whats going on with an event script:

te at gw.esn> show configuration event-options
policy hello-world {
    events RTPERF_CPU_THRESHOLD_EXCEEDED;
    then {
        execute-commands {
            commands {
                "request pfe execute command \"show threads\" target
fwdd";
            }
            output-filename pfe-event;
            destination local;
        }
    }
}
destinations {
    local {
        archive-sites {
            /var/tmp/;
        }
    }
}

That works quite well, but the output doesn't show any hint of high usage:

% cat gw.esn_pfe-event_20130306_210* | grep -i idle
GOT:   2 L  running   Idle                  1432/73824
1/565/979324106 ms 92%
GOT:   2 L  running   Idle                  1432/73824
1/565/979336591 ms 92%
GOT:   2 L  running   Idle                  1432/73824
0/565/979346145 ms 92%
GOT:   2 L  running   Idle                  1432/73824
1/565/979412535 ms 92%
GOT:   2 L  running   Idle                  1432/73824
1/565/979421720 ms 92%
GOT:   2 L  running   Idle                  1432/73824
1/565/979430940 ms 92%
GOT:   2 L  running   Idle                  1432/73824
1/565/979447566 ms 92%
GOT:   2 L  running   Idle                  1432/73824
1/565/979480812 ms 92%
GOT:   2 L  running   Idle                  1432/73824
0/565/979618783 ms 92%
GOT:   2 L  running   Idle                  1432/73824
1/565/979622447 ms 92%

The only hint which I have are the following messages:

Mar  6 21:05:22  gw.esn gw.esn Next-hop resolution requests from
interface 71 throttled

These are quite common in the log, if71 is the ipsec interface with a
/30 on it - but there is no reason for this message. I have grounded
all unused IPs with a discard route and for each IP I have also added
a static ARP entry, so that always either an ARP entry is available or
a route (eventually a discard default route).

Any hints here? Please don't point me out to jtac, I am quite sure they
don't go investigating without any direct network impacts..

Thanks,
Tom
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlE3r9kACgkQrUvjMoak8ZdUEwCfSqNvEugNxdhKxCFKMLlCMeQV
PMwAn33qyGnpkI0ZrECNb4EfuuJzFv6X
=GrF0
-----END PGP SIGNATURE-----