[j-nsp] JunOS 16.2R2.8 High CPU caused by python

[Anderson, Charles R]

Yes.  It was fixed in a later release.  Perhaps try 16.2R2-S8 if you don't want to change to a later version.

On Wed, Mar 27, 2019 at 10:59:09AM +1000, Philip Smith wrote:
> Hi everyone,
> 
> Has anyone seen anything like this before? Searching on Google etc has
> revealed nothing.
> 
> For a few months now, MX240s running 16.2R2.8 have showed one of the RE
> CPUs running at around 90%. It has not affected the MX80s in the same
> network running the same OS and having essentially the same config
> (simple dual stack network, IS-IS/BGP, that's about it).
> 
> And none of the 240s started doing this at exactly the same time. And
> there are no configuration changes around the times the CPU jumped.
> 
> Here is an example:
> 
> philip at TCR> show system processes extensive
> last pid: 33210;  load averages:  1.23,  1.15,  1.14  up 343+02:08:01
> 06:42:04
> 144 processes: 5 running, 138 sleeping, 1 waiting
> 
> Mem: 284M Active, 1474M Inact, 183M Wired, 12M Cache, 91M Buf, 32M Free
> Swap: 4096M Total, 319M Used, 3777M Free, 7% Inuse
> 
> 
>   PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME    WCPU COMMAND
> 11473 root        1  52    0   724M  6048K piperd 3408.8  79.05% python
>    10 root        1 155 ki31     0K    12K RUN    2883.3   2.20% idle
> 11367 root        2 -26  r26   813M  8904K nanslp 182.5H   1.66% chassisd
> 11825 root        3  20    0   903M 39936K kqread  36.9H   0.49% rpd
> 11387 root        1  20    0   784M 29580K select 119.3H   0.39% mib2d
> 11863 root        1  20    0   738M 10084K select  76.4H   0.29% snmpd
> 
> 
> Almost 80% caused by python. I'm not doing any automation (that I've
> knowingly set up).
> 
> Jumping into a shell, I see this:
> 
> philip at TCR> start shell
> % ps ax | grep python
> 11473  -  R    204530:44.49 /usr/bin/python /usr/libexec/icmd/icmd.py
> 33362  0  S+        0:00.00 grep python
> 
> ICMD according to the docs is the "internal communication health monitor
> daemon".
> 
> Weirdly, I only see the ICMD running on the MX240s, not the MX80s. Even
> though the docs suggest it applies to all MX.
> https://www.juniper.net/documentation/en_US/junos/information-products/topic-collections/release-notes/16.2/topic-113675.html
> 
> Any clues at all?
> 
> Is this just a reboot to make it go away? (Although one has been
> rebooted recently, and after about two weeks of uptime, the CPU jumped
> up to 85% and has stayed there.)
> 
> Thanks!
> 
> philip