[j-nsp] acx5048 message - fpc0 brcm_pkt_fp_thread

[Aaron Gould]

Wanted to circle back with y'all... this message...

fpc0 brcm_pkt_fp_thread:700 FP thread is unscheduled for more than 30ms.
Sleep time: 44ms.^M

...seemed to be coupled with, or somewhat related to other messages seen
within a 12 hour period of...

Nov 28 12:30:10  my-ACX5048 /kernel: swapping out newsyslog[80273] target:
12979 free: 8908
Nov 28 12:30:10  my-ACX5048 /kernel: swapping out sh[80271] target: 12979
free: 8908
Nov 28 12:30:10  my-ACX5048 /kernel: swapping out smihelperd[1722] target:
12979 free: 8908
Nov 28 12:30:10  my-ACX5048 /kernel: swapping out pmond[1720] target: 12979
free: 8908
Nov 28 12:30:10  my-ACX5048 /kernel: swapping out tnp.sntpd[1712] target:
12979 free: 8908
...several others....
Nov 28 12:30:10  my-ACX5048 /kernel: swapping out sdk-mgmtd[1562] target:
12979 free: 8908
Nov 28 12:30:10  my-ACX5048 /kernel: swapping out inetd[1558] target: 12979
free: 8908
Nov 28 12:30:10  my-ACX5048 /kernel: swapping out mgd[1557] target: 12979
free: 8908
Nov 28 12:30:10  my-ACX5048 /kernel: swapping out watchdog[1553] target:
12979 free: 8908
Nov 28 12:30:10  my-ACX5048 /kernel: swapping out cron[1472] target: 12979
free: 8908


...interestingly, when I saw these messages, there was occasions about 10-12
hours later, serious customer-affecting issues on a couple of my
ACX5048's... hard to telnet/ssh to it, customer complaints, etc.

Btw, we saw memory ramping up on these ACX5048's over a years time,
gradually in an ever-increasing manner... looking much like a memory leaking
process

Much appreciation to the Juniper JTAC for finding jdhcpd with "show system
processes extensive no-forwarding" and then instructing me to bounce it with
"restart dhcp-service gracefully"

Whew.  Memory was released immediately!  From 85% to 50%.  That'll buy me
some time to work on upgrading Junos to a fix version/jtac recommended
release.

- Aaron