[j-nsp] M20 PFE/SSB-E problems

[Jonas Frey]

Hi,

suddenly i am getting the following errors on a M20:
Mar 19 18:48:54  ffm2-edge /kernel: RT_PFE: NH IPC op 31 (CHANGE
AGGREGATE NEXTHOP) failed, err 6 (No Memory)
Mar 19 18:49:12  ffm2-edge last message repeated 2 times
Mar 19 18:49:17  ffm2-edge /kernel: RT_PFE: NH IPC op 31 (CHANGE
AGGREGATE NEXTHOP) failed, err 6 (No Memory)
Mar 19 19:01:14  ffm2-edge last message repeated 42 times
Mar 19 19:09:47  ffm2-edge last message repeated 39 times
Mar 19 19:19:39  ffm2-edge last message repeated 10 times
Mar 19 19:48:48  ffm2-edge /kernel: RT_PFE: NH IPC op 29 (ADD AGGREGATE
NEXTHOP) failed, err 6 (No Memory)
Mar 19 19:48:48  ffm2-edge /kernel: RT_PFE: NH IPC op 31 (CHANGE
AGGREGATE NEXTHOP) failed, err 6 (No Memory)
Mar 19 20:02:41  ffm2-edge /kernel: RT_PFE: RT msg op 1 (PREFIX ADD)
failed, err 1 (Unknown)

[messages are repeating over and over]

then sometime later:
Mar 19 20:02:41  ffm2-edge ssb RT: Failed prefix add IPv4:0 -
some.ip.addre.ss (radix add failed)
Mar 19 20:02:41  ffm2-edge ssb RT: Failed prefix add IPv4:0 -
some.ip.addre.ss, nh 11716, ifidx 68, nhifl 0, flag 0x0, cos 0
Mar 19 20:02:41  ffm2-edge ssb RT: Failed prefix add IPv4:0 -
some.ip.addre.ss (radix add failed)
Mar 19 20:02:41  ffm2-edge /kernel: RT_PFE: RT msg op 1 (PREFIX ADD)
failed, err 1 (Unknown)
Mar 19 20:02:41  ffm2-edge last message repeated 7 times
Mar 19 20:02:41  ffm2-edge ssb RT: Failed prefix add IPv4:0 -
some.ip.addre.ss, nh 11814, ifidx 76, nhifl 0, flag 0x0, cos 0
Mar 19 20:02:41  ffm2-edge ssb RT: Failed prefix add IPv4:0 -
some.ip.addre.ss (radix add failed)
Mar 19 20:02:42  ffm2-edge ssb RT: Failed prefix add IPv4:0 -
some.ip.addre.ss, nh 14090, ifidx 68, nhifl 0, flag 0x0, cos 0
Mar 19 20:02:42  ffm2-edge ssb RT: Failed prefix add IPv4:0 -
some.ip.addre.ss (radix add failed)
Mar 19 20:02:42  ffm2-edge ssb RT: Failed prefix add IPv4:0 -
some.ip.addre.ss, nh 14091, ifidx 68, nhifl 0, flag 0x0, cos 0


[repeating, beeing some.ip.addre.ss an advertised ip address inside our
ip space]

after that:
Mar 19 20:02:53  ffm2-edge ssb RT: Failed prefix add IPv4:0 -
some.ip.addre.ss (radix add failed)
Mar 19 20:02:53  ffm2-edge ssb RT: Failed prefix add IPv4:0 -
some.ip.addre.ss (radix add failed)
Mar 19 20:02:53  ffm2-edge ssb RT: Failed prefix delete IPv4:0 -
some.ip.addre.ss (unknown prefix)
Mar 19 20:02:53  ffm2-edge ssb RT: Failed prefix delete IPv4:0 -
some.ip.addre.ss (unknown prefix)
Mar 19 20:02:53  ffm2-edge ssb RT: Failed prefix delete IPv4:0 -
some.ip.addre.ss (unknown prefix)
Mar 19 20:02:53  ffm2-edge ssb RT: Failed prefix delete IPv4:0 -
some.ip.addre.ss (unknown prefix)
Mar 19 20:02:53  ffm2-edge ssb RT: Failed prefix delete IPv4:0 -
some.ip.addre.ss (unknown prefix)
Mar 19 20:02:53  ffm2-edge ssb RT: Failed prefix delete IPv4:0 -
some.ip.addre.ss (unknown prefix)

[repeating over and over]

and after this again the "change aggregate nexthop" error.

The SSB looks fine:

SSB status:
Slot 0 information:
  State                                 Master    
  Temperature                        35 degrees C / 95 degrees F
  CPU utilization                    21 percent
  Interrupt utilization               1 percent
  Heap utilization                   64 percent
  Buffer utilization                 58 percent
  Total CPU DRAM                     64 MB


As well as the RE:

  Slot 0:
    Current state                  Master
    Election priority              Master
    Temperature                 25 degrees C / 77 degrees F
    CPU temperature             27 degrees C / 80 degrees F
    DRAM                       768 MB
    Memory utilization          42 percent
   CPU utilization:
      User                       2 percent
      Background                 0 percent
      Kernel                     3 percent
      Interrupt                  1 percent
      Idle                      94 percent


Running 7.0R2.7

Regarding operational impact this error causes some IP addresses not
beeing able to be reached from outside. These IP addresses can be
reached by the router, e.g. a simple ping works. However nothing from
outside reaches them.

Anyone seen this before?

Regards,
Jonas