[f-nsp] [MLX-16] CAM lookup very slow

[Youssef Ghorbal]

Hello,

 On a MLX-16, one of the Switch Fabric suffer from a very slow CAM
lookup. When I issue a command like :
 $> show cam ip 2/18 192.168.90.0 255.255.255.0
 It takes nearly 2/3 seconds to have the prompt back. When it's
instant on other modules. I don't have the problem on the same module
when it comes to ethernet CAM.
 This CAM lookup issue increases the network latency by a factor of
30. Ping RTT reach 30ms when it was 1ms in the past.

 The Line Processor (LP) of the module seems to be beasy (beasier than
the other modules)
 LP-2#show cpu-utilization
... Usage average in the last 1 second  - 65 ...
last 1 second  utilization = 65
last 5 seconds utilization = 63
last 1 minute  utilization = 66
last 5 minutes utilization = 66

on other LP :
LP-1#show cpu-utilization
... Usage average in the last 1 second  - 1 ...
last 1 second  utilization = 1
last 5 seconds utilization = 1
last 1 minute  utilization = 1
last 5 minutes utilization = 1

The task that seems to be a CPU hog is "main" (very usefull name...)
LP-2#show task
Task Name   Pri   State     PC     Stack  Size  CPU Usage(%) task id  task vid
----------  ---  -------  ---------  ----------  -----  --------  ----- -----
       idle 0    ready  0004ebfc  00604fb0  4096    0           0        0
    monitor 20    wait  0004ea70  00602e08  16384   0           0        0
         wd 31    wait  0004ea70  0060ef50  8192    0           0        0
      flash 17    wait  0004ea70  00614f48  8192    0           0        0
        dbg 30    wait  0004ea70  00611f00  8192    0           0        0
       main 3    ready  01ab6584  2061bf08  131072  99          0        1
   i2c_task 3     wait  0004ea70  24512328  1024    0           0        1

The ip CAM utilization ratio is 60% but I don't think that's the
origin of the problem.

When I issue "rconsole 2" command I keep having this message logged in
the console :
Can not log denied traffic[No L4 session can be created] (of course,
nothing regaring this message in any manuel)
L4 session CAM has an utilization ration of 17% so there is no
exhaustion situation here...

How can I go deeper in diagnostics ?
Do you think it's a hardware issue ?

Youssef Ghorbal