[f-nsp] [MLX-16] CAM lookup very slow

[Niels Bakker]

* youssef.ghorbal at gmail.com (Youssef Ghorbal) [Sun 03 Oct 2010, 20:25 CEST]:
> On a MLX-16, one of the Switch Fabric suffer from a very slow CAM 
>lookup. When I issue a command like :
> $> show cam ip 2/18 192.168.90.0 255.255.255.0
> It takes nearly 2/3 seconds to have the prompt back. When it's 
>instant on other modules. I don't have the problem on the same module 
>when it comes to ethernet CAM.
> This CAM lookup issue increases the network latency by a factor of 
>30. Ping RTT reach 30ms when it was 1ms in the past.
>
> The Line Processor (LP) of the module seems to be beasy (beasier than 
>the other modules)
> LP-2#show cpu-utilization
>... Usage average in the last 1 second  - 65 ...
>last 1 second  utilization = 65
>last 5 seconds utilization = 63
>last 1 minute  utilization = 66
>last 5 minutes utilization = 66

Is that 30ms increase in latency for traffic across the module after 
CAM programming for a particular destination has taken place?  Because 
CPU load on the module should absolutely not impact forwarding speeds.
Your "show cam ip" command likely is slow because the CPU load on the 
module is high, lookups should not be delayed like that.


>The task that seems to be a CPU hog is "main" (very usefull name...)

Yes, IronWare is quite monolithic.  Try looking at what packets are 
traversing this linecard's CPU.


>The ip CAM utilization ratio is 60% but I don't think that's the 
>origin of the problem.
>
>When I issue "rconsole 2" command I keep having this message logged in 
>the console :
>Can not log denied traffic[No L4 session can be created] (of course, 
>nothing regaring this message in any manuel)
>L4 session CAM has an utilization ration of 17% so there is no 
>exhaustion situation here...

I had this as well a while ago for no discernable reason.  I opened a 
case with Brocade TAC but it didn't go anywhere.  I'd suggest removing 
"log" from a few input access lists and see if that helps as a 
workaround.

Did you configure hardware-assisted dropping of logged packets on 
interfaces on that module?


>How can I go deeper in diagnostics ?
>Do you think it's a hardware issue ?

Very unlikely.


	-- Niels.

--