[f-nsp] [MLX-16] CAM lookup very slow

[Youssef Ghorbal]

On Mon, Oct 4, 2010 at 12:02 PM, Niels Bakker
<niels=foundry-nsp at bakker.net> wrote:
> * youssef.ghorbal at gmail.com (Youssef Ghorbal) [Sun 03 Oct 2010, 20:25 CEST]:
>>
>> On a MLX-16, one of the Switch Fabric suffer from a very slow CAM lookup.
>> When I issue a command like :
>> $> show cam ip 2/18 192.168.90.0 255.255.255.0
>> It takes nearly 2/3 seconds to have the prompt back. When it's instant on
>> other modules. I don't have the problem on the same module when it comes to
>> ethernet CAM.
>> This CAM lookup issue increases the network latency by a factor of 30.
>> Ping RTT reach 30ms when it was 1ms in the past.
>>
>> The Line Processor (LP) of the module seems to be beasy (beasier than the
>> other modules)
>> LP-2#show cpu-utilization
>> ... Usage average in the last 1 second  - 65 ...
>> last 1 second  utilization = 65
>> last 5 seconds utilization = 63
>> last 1 minute  utilization = 66
>> last 5 minutes utilization = 66
>
> Is that 30ms increase in latency for traffic across the module after CAM
> programming for a particular destination has taken place?  Because CPU load
> on the module should absolutely not impact forwarding speeds.
> Your "show cam ip" command likely is slow because the CPU load on the module
> is high, lookups should not be delayed like that.

In fact, this makes sense.

>> The task that seems to be a CPU hog is "main" (very usefull name...)
>
> Yes, IronWare is quite monolithic.  Try looking at what packets are
> traversing this linecard's CPU.

How can I "look" at what packets are traversion this Linecard's CPU ?

>> The ip CAM utilization ratio is 60% but I don't think that's the origin of
>> the problem.
>>
>> When I issue "rconsole 2" command I keep having this message logged in the
>> console :
>> Can not log denied traffic[No L4 session can be created] (of course,
>> nothing regaring this message in any manuel)
>> L4 session CAM has an utilization ration of 17% so there is no exhaustion
>> situation here...
>
> I had this as well a while ago for no discernable reason.  I opened a case
> with Brocade TAC but it didn't go anywhere.  I'd suggest removing "log" from
> a few input access lists and see if that helps as a workaround.
>
> Did you configure hardware-assisted dropping of logged packets on interfaces
> on that module?

I don't get your point here ? what do you mean exactly by
"hardware-assisted dropping of logged packets on interfaces"

Youssef Ghorbal