[c-nsp] high CPU with snmp IS THERE A REAL FIX

[Phil Mayers]

Jeff Fitzwater wrote:
> We are running 12.2.SXI on sup-720-cxl

Short answer: no, it's just slow.

My advice, and the solution we adopted - fetch the ARP table using an 
expect script via the CLI, which does not have this problem - we had 
mean RP CPU use drop a *huge* amount when I implemented that.

If I ever get the "netconf" interface working for CLI, it'll become an 
even more attractive solution.

You'd probably need an RFE to get the SNMP method "fixed", and I am not 
optimistic about your chances of success.

> 
> We use snmp getnext and getbulk to get the ARP table from a router  
> that has ~16K entries and it takes about 10min to complete, with  
> ROUTER CPU at 100%.   Our other routers have the same hardware and IOS  
> but have <10K entries and work fine.
> 
> This is an old problem with CISCO and I thought they might have  
> finally fixed it in SXI, but NOT.
> 
> The problem is the ARP entries are stored internally in a hashed  
> format and SNMP needs them in an ordered format, so for every snmp PDU  
> the CPU must get the entire ARP table and crunch it to create the  
> ordered PDU response.  This will take place for as many PDUs as it  
> takes to retrieve the entire table.   There does appear to be a point   
> (size of ARP table) at which it takes very long to retrieve the whole  
> table which seems to be around 12K but with a 16K table it takes  
> forever.

Yep - it's an exponential slowdown.

You could try the cseCefFib MIB (CISCO-SWITCH-ENGINE-MIB) but in our 
environment it was only ~10% faster, with an even higher CPU usage.

> 
> In the attached PDF from CISCO they explain the problem and also state  
> the if you turn on CEF (has always been on for long time) that it is  
> much faster since the FIB is already in a lexical order that snmp  
> likes.   Since CEF is always on, why does it still take so long.

The explanation cisco offer has two problems:

  1. it really dates from the days of CPU routers where the CEF 
structure lives primarily in CPU RAM, as opposed to the 6500s where it's 
formatted to be optimal for the hardware

  2. the CEF table may be "in order" but you still have to do some 
sorting - namely for the ifIndex, because the ipNetToMedia table is 
indexed by:

ifIndex.ip0.ip1.ip2.ip3