[c-nsp] CoPP IS-IS traffic on N7k

[Lincoln Dale]

On 25/01/2011, at 12:23 AM, Matthew Melbourne wrote:

> Thanks Lincoln.
> 
> I've manually added the CoPP entries for IS-IS/FabricPath to the
> policy, but we are still seeing IS-IS adjacencies drop, so I'm not
> convinced CoPP is the issue here.

you can ascertain if CoPP may be limiting you by looking at whether you have any drops recorded due to CoPP.  "show policy-map interface control"


> Very occasionally we're seeing:
> 
> %MODULE-4-MOD_WARNING: Module 2 (serial: JAF1419BGPK) reported warning
> due to EOBC heartbeat failure in device 10 (device error 0xc000904d)

suggest you log a support case with TAC on this.  you should not be getting that at all.

> 
> However, these alerts are not co-incident with the loss of IS-IS
> adjacencies (though these alerts are seen for all modules), and the
> helpful advice so far is to "reseat all the modules to clear the ASIC
> buffers".

hmm, i don't see how reseating a card can "clear an ASIC buffer". :)

On 25/01/2011, at 1:12 AM, Matthew Melbourne wrote:

> As a follow-up to this, I've discovered that the CPU utilisation for
> 'netstack' process jumps to 100% for ~5 mins.

'netstack' is the software process that implements the IP / TCP stack for received frames hitting control-plane.
if netstack cpu is high for an extended period then it implies you have excessive traffic hitting that

key is probably to find out what traffic is hitting it.

did you disable any h/w rate limiters or CoPP at all?  or increase the rates on any of these?


use the embedded ethanalyzer (wireshark) to snoop the inband port would probably be the easiest way.
use "ethanalyzer local interface inband [..]"

if you don't want to hang around to wait for it to happen you could create an EEM action that is triggered on high cpu to do this for you, e.g.

	event manager applet debug_highcpu
	  event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.6.1 get-type exact entry-op gt entry-val 80 exit-op lt exit-val 40 poll-interval 30
	  action 1.0 syslog msg high cpu load condition detecteds
	  action 2.0 cli show clock >> bootflash:high_cpu.txt
	  action 2.5 cli show process cpu sort >> bootflash:high_cpu.txt
	  action 3.0 cli show hardware internal cpu-mac inband stats >> bootflash:high_cpu.txt
	  action 3.0 cli ethanalyzer local interface inband limit-captured-frames 200 >> bootflash:high_cpu.txt

something like that at least then gives you (or TAC) something else to go on.

> High ping times are
> observed to the L3 connected addresses (over which IS-IS runs).

sending/replying to icmp-echo is a function of control plane.  not unexpected that you'll get high(er) latencies associated with that.

> This is almost certainly responsible for the loss of IS-IS adjacencies.

sounds like it.


cheers,

lincoln.