[c-nsp] Sup720 software forwarding

[Phil Mayers]

On 03/08/2013 07:46 AM, Peter Rathlev wrote:
> Theoretically, if one would happen to have a Sup720 that does software
> forwarding, how is it that one can check what the reason for punts is?

An excellent question. In the past, TAC have found stuff like this with 
ELAM captures, but it's never been obvious to me how they inferred the 
"cause" from the ELAM output; I assume they have a cheat sheet somewhere 
with a bunch of hex values on it!

If there's no obvious cause, it could be a TCAM mis-program; we've had a 
series of these over the last few years where the box sporadically fails 
to correctly program the PFC or one of the DFCs. It seems this condition 
is completely invisible except for its effects - that is, the Sup 
doesn't seem to be able to detect it. I guess this means the HW FIB can 
be written but not read?

Anyway, these can be a super-bitch to troubleshoot; we've seen ones that 
affect a single IPv4 prefix, ones that affect a single MPLS label, and 
ones that affect a single adjacency entry. They're usually triggered by 
something local to the box (e.g. our last was triggered by moving an SVI 
from one VRF to another) - to the best of my knowledge we've never seen 
one triggered by an "external" event e.g. routing update.

Typically a linecard or chassis reboot is the only way to clear them; no 
amount of "clear blah" will help, which is extraordinarily irritating...

>
> A netdr capture shows packets like this:
>
> ------- dump of incoming inband packet -------
> interface Vl212, routine mistral_process_rx_packet_inlin, timestamp 00:00:33
> dbus info: src_vlan 0xD4(212), src_indx 0x142(322), len 0x47(71)
>    bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
>    3C020400 00D40400 01420000 47080000 00060050 860FFF7C 00000000 03800000
> mistral hdr: req_token 0x0(0), src_index 0x142(322), rx_offset 0x76(118)
>    requeue 0, obl_pkt 0, vlan 0xD4(212)
> destmac 00.00.0C.07.AC.02, srcmac 00.50.56.8A.49.15, protocol 0800
> protocol ip: version 0x04, hlen 0x05, tos 0x98, totlen 53, identifier 6448
>    df 1, mf 0, fo 0, ttl 128, src 10.83.12.125, dst 10.83.3.16
>      tcp src 57615, dst 1531, seq 3896459602, ack 2194696301, win 509 off 5 checksum 0x849A ack psh

I assume Vl212 is unremarkable in configuration? And that 10.83.3.16 
doesn't point back out of Vl212?

> I just can't see why this packet is punted. We see ~80 kpps inbound on
> the IBC interface. The only thing that seem out of place is this:

Are they from a variety of input interfaces? Can you seen anything 
common about the next-hops for things which are punting, versus things 
that aren't punting?