[c-nsp] 7604/Sup720 not MLS/CEF switching

[Sukumar Subburayan (sukumars)]

To answer your last question, since the packets that are punted to
software for switching are 
handled by one of the EARL7 rate-limiters, which don't have counters and
also you cannot see what packets,
are being punted to software, the best option would be use 

CPU-SPAN, to SPAN the traffic destined to RP-CPU and analyse that.


sukumar
 

 
Oh well.  I found the problem - someone leaked too many prefixes, and
it's  

%MLSCEF-SP-7-FIB_EXCEPTION: FIB TCAM exception for IPv4 unicast, Some
routes will be software switched.

Dunno why it's showing *these* symptoms, affecting some interfaces more
than others.  But still I'm interested in finding out how to see what
packets are not being MLS/CEF-switched, and why, for the next round of
debugging :-)



-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net
[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Gert Doering
Sent: Friday, January 25, 2008 8:07 AM
To: cisco-nsp at puck.nether.net
Subject: [c-nsp] 7604/Sup720 not MLS/CEF switching

Hi,

I could use a hint to start nailing this down.

We have two 7604/Sup720s with 12.2(18)SXF7 here, doing a pretty similar
traffic load (about 2-3 Gbit/s aggregate), and similar traffic pattern.

IPv4, IPv6, MPLS, netflow export for IPv4.

One of the boxes is running at 1-2% CPU, the other one is running at
60-80% (which started at 22:18 yesterday evening, with no significant
change in traffic patterns).

So, it's moving packets with a CPU not meant to be used for this.  

So I've checked two interfaces with very similar usage patterns (audio
streaming of life radio, long-lasting flows with medium-to-large packets
sizes), and there's a big difference in the percentage here:

vlan1700, about 4% "not MLS/CEF switched":

     Protocol       Path    Pkts In   Chars In   Pkts Out  Chars Out
           IP    Process      25150   24734247          0          0
            Cache misses          0
                    Fast 1328140746 1350996135423        191      58674
               Auton/SSE 30723864532 30882213532050 18184117236
1335974238797

vlan4062, about 0.1% "not MLS/CEF switched":

     Protocol       Path    Pkts In   Chars In   Pkts Out  Chars Out
           IP    Process     368914   54599634   31636639 3543640264
            Cache misses          0
                    Fast 1670054191 1924596882515        168       9913
               Auton/SSE 1029709651247 1137649776167566 229040036204
16614962888496

there's difference on L2 for these interfaces (4062 is coming in via a
dedicated port, 1700 is coming in via a trunk port), but I don't think
this should make any difference.

Most of the egress traffic for this is going via a L3 port-channel, or
via a single L3 port.  For both VLANs.


Traffic level is about 400 Mbit on vlan 1700, 500 Mbit on vlan 4062,
most of it "incoming".  No big difference here either.  Similar PPS
levels, about 50.000 pps incoming.

This is how vlan1700 looks like:

interface Vlan1700
 description Streaming2/Trust (an1)
 ip address 194.97.x.y 255.255.255.240
 ip verify unicast source reachable-via rx allow-default  ip flow
ingress  no mop enabled end


Something is funny here... - so - how do I start figuring out why 1/20
of those packets are not being MLS/CEF switched?


Oh well.  I found the problem - someone leaked too many prefixes, and
it's  

%MLSCEF-SP-7-FIB_EXCEPTION: FIB TCAM exception for IPv4 unicast, Some
routes will be software switched.

Dunno why it's showing *these* symptoms, affecting some interfaces more
than others.  But still I'm interested in finding out how to see what
packets are not being MLS/CEF-switched, and why, for the next round of
debugging :-)

gert


--
USENET is *not* the non-clickable part of WWW!
 
//www.muc.de/~gert/
Gert Doering - Munich, Germany
gert at greenie.muc.de
fax: +49-89-35655025
gert at net.informatik.tu-muenchen.de