[nsp] MSFC2 128,000 route limitation

[Matt Buford]

Excellent.  Thank you.  This type of detailed technical explanation is
exactly what I needed.  TAC always gave me descriptions that weren't
detailed enough for me to figure out exactly what was happening.  It was
clear that something was filling up related to too many routes, but that was
as detailed as it got.

Is there any way to see how much memory is used (or free) for the adjacency
table?  As I mentioned before, these boxes didn't really have anything but
dual bgp feeds (of full Internet routes) and OSPF providing two equal cost
paths to each BGP next-hop.  This doesn't seem like a particularly large or
unusual situation to me.  If anything I see this as more of a starting point
configuration for initial deployment - that will only get larger with time
and additional customers.  I understand that the command "set mls cef
per-prefix-stats disable" has reduced this (by half?) but I'd sleep better
if I knew how close I was to hitting this limitation again.

These specific switches are running hybrid.  This decision was made because
this situation calls for mostly switching - with very few router interfaces.
Basically there's just two upstream vlan interfaces (two backbones) then a
downstream customer interface.  Other than those interfaces, its all
switching.  CatOS's "hitless" upgrades plus the ability to upgrade the
routers without affecting switching seemed a big plus.  This allows router
upgrades without affecting even customers single-homed directly to these
switches.  Would you say the architectural improvements (such as the change
in adjacency table handling you mentioned) are such that you recommend
native IOS in all situations?

----- Original Message -----
From: "Ian Cox" <icox@cisco.com>
To: "Matt Buford" <matt@overloaded.net>; <cisco-nsp@puck.nether.net>
Sent: Thursday, August 22, 2002 8:15 PM
Subject: Re: [nsp] MSFC2 128,000 route limitation


>
> The bug you site has nothing to do with filling up the FIB tcam, it has to
> do with filling up the memory that contains the adjacency rewrite
> information. Filling up either of these two resources will cause problems.
> If the problem you are running into is the DDTs you refer to then an
> accurate description of the problem is:
>
> [snip]
>
> only with a large network configuration in which many prefixes have
> multiple paths to them resulting in adjacency exception condition
> (condition where the NMP runs of adjacency table ) on the NMP. This
happens
> because we do not share adjacencies among prefixes even if they have the
> same multiple paths ... Because of constant network updates, some of the
> adjacencies get deleted and when the NMP comes of out adj exception, we
> issue a reload of FIB/ADJ table . This cycles go on and on resulting in
the
> high CPU on NMP.
>
> [end snip]
>
> If you only have 50k routes, then it only consume 50k TCAM entries. The
> structure of the forwarding information looks like this:
>
> FIB TCAM               Adjacency Table
> +-----------+          +------------------+
> | 1.0.0.0/8 | --1:N--< | path1  rewrite 1 |
> | 2.1.0.0/16|          | path2  rewrite 2 |
> |           |          | path3  rewrite 3 |
> +-----------+          +------------------+
> 256k entries            256k entries
>
> If you have two parallel paths and you had 50k entries with CatOS you
would
> consume 50k x 2 entries in the adjacency table. In IOS for the Catalyst
> 6000 this is done differently and you would consume just 2 entries since
> the prefixes can share adjacency table entries.
>
>
> I have not dealt with CatOS on the platform for over 18 months, only IOS
> for the Catalyst 6000. IOS for the Catalyst 6000 does not have this
> problem, it handles the adjacency table programming by allowing prefixes
to
> share adjacencies entries. The way to show how much fib tcam has been
> consumed on IOS for the Catalyst 6000 is:
>
> tromso#sh mls cef summary
>
> tromso-sp#
> Total CEF switched packets:  0000019424896725
> Total CEF switched bytes:    0001857776306791
> Total routes:                110568
>      IP unicast routes:       110555
>      IPX routes:              0
>      IP multicast routes:     13
> tromso#
>
>
> Looking at the manual the command for CatOS is "show mls cef" to get the
> equivalent information.
>
> To get the number of entries used in the adjacency table for IOS for
> Catalyst 6000 use:
>
> tromso#sh mls cef adjacency count
>
> tromso-sp#
> Total adjacencies:           24
> tromso#
>
>
> The only similar count for adjacency usage I can see for a system running
> CatOS is under "sh polaris fibmgr usage"
>
>
> Ian
>
>
>
> At 06:24 PM 8/22/2002 -0400, Matt Buford wrote:
> >On Wed May 15 2002 - 07:53:32 EDT, Ian Cox wrote:
> >
> > > The TCAM that holds the FIB table is capable of holding 256,000
entries.
> > > Without unicast RPF checking turned on the maximum number of unicast
> > > entries that can be held in the hardware FIB table is 244,000. The
> > > remaining 12,000 entries are reserved for multicast routes. If unicast
RPF
> > > checking is enabled then the number of routes that are held in the
TCAM is
> > > halved.
> >
> > > You can exceed the capacity of the hardware forwarding table, and the
> > > consequences are that the routes that are not programmed into the TCAM
> >that
> > > holds the FIB table will be switched in software by the MSFC2 / RP.
> >
> >I have apparently ran into this limitation, with much worse consequences
> >(running Sup2/MSFC2 hybrid).  The supervisor CPU shot up to 100%, and all
> >updates from the MSFC to the supervisor/PFC stopped.  This happened in
both
> >of a pair of redundant 6500s, bringing both down and leaving me unable to
> >bring them back up with a full routing table.
> >
> >Cisco TAC found bug cscdw89942, and said the internal notes recommend
using
> >the "set mls cef per-prefix-stats disable" to reduce the number of
entries.
> >
> >It appears that at this point the limitiation is not something to take
> >lightly.  Reaching it (at least under Hybrid) apparently brings
everything
> >down.  There is no software yet available that fixes this, and the only
> >workaround is to take measures to reduce your CEF table size (such as
> >turning off per-prefix-stats).
> >
> >For perspective, the routers that failed each see two BGP feeds of full
> >Internet routes, as well as about 12 OSPF routes (each of which has 2 or
3
> >paths to get there).  This doesn't seem like a particularly large number
of
> >routes to me, however it certainly passes the limit listed in the bug of
> >50,000 routes with dual paths.
> >
> >Is there anywhere I can get a count of the actual current number of
entries
> >and/or space free, or is the only way to tell to show the cef table size
and
> >manually figure out if you need to multiply it if you have unicast RPF
on,
> >then make sure that is less than 244,000?  I want to go through all my
6500s
> >and make sure I'm not about to hit the limit on any of them (some are
hybrid
> >and some are native).  The thought of  all my 6500s falling over at once
and
> >staying down because I reached the maximum limit on routes scares me
> >greatly.
>