[c-nsp] network rebuild questions

[Bill Desjardins]

Hi all,

First, sorry a bit long here with a bunch of questions...

I have a small ISP (~50 customers) and am in the process of
building out a new network. traffic runs ~250Mbit in/out
with ~230k PPS in/230k PPS out. 3 providers, all gig uplinks. the
network does nothing fancy, no MPLS, no QOS, etc. just basic routing.

the soon to be smelted network consists of a pair of 6506's/SUP1's which
each get a full BGP feed and contain customer SVI's w/HSRP. connected to
these are 2 5500's for customer connections. all layer2.

the new core tier is a pair of 6509/(2)SUP2/(2)6516+DFC/6548/(2)SFM
with 5 pairs of 3550's (12T's & G's) hanging off those via p2p fiber
in a U-shape configuration for the access. eg:

coreA <--> 3550 <--> 3550 <--> coreB.

there are 4 2651XM/256MB route-reflectors which handle only customer
assigned subnets from the 3550's with a few customers doing ebgp to the
the access layer. the RR's also do some basic traffic engineering via
community assignments and get BOGON bgp feeds.

The IGP is ospf and handles only loopbacks and the p2p links between
everything. all /31's and all from a single /26.

for my uplinks, I've decided to go with separate border routers for
each provider. I am planning to use 6503/SUP2/(2)6516 taking limited
BGP (~240k routes) + default. should I be peering each brdr with each
other? if so, should they be p2p direct to each other in addition to
connected via the core?

wrt to default routes, should they come from the brdrs via ospf
default-originate or would propagating default via ibgp to the edge be
preferred?

what's your experience with SFM+DFC's? are they worth the extra complexity?

some sup2 specific questions....

I am currently running 12.2(18)SXF14 and would like to hear any thoughts
on your experiences with the stability of the SXF branch for the SUP2's.

at my traffic level's, how much further can the sup2's go?
is there any critical traffic level/pps line I shouldnt cross?

How far can I push the FIB table? iirc, the pfc2 can handle 259k cef
entries. Ive got my bgp tuned to ~245k routes and "sh ip cef sum"
shows 250k entries. Is this pushing it too close to hitting software
switching? I'm not seeing any complaints in the log about fib table
drops.

wrt to CEF, any recommendations on configuring it or what I should
not/never be doing?

single/dual sup's...  What has been your real-world experience with
sup2 failover using SSO/NSF? any gotchas with SSO/NSF and the
SFM+DFC's?I would rather have the chassis completely die vs
having it upset the network until its fixed.

many thanks for your time if you made it this far!

Best,

Bill