[nsp] Cisco 6400 NRP-1 Performance

[Clayton Zekelman]

OK, after what I THOUGH was an issue last time around with OSPF and the
6400's NRP-1 card, I'm not going to be so quick to blame the architecture
this time, but its now 8:00am and I've been up on a failed cutover since
1:00am trying to work this issue through, and any light anyone could shed
on this would be appreciated.

We have around 750 L2TP sessions coming in over a DS3 ATM interface into a
PA-A3-T3 card on a 7200/VXR with an NPE-300.  The router is pulling double
duty, and I wanted to migrate the L2TP sessions off it.  During peak
periods, it can hit as much as 75% CPU utilization.

I decided to migrate the sessions to an NRP-1 card.  Its supposed to be the
processing equivelent of an NPE-200 with a PA-A1-OC3 interface and an FE
port I beleive.

We use several NRP-1's in a 6400 chassis - a couple of them terminating
between 700 and 900 L2TP sessions strictly through the FE ports on VLANs
with the IP transit coming in through the same interface.  We run roughly
45% CPU  on 'em.  Pretty decent.  We also have an NRP-1 terminating 400 or
so PPPoEoATM sessions through the ATM backplane of the 6400 with the IP
transit coming in the FE.  It runs 25% or so CPU.  So far so good.

All the NRP-1's are equipped with 128M RAM, 16M Flash.

Early this morning, I cut over the T3 with the 750 L2TP sessions.  At that
time of day, only a couple of hundred of them will actually restore, so the
load shouldn't be too bad.  We're using subinterfaces, so there's no need
to preclone Vi templates.  Works fine on the other routers that way.

As soon as I cut the T3, the CPU goes to 100% and stays pegged for a while.
 It floats down to 80%, and hovers there, spiking to 100% from time to
time.  All this with 110 user sessions.  Continues like this for quite some
time.  I'm barely able to issue commands to the router.  I go over my
configs, looking for something - anything - and find nothing.  I strip down
to bare bones.  Zip. No change. I should probably point out that there are
60 active tunnels and around 175 PVCs defined (lots of LAC's).

I even swap NRP-1 cards "just incase".  No help.

I've now fallen back to the "overloaded" NPE-300 in the 7200, and I'm
running with 241 sessions (early Sunday morning) and 20% CPU utilization.

I'm running 12.2(15)T5.  Anyone have any thoughts on this one?

I went over all the L2TP scaleability documentation I could find, and tried
the tricks in there - it was a long night.

What gets me is that the NRP-1's perform fine in the other applications.  



---
Clayton Zekelman
Managed Network Systems Inc. (MNSi)
344-300 Tecumseh Rd. E.
Windsor, Ontario
N8X 5E8

tel. 519-985-8410
fax. 519-258-3009