[c-nsp] RSP720/SRB in production?

[Justin Shore]

Phil,

I have some thoughts on SRB.  I apologize in advance for the length.

We have a single Sup720-3BXL in a pair 7613s and we're running 
12.2(33)SRB.  Each chassis has a FWSM, IDSM2 (all but unusable in the 
7600s, FYI), ACE, SSC-400, 6748 (with DFC) and 6724 (also with DFC).  We 
had to run SRB thanks to CALEA.  We'd have at least waited a few months 
if it wasn't for CALEA but our hand was forced.


CSCeg62206
The work-around was to disable TACACS and use RADIUS which is only a 
temporary solution for us.  The TAC engineer said the DE was going to 
have it fixed in a new SRB release.


We ran into a bug last week that required a complete reboot of the 
chassis to resolve.  FWSM fail-over died.  While troubleshooting this we 
noticed that what was the active FWSM could no longer see a couple VLANs 
that had been assigned to its firewall vlan-group.  A 'show int vlanABC' 
from the MSFC showed the SVI was up/up but the same show from the FWSM 
showed that the VLAN was down.  The VLAN also showed up in the FWSM's 
running-config and could be assigned to contexts.  However 'show vlan' 
in the FWSM would not show the VLAN.  The SVI was passing traffic within 
the VRF too.  We tried everything from removing the affected VLANs from 
firewall vlan-group and re-adding them, to removing all the affected 
VLANs from firewall vlan-group and re-adding, to rebooting the FWSM. 
Newly added SVIs to the firewall vlan-group also wouldn't show up.  We 
had to reboot the chassis to fix the problem.  TAC couldn't point us to 
a bug on this one but did recommend the reboot.  The FWSMs were running 
3.1(5).


We've run into a problem with VRFs in SRB where if we remove a VRF and 
re-add it (and all the config that's removed when you remove a VRF) the 
VRF won't work.  Both the RD and VRF name can not be used again on that 
chassis without a reboot.  We verified this prior to the reboot.  After 
the reboot the VRF RDs and VRF names that had been removed could be used 
again, once.  Removing the VRF and re-adding the identical VRF, changing 
the RD within a VRF, or removing a VRF and re-adding the VRF with a 
different name and we'd be right back where we were with a broken VRF. 
TAC couldn't point us to a bug on this one either.


We could not get a L2L IPSec tunnel to establish with a ASA from one of 
the Sup720s.  We had to get this working for our CALEA compliance.  The 
fix that TAC recommended was to move the tunnel to a 3845 while they 
worked on the problem.


Lawful Intercept packets can't be sourced from a specific interface. 
Prior to bringing the tunnel online we attempted to confirm that LI was 
working with the out TTP's MD across the 'Net.  In our case the LI 
packets were sourced from the interfaces facing the border routers which 
are privately-addressed interfaces.  A similar problem would exist if 
the MD was located onsite but not directly connected to the 7600s.  The 
source interface would change with the routes as failures happen, as we 
do maintenance, etc.  The MD filters traffic based on the IP that it 
commanded a LI-compliant device to initiate an intercept.  This isn't 
really a bug but a really annoying lack of a feature.


We had a couple Security SMEs out a few months back to help out with a 
few things.  They couldn't get IPSec HA to work properly for L2L or 
client VPN connections.  I don't know if they found a bug on this or not.

I have also run into a number of IS-IS related issues with our 
Sup720-3BXLs running SRB.  Here are the 2 worst ones.

TAC says that Integrated Multi-area IS-IS is not supported.  Some docs 
agree and some disagree.  Some TAC engineers disagree as did the SEs 
that recommended we go that route this time last year.  The "IS-IS 
Network Design Solutions" book appears to disagree as well with the 
example on page 290.  Configuring multiple IS-IS processes works but you 
can't assign more than one tagged process to an interface.  This 
prompted us to switch to a single L2 area which is probably the better 
solution in the end anyhow.  Still it was many week of aggravation and 
frustration.

At least once a week we have a router that's connected to the 7600s lose 
one or both of the 7600s' IS-IS adjacencies.  Debugging on the 7600 with 
the lost adjacency shows that it is sending the IIH's of the maximum MTU 
size whereas all the other IIH's are the usual small size (under 100 
bytes).  A reboot of the remote router fixes this.  I can't explain this 
one.


We've got full BGP tables on both 7600s.  Loading the full table from 
scratch is extremely painful, much more so than loading 3 simultaneous 
copies of the table on a 7200-NPE-G1.  Overall the load is fairly low. 
Occasionally though BGP will cause some fairly high load.  The 3BXL 
shouldn't even break a sweat with the trivial load we're throwing at it. 
  With our load we should be able to process switch everything without 
the load getting anywhere near 50%.


Now for some more general 7600 rants that are more of a product of the 
BU split.

The IDSM2 module is essentially worthless in a 7600 chassis.  The 7600 
is essentially limited to one usable monitor session; two really but one 
is burnt for the service module session.  The IDSM2 needs at least one 
session for passive mode.  Inline mode is not supported on the 7600 
platform.  In case it was missed the first time I'll repeat it again.

***IDSM2 INLINE MODE IS NOT SUPPORTED ON THE 7600 PLATFORM.***

Unfortunately this is not documented anywhere on Cisco's public site 
that we've been able to find.  The Advanced Services Security SME 
finally found a post to an internal mailing list that stated that inline 
mode wasn't supported.  This eliminates all IPS functionality.  I can't 
let the IDSM2 use my only remaining SPAN session for passive mode so 
basically it's useless to me (going cheap anyone?).  I didn't want it 
anyhow.  For this particular application standalone 4200s would have 
been the better solution.

We originally planned on buying WebVPN SMs as well.  Unfortunately these 
aren't supported in the SR code.  This module would have greatly 
simplified putting client SSL VPNs into VRFs.  The recommended 
work-around was 2 3845s to terminate the SSL VPN clients and stick them 
in the appropriate VRFs.


Overall I'm happy with SRB and the Sup720-3BXL.  I can't necessarily 
fault SRB for our specific problems.  I do wish we'd been told about the 
RSP720 before the BoM was finalized.  Delaying a few months to get the 
new Sup from the right BU would have been desirable.  I'm going to push 
our AM to take the IDSM2s back.  They are completely worthless to us and 
can not provide the advertised services.  I'd like to trade them for 2 
RSP720s if possible, minus the difference in price of course.

That's my thoughts on SRB so far.
  Justin



Phil Bedard wrote:
> 	I am curious if anyone is using the RSP720/SRB software in a  
> production network yet, or using SRB in production with a Sup720?   
> What issues or anomalies you have run into to date.
> 
> Thanks,
> 
> Phil