[c-nsp] SXI4a (Was: 7606 config issue !!!)

[Steve Colam]

On Fri, 8 Oct 2010, Dan Holme wrote:

> Some time has passed; can anybody elaborate on their experiences with
> SXI4/SXI4a?

Yes, we've been running SXI4a advanced ip services on Sup720 3Bs, 3BXLs, and
VSS-10G on 6509-E & 6513 chassis, some chassis single sup, others dual sup
with SSO.

We run TE with FRR, with ISIS, and bgp with v4/vpnv4/v6 addr families, otherwise
std stuff ie ingress/egress policers, some ingress netflow, urpf, COPP, etc.

SXI3 & SXI4a have the following three serious issues:

* SXI no SVI ingress counters - SR 614334473

Ingress SVI counters are in-accurate, only packets hitting the
control plane are counted when MPLS FRR is implemented.

Cisco say:
The problem is caused by the way we handle the h/w logic of recirculated
MPLS packets.

In SXH/SXI, along with CEF/MFI code rewrite in ip/mpls forwarding, we
added the feature to support prefix independent FRR which allows fast
convergence on FRR cutover regardless of the number of prefixes to go
over the FRR tunnel. Recirculation on mpls packets is needed to achieve
this. We just realized that as a result vlan stats are disabled on
packets that need to be recirculated and have MPLS format.  This is the
reason that you did not hit the problem on SXF, but on SXI image.

Unfortunately, the code cannot be reverted back to the logic used in SXF
and changing it would require an entire rewrite of the FRR code.
Basically, if you have FRR, you lost the ingress interface stats.
/Cisco say

The limitation is an EARL7 on 6k & 7k, which means Nexus or another
vendor to fix.

We have also seen this on .1q ES20 interfaces on SRD4 with 3BXLs.

* SXI crash CPU_MONITOR-6-NOT_HEARD - bug CSCtj11500 / SR 615445833

Initially seen on SXI4a, later on SXI3, on VSS-10G and 3BXL 50/50
re-producable, by shutting an MPLS tunnel, defaulting it and removing it,
120 seconds later crash and burn. Work around so far is to shut the tunnel,
wait a bit, default the tunnel, wait a bit more, then remove it. The
MSFC and Sup crashinfo are next to useless, we've got an ftp server
directly connected to relevant nodes now with exception dumping enabled
to capture a full diags image for TAC to investigate further.

* SXI STP issues

We have found a new issue we are currently investigating (only just
opened with TAC) regard STP BPDUs not being sent/received, and
multicast traffic not working.

In our setup, we have 6748s in 6513s that do not seem to work,
but 6724s in the same 6513s that do.


But good news:

* IDBs

IDBs quantity is now significantly improved (in volume) over SXF. This
may not initially seem useful, but it is if you're using mpls auto-tunnels
as the IDBs become consumed in about 6 months on SXF. There are no official
docs from Cisco to detail the IDB change, but we got this from TAC:
12.2(33)SXI 12k IDB, 12.2(33)SXH 12k IDB, Earlier IOS 5k IDB. SXJ will
support 12k IDB.

* MUX-UNI works very nicely.

* tcp window size is increased to 1048560, only 65535 in SXF - so good times
for your bgp.






Steve Colam
Head of Network Operations
Daisy Communications/Vialtus Solutions
steve.colam at vialtus.com
PGP Key ID: 0x1C19D542