[c-nsp] 65K: 10G SPAN destination interface outputs is significantly less traffic than sum of all source interfaces -- (not oversubscribed)...

[Jeremy Reid]

Hi,

I'm wondering if anyone else on the list here has seen this issue we've been struggling to pin down:

We are using interface SPAN (both rx tx) on the 65k platform (S720/3BXL, currently running SXH3a) to aggregate data from (3) different 10G interfaces into a 10G output port for use with a BGP route control product. The three input interfaces have a *combined* peak traffic rate of around 8Gbps. The SPAN destination interface, however, is only indicating that we are sending around 5Gbps at peak. This does not appear to be a counters problem, as we can confirm from the destination device on the other end of the SPAN port that it is indeed only seeing 5Gbps worth of traffic.

Doing a little 'deconstructuve' unit testing -- we have tried eliminating the 'aggregation angle' and picked a single source 10G interface that only had about 1Gbps worth of traffic to span. Looking at the destination interface, it was consistantly only reporting about 600mbps. We have tried various such tests and we always seem to get simillar results in that the destination interface traffic is always significantly (between 20 and 40%) LESS than the whatever the source interface is actually carrying -- at least on the egress side of things (our ingress traffic is not sizable enough to gauge accurately).

There are no physical errors/malformed frames/drops (including queue drops) being reported on either the SPAN source interface(s) or the destination. Jumbos aren't allowed on either interface, so its not related to that either. The only plus to this (from a troubleshooting perspective, anyway) is that it is consistantly 'broke' -- which should make finding the solution easier, but so far, it has proved rather ellusive.

We have replicated this scenario on both our current code (SXH3a) as well as SXF14 (previous code until very recently). Further, we can replicate it on multiple independant 65k platforms (all equipped simmillarly). We have also verified there is no bus/proc oversubscription or anything of the sort going on -- but even went to the extent of moving two test interfaces containing both the SPAN source and destination to the same physical linecard (6704-10GE) and even popped in a DFC3BXL on this linecard for good measure (even though we saw no reason to do so from a numbers point of view). No change in the behavior with the DFC.

Anyone seen anything along these lines? Couldn't find anything publically on the bug toolkit that seemed relevant... (big surprise). Just thought I'd try the list here before getting on the TAC merry-go-round. 

Thoughts?

-Jeremy

Jeremy Reid
Network Engineer
Mojohost