[c-nsp] Nexus 7k with F2e line cards and egress queuing

[James Bensley]

On Sat, 14 Dec 2019 at 16:18, Curtis Piehler <cpiehler2 at gmail.com> wrote:
> Has anyone had egress VQ Congestion issues on the Nexus 7k using F2e line
> cards causing input discards?  There has been intentional influx of traffic
> over the past few months to these units (Primarily VoIP traffic) IE: SBCs
> and such.  These SBCs are mostly 1G interfaces with a 10G uplink to the
> core router.  At some point of traffic shift the switch interfaces facing
> the SBC accrue egress VQ congestion and input discards start dropping
> packets into the switches from the core router uplinks.
>
> We have opened a Cisco TAC ticket and they go through the whole thing about
> the Nexus design and dropping packets on ingress if the destination port is
> congestion, etc... and I get all that.  They also say going from a 10G
> uplink to a 1G downlink is not appropriate however those downstream devices
> are not capable of 10G.  They amount of traffic influx isn't that much
> (your talking 20-30M max of VoIP).  We have removed Cisco FabricPath from
> all VDCs and even upgraded our code from 6.2.16 to 6.2.20a on the SUP-2E
> supervisors.
>
> I understand the N7K-F248XP-23E/25E have 288KB/Port and 256KB/SOC and I
> would think these would be more than sufficient.  I know the F3248XP-23/25
> have 295KB/Port 512KB/SOC however I can't see the need to drop 7x the
> amount for line cards that should be able to handle this traffic?

Hi Curtis,

I haven't touched Nexus 7Ks in a few years and only worked with F2e
cards, no others, so I'm no Nexus expert...

However, even at low traffic volumes it is possible to fill the egress
interface buffers and experience packet loss. Seeing as you have voice
traffic flying around does this mean you also use QoS (even if just
default settings)?

Assuming the media streams are in an LLQ coming over the 10G link into
the 7K, with the egress link as 1G with a 288KB per port as you say;
that's about 0.2ms at 10Gbps or about 2ms at 1Gbps.

If you have some signalling traffic into your core router (someone is
setting up a new call for example) and at the same time some RTP
traffic comes into the core router from somewhere else (e.g. media
from an established call) and both of these flows are destined to the
same SBC attached to the 7k; the media traffic will be dequeued over
the 10G link towards the 7K first because it's in an LLQ, then the
signalling traffic will have been buffer for one or two ms, then
you'll have a short burst of signalling traffic be dequeued over the
10G link to the 7K at a rate of 10Gbps, the 7K won't be able to
dequeue this traffic out of the 1G link towards the SBC as fast as it
comes in from the core router.

The volume of traffic doesn't need to be anywhere near 1Gbps, as long
as it's greater than 288KBs of traffic it will be coming into the 7K
quicker than it can dequeue it towards the SBC (so, if the burst from
core router to 7K last for longer than 0.2ms at 10Gbps it will exceed
the 2ms buffer on your 1Gbps port). If possible, run a packet capture
on the 10G link and look at the traffic coming into the 7K link when
you have 1G egress congestion / 10G input discards, even if you're
seeing an average out speed of 30Mbps in a single 1 second sampling
period meaning the 10G link will be idle for 99.7% of the 1 second
sampling period, if the 10G link runs at full speed (meaning it never
idles) for 0.2ms consistently you'll have packet loss.

If you don't explicitly have QoS configured, check what your default
settings are implementing and if you can increase the per-port
buffers. It's been a while since I touched a 7K but I do recall that
the default settings were not great.

Cheers,
James.