[c-nsp] Weird throughput issue

[Andrew Miehs]

On Sun, Jul 24, 2016 at 7:11 PM, Curtis Piehler <cpiehler2 at gmail.com> wrote:
> Hi Andrew,
>
> Thank you for the reply.  The cat4900 and 6504 are acting as a pure Layer 2
> trunking device.   The carrier NNI comes into the Cat4900 and the customer

Understood, which, but you mentioned 3-4ms between 2911 and c4900.
How did you measure this time if the c4900 is trunking?

...
> out.  Based on this I know I should at least see 20-25M to this site from my
> IPERF servers.  Latency to one of my IPERF servers is 17ms and the other is
> 35ms.  The IPERF server that sees 17ms can not push anymore than 5-6M
> through the router.  The IPERF server that sees 35ms can not push anything
> more than 2M to the router.
....
> to TTCP test to this site I can push nearly 20-25M based on the calculation
> above with no issues (The processor maxes out near 25M).  When I repeat the
> same test to the site with the problem (using the same Transmit/Receive
> Window) the results are consistent with the paragraph above,.

What are your receive window sizes? (And I don't mean MSS or MTU)

The descriptions above are a little unclear, and it may help you to
create some form of table to see what happens with which combinations.

> The only difference in the setup between both sites is 1) the carrier(s)
> that is used and 2) the site with the problem trunks through a Cat4948
> initially before hitting the 6504.  We have other customers off of the
> Cat4948 who report no issues.  I have looked over the Cat4948 config and
> there is nothing apparent that would hinder TCP taffic.

So based on this description, you believe you only have one
difference, which is the link from the carrier itself.
I would recommend confirming with them whether they have incorrectly
provisioned your link, and set policers/ qos settings incorrectly. I
would also confirm with them that they haven't oversubscribed part of
their backbone.

Have you run tcpdump on your iperf instance to see what is happening?
This will show you pretty quickly whether you are experiencing
retransmits, or if one side has not enabled window scaling.


> This whole problem started when the customer went to do a speed test from
> the problem site and couldn't even break 5M on the download and 50-60M
> upload.  When the customer speed tests from the other site that comes into
> the Cat6504 that trunks up to the same ASR9K they can obtain 90/90.

And I assume you have checked the interfaces and CPU on the c2911? No
drops, pause frames?
Once again, ask the carrier whether they have any issues on their link.

Should you have no luck with the tcpdumps, it may be quickest to
connect PCs directly to either end of this link (it is copper?) and
then test point to point to take all other hardware out of the
equation.

>
> I've also applied 90M shapers on both ends of the WAN to avoid hitting the
> carrier policers in the middle and it had no effect.

This is a good idea, but just be aware that this may increase jitter
which may be an issue if the link is being used for voip.
Is it correct to assume that you configured the shapers on the asr9k
and the c2911?