[c-nsp] Weird throughput issue

[Curtis Piehler]

I've been scratching my head over the past week on this issue as it doesn't
make any sense but here are the details.

I have a 100M circuit that is sold through a transport provider to an ILEC
(Type 2).  The circuit comes into one of our facilities with the following
equipment:

Customer Site (2911 Router) -> ILEC -> Transport Provider -> Carrier NNI
(1G Copper) -> Cat4900 -> Cat6504 -> ASR9k

The ASR9k terminates the IP for the customer so the in between devices are
Layer 2 trunking all the way through.

Latency from the customer site to the Cat infrastructure is only 3-4ms.
Once it trunks over to the ASR9k for IP termination it increases to 17ms (A
different facility).  We have several IPERF servers to test from however
the latency never exceeds 40ms to any given server.

When performing IPERF tests we can not pass anything more than 10M to the
site (down stream).  Upload from the site maxes out around 50-60M when we
test to anything behind the ASR9k IP network.  If we test to an IPERF
server off of the Catalyst infrastructure where the NNI comes into we can
pass full throughput.  We have tried all different size windows on IEPRF
which do not yield any improved results.  The further away we get from the
site (again not exceeding 40ms) the less throughput is seen.  UDP tests to
the site are ok as we can force 85-90M and the site receives it with very
little packet loss.  The issue is just TCP.

We have a had a facility tech plug a laptop directly into the Cat 4900 on a
separate access port and test back to the same IP termination the customer
lands in our network and there are no issues.

Now here is the interesting thing:
The customer has another 100M site from us (a different carrier) that comes
into the Cat6504 in the above path.  It trunks up to the same ASR9k that
this site with the issues does.  No throughput issues are seen from this
site at all.

The ILEC and Transport provider have dispatched out multiple times and
RFC2544 test end to end with no issues.  The transport provider also peeled
the circuit off our NNI and configure it as Layer 3 back to their network
and did RFC6544 testing with no issues.

We've looked at the Cat4900 and Cat6504 configs and can find nothing that
would inhibit this throughput.

Most modern day computer OS's do dynamic TCP scaling so the bandwidth of
distance theory shouldn't really apply here (also considering the distance
isn't that significant).  I've tried decreasing the TCP-MSS on both WAN
interfaces (Core and customer router) to as low as 1000 with no impact at
all.

For technical sanity sake all RJ-45 Ports on site are locked at 100M/Full
Duplex.

Does anyone have any experience with this type of scenario?

Curtis