[c-nsp] Weird throughput issue

[Brault, Ryan]

Check your policy on the ASR9k.  We had this exact same issue with TCP on ASR9ks and specific transport providers.  In short, the problem was that the provider had a strict policer (for bursting), while we were shaping a sub-if on the ASR9k.  The ASR9k was bursting beyond the strict carrier policer, so the carrier was dropping.  Thus, TCP windowing came into play.  The fix was to enable low-burst mode on the ASR9k and create a policy like this:

Policy-map Default-child
  Class class-default
    Bandwidth percent 100
!
Policy-map Shape-100Mb
   Class class-default
     Service-policy Default-child
     Shape average 100 mbps 1 us


This fixed all our issues.

--
Ryan Brault
Illinois Department of Innovation & Technology
815-936-4647


-----Original Message-----
From: cisco-nsp [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Curtis Piehler
Sent: Sunday, July 24, 2016 9:41 AM
To: Cisco Network Service Providers <cisco-nsp at puck.nether.net>
Subject: [c-nsp] Weird throughput issue

I've been scratching my head over the past week on this issue as it doesn't
make any sense but here are the details.

I have a 100M circuit that is sold through a transport provider to an ILEC
(Type 2).  The circuit comes into one of our facilities with the following
equipment:

Customer Site (2911 Router) -> ILEC -> Transport Provider -> Carrier NNI
(1G Copper) -> Cat4900 -> Cat6504 -> ASR9k

The ASR9k terminates the IP for the customer so the in between devices are
Layer 2 trunking all the way through.

Latency from the customer site to the Cat infrastructure is only 3-4ms.
Once it trunks over to the ASR9k for IP termination it increases to 17ms (A
different facility).  We have several IPERF servers to test from however
the latency never exceeds 40ms to any given server.

When performing IPERF tests we can not pass anything more than 10M to the
site (down stream).  Upload from the site maxes out around 50-60M when we
test to anything behind the ASR9k IP network.  If we test to an IPERF
server off of the Catalyst infrastructure where the NNI comes into we can
pass full throughput.  We have tried all different size windows on IEPRF
which do not yield any improved results.  The further away we get from the
site (again not exceeding 40ms) the less throughput is seen.  UDP tests to
the site are ok as we can force 85-90M and the site receives it with very
little packet loss.  The issue is just TCP.

We have a had a facility tech plug a laptop directly into the Cat 4900 on a
separate access port and test back to the same IP termination the customer
lands in our network and there are no issues.

Now here is the interesting thing:
The customer has another 100M site from us (a different carrier) that comes
into the Cat6504 in the above path.  It trunks up to the same ASR9k that
this site with the issues does.  No throughput issues are seen from this
site at all.

The ILEC and Transport provider have dispatched out multiple times and
RFC2544 test end to end with no issues.  The transport provider also peeled
the circuit off our NNI and configure it as Layer 3 back to their network
and did RFC6544 testing with no issues.

We've looked at the Cat4900 and Cat6504 configs and can find nothing that
would inhibit this throughput.

Most modern day computer OS's do dynamic TCP scaling so the bandwidth of
distance theory shouldn't really apply here (also considering the distance
isn't that significant).  I've tried decreasing the TCP-MSS on both WAN
interfaces (Core and customer router) to as low as 1000 with no impact at
all.

For technical sanity sake all RJ-45 Ports on site are locked at 100M/Full
Duplex.

Does anyone have any experience with this type of scenario?

Curtis
_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


CONFIDENTIALITY NOTICE: The information contained in this communication is confidential, may be attorney-client privileged or attorney work product, may constitute inside information or internal deliberative staff communication, and is intended only for the use of the addressee. Unauthorized use, disclosure or copying of this communication or any part thereof is strictly prohibited and may be unlawful. If you have received this communication in error, please notify the sender immediately by return e-mail and destroy this communication and all copies thereof, including all attachments. Receipt by an unintended recipient does not waive attorney-client privilege, attorney work product privilege, or any other exemption from disclosure.