[nsp] IPSEC throughput impact?

[Streiner, Justin]

I'm diagnosing a case where a customer who has a site-to-site VPN tunnel
with us is complaining that their throughput degrades significantly when
the site-to-site traffic is routed across 3 private T1s between them and
us.  The 3 T1s are running CEF per-packet load-sharing on both sides and
are short-haul only, so I feel pretty confident in ruling out RTT
variance across the 3 circuits interfering with the load-sharing and
eventual packet reassembly/decryption in this case.

The customer sees throughput of about 2.1 Mb/s across the T1s with the VPN
traffic routed across them.

When we let the site-to-site traffic flow across the Internet, their
throughput across the VPN doubles to about 4 Mb/s.  Oddly enough, they
don't appear to come close to maxing out the T1s during our testing -
maybe leaking out at 50-60% usage.  I can access the test server on their
network with or without the VPN, and when I access it without the VPN, the
throughput is about 3x higher than when I connect to their VPN.  This is
with the site-to-site traffic going across the Internet in both
directions.

Beyond the overhead of the encryption itself, and perhaps some choke point
on the path of the T1s, I'm at a loss to explain the large loss of
throughput.

I understand the size of the original unencrypted packet will factor
greatly into the throughput measurements since a large original packet may
need to be broken up into two IPSEC packets upon encryption, and
reassembled upon decryption.

The VPN itself is IPSEC-3DES, using the Cisco Unity VPN client.

My question is:
Has anyone had personal experience with a similar case as far what kind of
throughput a customer should be able to realistically expect over their
VPN?

jms