[c-nsp] 6500 SUP720 High Latency and Jitter issues

[Dan Benson]

At this point here is what I have found:

1: I have no physical errors on any of my ports that are ingress or 
egress to the router.

2: I removed NAT from one of the ports that my upstream ISPs connects to 
and it immediately stopped the traffic Jitter and Latency.  Seems the 
router only liked one outside nat interface. 

Here is where I currently stand:

1:  I have deleted and rebuilt my tunnels, in the beginning I was 
running the tunnel source as the Loopback of my local machine and my 
destination as the Loopback of the router at the far end.  Now I am 
homing the tunnels to my public vlan ip addresses.  This seems to have 
dropped the CPU load on the router by 30%.   When I went to do this same 
change to another tunnel, I spiked the CPU right back up 30%.  I have no 
clue why on earth I could help the CPU with changing the source and 
destinations of one tunnel, and then doing the same exact thing to 
another tunnels kills me again.  Any ideas?  It seems half the time I 
build a tunnel in these machines, they say the packets will be software 
switch, and the other half of the time, they are hardware.  There seems 
to be no science behind this.  Could it be that I have too many 
Tunnels?  Is there a limit as to how many tunnels the SUP can handle in 
hardware? 

2:  I have rebuilt the NAT config to use a pool for overload instead of 
an interface.  I would have liked to have used my public Vlan interface 
as the outside and my private Vlan interface as the inside, but when I 
do I have no luck with the translations.  After rebuilding the NAT 
config to you use the pool other then an interface,  I could readd the 
ip nat outside to my upstream interfaces without affecting my traffic 
traversing the router.

So my unanswered questions you'll might be so kind to help me with:

1:  From the sounds of it, the 3BXL is able to hardware switch tunnels, 
can the normal SUP720 not?  If so, is there a limit as to how many 
tunnels will be hardware/software switched?  From my testing, there 
seems to be no method for this, just dumb luck.  I honestly fell as 
thought I have hit the Max Hardware switchable tunnels I can.  I 
currently have 11 tunnels on this router.

2:  Can I NAT from a Vlan interface that is Public to a Vlan interface 
that is private?  Can I overload to the public vlan interface?  If I 
should be able to, I cannot in the code version I am running.  It seems 
strange that my latency and jitter disappeared the second I remove the 
IP nat outside statement on my ISP's interface, and that I was able to 
keep the traffic stable by using a pool for overload.

At this point, the router is running well below 30% CPU at a peak, and 
traffic is fine and dandy.  I just hope I can figure out what I will do 
when I have to add more tunnels.  FYI, these tunnels are low throughput, 
High packets per second (SIP VOIP), so they are using a lot more of the 
CPU then normal ISP internet traffic would.  Thank for your help in 
advance.   A paste from  my current tunnel is below..  //db


Tunnel1 is up, line protocol is up
  Hardware is Tunnel
  Description:
  Internet address is 192.168.253.58/30
  MTU 1514 bytes, BW 9 Kbit, DLY 500000 usec,
     reliability 255/255, txload 248/255, rxload 248/255
  Encapsulation TUNNEL, loopback not set
  Keepalive not set
  Tunnel source XXX.XXX.XXX.XXX (Vlan800), destination XXX.XXX.XXX.XXX, 
fastswitch TTL 255
  Tunnel protocol/transport GRE/IP, key disabled, sequencing disabled
  Tunnel TTL 255
  Checksumming of packets disabled, fast tunneling enabled
  Last input 00:00:01, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/0 (size/max)
  5 minute input rate 22458000 bits/sec, 11624 packets/sec
  5 minute output rate 21102000 bits/sec, 12076 packets/sec
  L2 Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
  L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes mcast
  L3 out Switched: ucast: 0 pkt, 0 bytes mcast: 8970 pkt, 924140 bytes
     717866670 packets input, 172903508876 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicast)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     740531416 packets output, 161555984163 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
NYC-BV-RTR#