[c-nsp] ATM packet loss

[James Berwick]

Hello!

We have a 7500 series router, RSP16 with IOS Version 12.3(21) 
terminating a bunch of ATM circuits.

We have 3 ATM DS3s terminating on PA-A3-T3 cards, and an ATM OC3 
terminating on a PA-A3-OC3-SMI.  All line cards are VIP-480s.

The problem we are seeing is customers with PVCs on our ATM OC3 have 
begun getting extremely poor throughput.  30mbit PVCs that can pull at 
most 3-5mbit/sec.  Overall utilization of the ATM OC3 dropped from 
around 90mbit/sec last year to around 25mbit/sec this year.  We've had 
the LEC test the OC3 and the customer side (DS3s to multiple customers) 
at the ATM PVC level and show us results that the OC3 is able to push 
155mbit/sec worth of cells and the DS3s are able to push 45mbit/sec 
worth of cells.  There are no errors on the interfaces and no AAL5 
errors on the PVCs.  We're completely at a loss.  One commonality we've 
found is that when we ping the opposite side of a customer on one of the 
ATM DS3s in our chassis with large packets it works fine.  Pinging a 
customer on our OC3 with any packet larger than 576 bytes intermittently 
fails.

ATM DS3:
er02.penn-nj#ping x.x.x.x size 577 repeat 1000

Type escape sequence to abort.
Sending 1000, 577-byte ICMP Echos to x.x.x.x, timeout is 2 seconds:
...
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/4/40 ms


ATM OC3:
er02.penn-nj#ping x.x.x.x size 577 repeat 1000

Type escape sequence to abort.
Sending 1000, 577-byte ICMP Echos to x.x.x.x, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!
!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (996/1000), round-trip min/avg/max = 1/2/8 ms

Same destination as above with 576 byte pings:
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/2/8 ms

This is the ATM OC3 config:
interface ATM1/1/0
 mac-address 0000.0c71.1148
 bandwidth 155000
 no ip address
 load-interval 30
 atm ilmi-keepalive
end

This is a PVC config (all are very similar and all show identical 
performance):
interface ATM1/1/0.36 point-to-point
 bandwidth 45000
 ip address x.x.x.x 255.255.255.252
 pvc 1/141
  ubr 44209
  oam-pvc manage
  oam retry 3 5 1
  encapsulation aal5snap
 !
end


Thanks anyone in advance for any suggestions!