[nsp] OAM vs ping RTT

[james]

I am troubleshooting an issue with DSL customers in a specific city
reporting stalled web page loading and dropped or very slow connections.
Ping RTT is normal, ~60ms, to the CPE. Using ATM ping to the
end-loopback yields 400-500 ms. However some customers in this city have
normal ping and OAM RTT, close to the same RTT for either method, and
are not reporting problems. OAM to the seg-loopback yields short, single
digit RTT, I assume this is the next ATM switch from my side of the PVC.
I am seeing OAM CRC's. 2% loss of OAM.

Hardware is 7206 NPE-400, PA-T3-A3,
c7200-js-mz.122-15.T7.bin, 512mb mem & the router 
is showing no signs of distress or excessive droppage. Problem customers
are either PPPoA or atm-routed bridge ip. Interface is a DS3, we are
at ~50% utilization. 

I am confused as to how OAM is slow, yet L3 seems normal.
Any comments on any of this would be a great help.

The ILEC has ruled out the customers local DSLAM, the ATM switches
in that city and the next cities ATM switch. RTT for OAM is normal
to this point and the ILEC is relocating to my side of the circuit
(another city) so they can look toward me and the customer.

This is one of the customers who is having trouble:

alb-router5#sho atm pvc 1/522
ATM2/0.522: VCD: 495, VPI: 1, VCI: 522
UBR, PeakRate: 40704
AAL5-LLC/SNAP, etype:0x0, Flags: 0xC20, VCmode: 0x0
OAM frequency: 10 second(s), OAM retry frequency: 1 second(s)
OAM up retry count: 3, OAM down retry count: 5
OAM Loopback status: OAM Received
OAM Managed VC Status: Verified
ILMI Managed VC status: Not Managed
VC is managed by OAM.
VC TxRingLimit: 40 particles
VC Rx Limit: 480 particles
InARP DISABLED
Transmit priority 6
InPkts: 16384, OutPkts: 37587, InBytes: 4375983, OutBytes: 6311480
InCells: 98221, OutCells: 143631
InPRoc: 660, OutPRoc: 23108
InFast: 15723, OutFast: 14479, InAS: 0, OutAS: 0
InPktDrops: 0, OutPktDrops: 0/0/0 (holdq/outputq/total)
InCellDrops: 0, OutCellDrops: 0
InByteDrops: 0, OutByteDrops: 0
CrcErrors: 328, SarTimeOuts: 0, OverSizedSDUs: 0, LengthViolation: 0,
CPIErrors: 0
Out CLP=1 Pkts: 0, Cells: 0
OAM cells received: 611
F5 InEndloop: 606, F5 InSegloop: 5, F5 InAIS: 0, F5 InRDI: 0
F4 InEndloop: 0, F4 InSegloop: 0, F4 InAIS: 0, F4 InRDI: 0
OAM cells sent: 617
F5 OutEndloop: 612, F5 OutSegloop: 5, F5 OutRDI: 0
F4 OutEndloop: 0, F4 OutSegloop: 0, F4 OutRDI: 0
OAM cell drops: 0
Status: UP

ATM2/0.522 is up, line protocol is up
  Hardware is ENHANCED ATM PA
  Description: BRIDGED for xxxx
  Internet address is xx.xx.xx.xx/29
  MTU 4470 bytes, BW 40704 Kbit, DLY 190 usec,
     reliability 255/255, txload 82/255, rxload 38/255
  Encapsulation ATM
  16387 packets input, 98233 cells, 4376439 bytes
  37612 packets output, 143681 cells, 6313580 bytes
  623 OAM cells input, 629 OAM cells output
  AAL5 CRC errors : 328
  AAL5 SAR Timeouts : 0
  AAL5 Oversized SDUs : 0
  ARP type: ARPA, ARP Timeout 00:20:00
  Last clearing of "show interface" counters never

Network is less loaded now, looks a bit better:

alb-router5#ping atm interface ATM2/0.522 1 522 end-loopback 5
Sending 5, 53-byte end-to-end OAM echoes, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 360/361/368
ms
alb-router5#ping <customer IP>.
Sending 5, 100-byte ICMP Echos to 65.19.31.194, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 36/39/52 ms