[c-nsp] ATM packet loss

[James Berwick]

Mikael Abrahamsson wrote:
> On Wed, 2 Sep 2009, James Berwick wrote:
>
>> works fine.  Pinging a customer on our OC3 with any packet larger 
>> than 576 bytes intermittently fails.
>>
>> This is the ATM OC3 config:
>> interface ATM1/1/0
>> mac-address 0000.0c71.1148
>> bandwidth 155000
>> no ip address
>> load-interval 30
>> atm ilmi-keepalive
>> end
>
> First thing I'd try here would be to configure some ubr value on the 
> OC3, set it to 120 megabit/s or something like that. If that helps, 
> increase until you get close to wirespeed and see if the problem 
> creeps up the higher the UBR. The behaviour you're seeing is alike to 
> what I've seen when there is a cell policer dropping cells somewhere.
>
I rebuilt a PVC on the circuit to ubr 149760 (full linespeed of our OC3) 
and observed the same results.  Continuous pings of 576 bytes work 
flawlessly.  577 byte pings show between 1% and 2% packet loss.  We 
don't maintain the ATM network between us and our customers.  The LEC 
that does has gone to two of our customer's facilities and put an ATM 
test kit on the DS3 and tested sending 45mbit/sec of cells across that 
particular customer's PVC.  We've then looped the OC3 in the router with 
loopback line on the ATM1/1/0 interface and they saw 45mbit/sec worth of 
cells returned.

I'm not sure where to look for a cell policer.  This OC3 comes from the 
LEC into our 7500 series and we don't have any QoS or rate limiting on 
the ATM interface itself or any of the PVCs.

I'm not sure if this is helpful information or not but if we use 
something like speedtest.net from a customer's network we end up with 
results like 5mbit down, 30mbit up.  It seems like the throttling is 
taking place in only one direction, but we aren't (intentionally) doing 
it on our router and the LEC who handles the ATM cloud swears up and 
down they aren't doing anything that would cap throughput and none of 
the switches between us and the customer are oversubscribed.

I just tested now by kicking off enough ping -f x.x.x.x -s 548's (Ping 
flood with 576 byte packets) to push 20mbit/sec worth of ICMP traffic 
across a PVC with 0% packet loss, while the same test with a 1000 byte 
ping fails miserably.  I'm at a loss as to why either our router, our 
LEC's ATM network (which according to them is doing no policing at all 
of our UBR traffic), or a dozen of our customer's Cisco routers have all 
suddenly decided that packets over 576 bytes need to get dropped on the 
floor randomly.

This is what our interface looks like:
ATM1/1/0 is up, line protocol is up
  Hardware is cyBus ENHANCED ATM PA
  MTU 4470 bytes, sub MTU 4470, BW 155000 Kbit, DLY 80 usec,
     reliability 255/255, txload 48/255, rxload 13/255
  Encapsulation ATM, loopback not set
  Encapsulation(s): AAL5
  4095 maximum active VCs, 32 current VCCs
  VC Auto Creation Disabled.
  VC idle disconnect time: 300 seconds
  0 carrier transitions
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 1d05h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 60
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 8191000 bits/sec, 3544 packets/sec
  30 second output rate 29327000 bits/sec, 4090 packets/sec
     289263860 packets input, 803308614 bytes, 0 no buffer
     Received 254286 broadcasts, 0 runts, 6 giants, 0 throttles
     7 input errors, 1 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     326860666 packets output, 1743804746 bytes, 0 underruns
     60 packets late drop, 86107 bytes late drop
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out

Interface ATM1/1/0:
AAL enabled:  AAL5  , Maximum VCs: 4095, Current VCCs: 32

Maximum Transmit Channels: 0
Max. Datagram Size: 4528
PLIM Type: SONET - 155000Kbps, TX clocking: LINE
Cell-payload scrambling: ON
sts-stream scrambling: ON
178409785 input, 327045383 output, 29153365 IN fast, 109018418 OUT fast, 
0 out dropVBR-NRT : 672
 Avail bw = 149088
Config. is ACTIVE