[c-nsp] How to find the root cause of packet loss

[Anton Turygin]

Hello,

Getting output drops and packet loss on Catalyst WS-C2960G-48TC-L.

The traffic is relatevely small but output drops are growing hundreds per 
second.

GigabitEthernet0/45 is up, line protocol is up (connected)
   Hardware is Gigabit Ethernet, address is 2893.fe8e.3b2d (bia 2893.fe8e.3b2d)
   MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
      reliability 255/255, txload 122/255, rxload 14/255
   Encapsulation ARPA, loopback not set
   Keepalive not set
   Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000BaseTX
   input flow-control is off, output flow-control is unsupported
   ARP type: ARPA, ARP Timeout 04:00:00
   Last input 00:00:00, output 00:00:20, output hang never
   Last clearing of "show interface" counters never
   Input queue: 0/4096/0/0 (size/max/drops/flushes); Total output drops: 32013814
   Queueing strategy: fifo
   Output queue: 0/4096 (size/max)
   5 minute input rate 57250000 bits/sec, 33933 packets/sec
   5 minute output rate 480872000 bits/sec, 47800 packets/sec
      2298025991 packets input, 456494816566 bytes, 0 no buffer
      Received 294650 broadcasts (245203 multicasts)
      0 runts, 0 giants, 0 throttles
      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
      0 watchdog, 245203 multicast, 0 pause input
      0 input packets with dribble condition detected
      3306257265 packets output, 4196833602160 bytes, 0 underruns
      0 output errors, 0 collisions, 1 interface resets
      0 babbles, 0 late collision, 0 deferred
      0 lost carrier, 0 no carrier, 0 PAUSE output
      0 output buffer failures, 0 output buffers swapped out



Buffers statistics:

Buffer elements:
      1505 in free list (500 max allowed)
      1652435 hits, 0 misses, 1024 created

Public buffer pools:
Small buffers, 104 bytes (total 34, permanent 25, peak 184 @ 18:14:40):
      33 in free list (20 min, 60 max allowed)
      253962 hits, 72 misses, 207 trims, 216 created
      0 failures (0 no memory)
Middle buffers, 600 bytes (total 21, permanent 15, peak 81 @ 17:22:12):
      19 in free list (10 min, 30 max allowed)
      6054 hits, 75 misses, 219 trims, 225 created
      0 failures (0 no memory)
Big buffers, 1536 bytes (total 53, permanent 50, peak 92 @ 17:22:12):
      29 in free list (5 min, 150 max allowed)
      315152 hits, 20 misses, 57 trims, 60 created
      0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 11, permanent 10, peak 13 @ 09:49:47):
      1 in free list (0 min, 100 max allowed)
      37614 hits, 2 misses, 3 trims, 4 created
      0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
      0 in free list (0 min, 5 max allowed)
      0 hits, 0 misses, 0 trims, 0 created
      0 failures (0 no memory)
Huge buffers, 18024 bytes (total 1, permanent 0, peak 2 @ 18:14:30):
      1 in free list (0 min, 2 max allowed)
      1097 hits, 1 misses, 1 trims, 2 created
      0 failures (0 no memory)

Interface buffer pools:
RxQFB buffers, 2040 bytes (total 150, permanent 150):
      141 in free list (0 min, 150 max allowed)
      13720 hits, 0 misses
RxQ1 buffers, 2040 bytes (total 128, permanent 128):
      7 in free list (0 min, 128 max allowed)
      231357 hits, 10964 fallbacks
RxQ3 buffers, 2040 bytes (total 16, permanent 16):
      1 in free list (0 min, 16 max allowed)
      42632 hits, 2705 fallbacks
RxQ4 buffers, 2040 bytes (total 64, permanent 64):
      1 in free list (0 min, 64 max allowed)
      3301 hits, 51 fallbacks
RxQ6 buffers, 2040 bytes (total 64, permanent 64):
      0 in free list (0 min, 64 max allowed)
      155 hits, 91 misses
RxQ7 buffers, 2040 bytes (total 96, permanent 96):
      30 in free list (0 min, 96 max allowed)
      581219 hits, 490 misses
RxQ8 buffers, 2040 bytes (total 32, permanent 32):
      0 in free list (0 min, 32 max allowed)
      203101 hits, 202315 misses
RxQ10 buffers, 2040 bytes (total 16, permanent 16):
      0 in free list (0 min, 16 max allowed)
      16 hits, 0 fallbacks
RxQ12 buffers, 2040 bytes (total 16, permanent 16):
      0 in free list (0 min, 16 max allowed)
      16 hits, 0 misses
RxQ15 buffers, 2040 bytes (total 4, permanent 4):
      0 in free list (0 min, 4 max allowed)
      1303978 hits, 1303974 misses
RxQ11 buffers, 2040 bytes (total 4, permanent 4):
      0 in free list (0 min, 4 max allowed)
      4 hits, 0 misses

Header pools:




Output interpreter says:
ERROR: Since it's last reload, this router has created or maintained a 
relatively
large number of 'Big buffers' yet still has very few free buffers.

ERROR: Since it's last reload, this router has created or maintained a 
relatively
large number of 'VeryBig buffers' yet still has very few free buffers.

The above symptoms suggest that a buffer leak has occurred.



Upgrading IOS does not help. I even changed the switch but this did not 
help either.

I also get:

#show controllers ethernet-controller gi0/45

      Transmit GigabitEthernet0/45             Receive
     372161753 Bytes                       3922461464 Bytes
    3323064978 Unicast frames              2309878058 Unicast frames
          3348 Multicast frames                246432 Multicast frames
        158248 Broadcast frames                 49637 Broadcast frames
             0 Too old frames              3251834333 Unicast bytes
             0 Deferred frames               16990019 Multicast bytes
             0 MTU exceeded frames            3176768 Broadcast bytes
             0 1 collision frames                   0 Alignment errors
             0 2 collision frames                   0 FCS errors
             0 3 collision frames                   0 Oversize frames
             0 4 collision frames                   0 Undersize frames
             0 5 collision frames                   0 Collision fragments
             0 6 collision frames
             0 7 collision frames               41549 Minimum size frames
             0 8 collision frames          1836935154 65 to 127 byte frames
             0 9 collision frames            22806228 128 to 255 byte frames
             0 10 collision frames          234511817 256 to 511 byte frames
             0 11 collision frames          115680563 512 to 1023 byte frames
             0 12 collision frames           18333995 1024 to 1518 byte frames
             0 13 collision frames                  0 Overrun frames
             0 14 collision frames                  0 Pause frames
             0 15 collision frames
             0 Excessive collisions                 0 Symbol error frames
             0 Late collisions                      0 Invalid frames, too large
             0 VLAN discard frames           81864821 Valid frames, too large
             0 Excess defer frames                  0 Invalid frames, too small
          2207 64 byte frames                       0 Valid frames, too small
     331806887 127 byte frames
      57469089 255 byte frames                      0 Too old frames
      94297557 511 byte frames                      0 Valid oversize frames
     135809581 1023 byte frames                     0 System FCS error frames
    1481837111 1518 byte frames                     0 RxPortFifoFull drop frame
    1222004142 Too large frames
             0 Good (1 coll) frames
             0 Good (>1 coll) frames


There are too many "Too large frames".


Where could be the problem with the switch?


Regards,
Anton Turygin