[c-nsp] Cisco 7600 with line card 6708 and interface output drops

[Aivars]

At some point in time we noticed that one of the interfaces on 6708 card is having output errors:
C7600# sh int te3/5
TenGigabitEthernet3/5 is down, line protocol is down (notconnect)
  Hardware is C7600 10Gb 802.3, address is xxxxxx
  Description: xxxxx
  MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec, 
     reliability 237/255, txload 0/255, rxload 0/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s
  Transport mode LAN (10GBASE-R, 10.3125Gb/s)
  input flow-control is off, output flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 1d01h, output 1d01h, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     735991387981 packets input, 564956548776674 bytes, 0 no buffer
     Received 129092 broadcasts (129082 multicasts)
     1 runts, 4294967295 giants, 0 throttles 
     2 input errors, 2 CRC, 1 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     733857269101 packets output, 617935592654457 bytes, 0 underruns
     1006094320 output errors, 0 collisions, 3 interface resets
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

I know that this is 2:1 oversubscribed card and that the other interface in the pair is Te3/7:
TenGigabitEthernet3/7 is up, line protocol is up (connected)
  Hardware is C7600 10Gb 802.3, address is xxxxx
  Description: xxxxx
  Internet address is 192.168.0.0/xx
  MTU 9216 bytes, BW 10000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 110/255, rxload 131/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Carrier delay is 0 msec
  Full-duplex, 10Gb/s
  Transport mode LAN (10GBASE-R, 10.3125Gb/s)
  input flow-control is off, output flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/2248/2248 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 5171150000 bits/sec, 734833 packets/sec
  5 minute output rate 4314582000 bits/sec, 663382 packets/sec
  L2 Switched: ucast: 69581248 pkt, 5785437525 bytes - mcast: 1927660 pkt, 167102181 bytes
  L3 in Switched: ucast: 1018070600245 pkt, 884936045028239 bytes - mcast: 0 pkt, 0 bytes mcast
  L3 out Switched: ucast: 1684697143214 pkt, 1386736429250007 bytes mcast: 0 pkt, 0 bytes
     1017983838354 packets input, 884575887492641 bytes, 0 no buffer
     Received 1972229 broadcasts (0 IP multicasts)
     0 runts, 533967501 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     1684308105008 packets output, 1392637653336108 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out 
3/7 at the same time has 0 output errors.
Qos on 3/5 and 3/7 shows 0 drops:
C7600#sh mls qos queuing interface te3/5 | b dropp
  Packets dropped on Transmit:

    queue     dropped  [cos-map]
    ---------------------------------------------
    1                        0  [0 1 ]
    2                        0  [2 3 4 ]
    3                        0  [6 7 ]
    4                        0  []
    5                        0  []
    6                        0  []
    7                        0  []
    8                        0  [5 ]

  Packets dropped on Receive:
    BPDU packets:  0

    queue              dropped  [cos-map]
    ---------------------------------------------------
    1                        0  [0 1 2 3 4 5 6 7 ]
    2                        0  []
    3                        0  []
    4                        0  []
    5                        0  []
    6                        0  []
    7                        0  []
    8                        0  []
C7600#sh mls qos queuing interface te3/7 | b dropp
  Packets dropped on Transmit:

    queue     dropped  [cos-map]
    ---------------------------------------------
    1                        0  [0 1 ]
    2                        0  [2 3 4 ]
    3                        0  [6 7 ]
    4                        0  []
    5                        0  []
    6                        0  []
    7                        0  []
    8                        0  [5 ]

  Packets dropped on Receive:
    BPDU packets:  0

    queue              dropped  [cos-map]
    ---------------------------------------------------
    1                        0  [0 1 2 3 4 5 6 7 ]
    2                        0  []
    3                        0  []
    4                        0  []
    5                        0  []
    6                        0  []
    7                        0  []
    8                        0  []

    If this is oversurscription issue, why there were errors only on
    one interface in the pair?

The workaround was to move the connection together with X2 module to interface Te3/4 (3/1 has quite low traffic).
A little additional information. At the moment we are running IOS 151-3.S4. This 7600 has several full BGP feeds in VRF. To preserve memory, we moved to label per VRF model for the VRF that has full table.

Any ideas on what can be the root cause?



 Aivars