[c-nsp] 20G Port-Channel Drops
Albert Goerend
albert.goerend at gmail.com
Wed Mar 10 16:30:14 EST 2010
I run a 2x 10G Port-Channel (PAGP) between two 6509 boxes.
Physical interfaces are:
te8/3 and te8/4 on box1
te4/1 and te4/2 on box2
Both linecards are WS-X6704-10GE (CFC, no DFC).
The port-channel is fine until the outbound traffic from box2 to box1
goes beyond ~ 8 GBPs. Traffic then significally drops down to 4-5
GBPs, although the opposite should happen: more bandwidth > 8G!!
As a consequence a few events can be observed:
a) high latency. Both boxes are only a few 100ft. away. Under normal
circumstances latency is < 1ms, but now it is between 6-8 ms
b) packet loss between 5-25%
c) massive input queue drops and overruns on the port-channel and the
physical interfaces, but only on box1. box2 has no drops and overruns.
there are no output queue drops
I can rule out bad cabling. I tried it with different pairs of single
mode fibers, the results are always the same.
Some configs:
****************************
BOX1 (after having cleared the counters shortly before!!)
interface Port-channel25
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,99
switchport mode trunk
hold-queue 4096 in
interface TenGigabitEthernet8/3
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,99
switchport mode trunk
no cdp enable
channel-group 25 mode desirable
hold-queue 4096 in
interface TenGigabitEthernet8/4
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,99
switchport mode trunk
no cdp enable
channel-group 25 mode desirable
hold-queue 4096 in
#sh int te8/3
TenGigabitEthernet8/3 is up, line protocol is up (connected)
Hardware is C6k 10000Mb 802.3, address is 001a.6c97.d076 (bia 001a.6c97.d076)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 35/255, rxload 55/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is 10Gbase-LR
input flow-control is on, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 00:50:55
Input queue: 0/4096/39430539/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2160739000 bits/sec, 353615 packets/sec
5 minute output rate 1389353000 bits/sec, 265032 packets/sec
1092321725 packets input, 848337038871 bytes, 0 no buffer
Received 83931 broadcasts (50474 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 39430539 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
826180894 packets output, 540347147697 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
#sh int te8/4
TenGigabitEthernet8/4 is up, line protocol is up (connected)
Hardware is C6k 10000Mb 802.3, address is 001a.6c97.d077 (bia 001a.6c97.d077)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 33/255, rxload 55/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is 10Gbase-LR
input flow-control is on, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 00:51:52
Input queue: 0/4096/34481197/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2162120000 bits/sec, 347715 packets/sec
5 minute output rate 1328243000 bits/sec, 254235 packets/sec
1083368037 packets input, 858740641278 bytes, 0 no buffer
Received 91178 broadcasts (55459 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 34481201 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
791867650 packets output, 509737273378 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
#sh int port-channel 25
Port-channel25 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 001a.6c97.d076 (bia 001a.6c97.d076)
MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec,
reliability 255/255, txload 34/255, rxload 55/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is unknown
input flow-control is on, output flow-control is off
Members in this channel: Te9/3 Te9/4
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 00:52:24
Input queue: 0/4096/75545675/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 4337063000 bits/sec, 703124 packets/sec
5 minute output rate 2732391000 bits/sec, 520229 packets/sec
2221968796 packets input, 1742752045593 bytes, 0 no buffer
Received 179112 broadcasts (108243 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 75545675 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
1652325958 packets output, 1072874496131 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
****************************
BOX2:
interface Port-channel1
switchport
switchport access vlan 99
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,99
switchport mode trunk
no ip address
interface TenGigabitEthernet4/1
switchport
switchport access vlan 99
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,99
switchport mode trunk
no ip address
channel-group 1 mode desirable
!
interface TenGigabitEthernet4/2
switchport
switchport access vlan 99
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,99
switchport mode trunk
no ip address
channel-group 1 mode desirable
#sh int te4/1
TenGigabitEthernet4/1 is up, line protocol is up (connected)
Hardware is C6k 10000Mb 802.3, address is 0017.5a74.c100 (bia 0017.5a74.c100)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 54/255, rxload 34/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s
input flow-control is off, output flow-control is desired
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 00:45:20
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1346303000 bits/sec, 260307 packets/sec
5 minute output rate 2137862000 bits/sec, 349946 packets/sec
737141947 packets input, 482718766191 bytes, 0 no buffer
Received 33195 broadcasts (27489 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
974766586 packets output, 758313398451 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
#sh int te4/2
TenGigabitEthernet4/2 is up, line protocol is up (connected)
Hardware is C6k 10000Mb 802.3, address is 0017.5a74.c101 (bia 0017.5a74.c101)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 54/255, rxload 32/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s
input flow-control is off, output flow-control is desired
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 00:45:57
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1278026000 bits/sec, 248450 packets/sec
5 minute output rate 2125812000 bits/sec, 342069 packets/sec
703270291 packets input, 451833566861 bytes, 0 no buffer
Received 93622 broadcasts (28170 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
963715736 packets output, 765779444992 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
#sh int port-channel 1
Port-channel1 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0017.5a74.c100 (bia 0017.5a74.c100)
MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec,
reliability 255/255, txload 54/255, rxload 33/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s
input flow-control is off, output flow-control is off
Members in this channel: Te4/1 Te4/2
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 00:46:49
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2659169000 bits/sec, 513030 packets/sec
5 minute output rate 4287070000 bits/sec, 695423 packets/sec
1477429388 packets input, 958861847398 bytes, 0 no buffer
Received 130220 broadcasts (57162 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
1987983366 packets output, 1562114553455 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
What really puzzles me in this set-up is that if my port-channel wants
to go beyond this magic number of 8G, all this is starting to happen.
I know that 6704 has one ASIC for port 1&2 and one for 3&4. Is there a
hardware limitation? I would understand 8G per port, so the limitation
would be at 16G on a channel. Both physical ports are equally balanced
by the way, so not a single port is carrying all the traffic.
I tried to tweak with hold-queue values, but no luck.
Any idea what this might be?
Am I hitting a limitation or am I missing something so trivial that I
should feel ashamed?
- Albert
More information about the cisco-nsp
mailing list