[c-nsp] 3550 Port Fault that only forwards small packets

[Paul Cosgrove]

Keep in mind that when you specify 16000 for the icmp data size, the PC
is fragmenting those packets and sending multiple separate frames; it is
not actually sending 16000 bytes of data in each frame as that would
produce a frame much larger than the interface MTU (which is probably
1500).  With a 1500 byte MTU the largest ICMP data size you can specify
is 1472.

Similarly the switch may also be fragmenting and sending multiple frames
when you specify 2000.

Faulty cabling can cause issues like this where error rate increases
with increasing packet size.  The larger the packet, the greater the
chance of it being corrupted during transmission.  I know this sounds
odd when you are seeing no errors at all with small packets but remember
that the 1000 byte pings will each take 10 times longer to transmit on
the wire than a 100 byte ping.

Cleaning the fibre and connectors may help (using a proper fibre
cleaning kit).  You have probably already tested with a different patch
lead to the 2940.

I can only hazzard a guess about why the PC works but the switch
doesn't.  If the path of the traffic is similar in each case, then
perhaps they are actually sending different size frames, i.e. the PC is
using a smaller frame size than the switch.  Would be surprising though
if it is using <1000.

When pinging from the PC set the DF bit to prevent fragmentation, e.g.
ping -f -l 1472 x.x.x.x .  This will let you determine the end to end
MTU to the destination.  Then increase the number of pings you are
sending using the '-n' option to say 5000 and see if any errors occur.

When pinging from the switch use an extended ping with a size value 28
bytes larger than you used with the PC.  Keep the repeat count the same.
  Both will then be sending the same sized frames and the results should
be easier to compare.

Run a traceroute from each as well if there is any possibility of a
different path being used.

Paul.

Kevin.X.White at corusgroup.com wrote:
> 
> Further info following response:
> 
> Failures get progressively worse pinging from a switch as packet size
> increases, 10/10 from a PC at any size
> Traffic levels are low in general, path from  the ping source 2950 below is
> Gb to core>>> Gb to other core >>>Gb to distrib 3550>>> 100Mb to 2nd
> distrib 3550 >>> 100Mb suspect interface to test 2940
> From the ping source to the 3550 with the failed interface 16K packets are
> fine just not acros the last faulty interface to the Test 2940.
> 
> 
> Voice VLAN: none (Inactive)
> Appliance trust: none
> Name: Fa0/3
> Switchport: Enabled
> Administrative Mode: dynamic desirable
> Operational Mode: trunk
> Administrative Trunking Encapsulation: dot1q
> Operational Trunking Encapsulation: dot1q
> Negotiation of Trunking: On
> Access Mode VLAN: 1 (default)
> Trunking Native Mode VLAN: 1 (default)
> Administrative private-vlan host-association: none
> Administrative private-vlan mapping: none
> Operational private-vlan: none
> Trunking VLANs Enabled: ALL
> Pruning VLANs Enabled: 2-1001
> 
> FastEthernet0/3 is up, line protocol is up
>   Hardware is Fast Ethernet, address is 0009.xxxx.xxxx (bia 0009.xxxx.xxxx)
>   Description: faulty
>   MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
>      reliability 255/255, txload 1/255, rxload 1/255
>   Encapsulation ARPA, loopback not set
>   Keepalive set (10 sec)
>   Full-duplex, 100Mb/s
>   input flow-control is off, output flow-control is off
>   ARP type: ARPA, ARP Timeout 04:00:00
>   Last input 00:00:01, output 00:00:00, output hang never
>   Last clearing of "show interface" counters 4d19h
>   Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
>   Queueing strategy: fifo
>   Output queue :0/40 (size/max)
>   5 minute input rate 0 bits/sec, 0 packets/sec
>   5 minute output rate 0 bits/sec, 0 packets/sec
>      303603 packets input, 197232924 bytes, 0 no buffer
>      Received 90320 broadcasts, 0 runts, 0 giants, 0 throttles
>      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
>      0 watchdog, 90288 multicast, 0 pause input
>      0 input packets with dribble condition detected
>      11901675 packets output, 1954047601 bytes, 0 underruns
>      0 output errors, 0 collisions, 0 interface resets
>      0 babbles, 0 late collision, 0 deferred
>      0 lost carrier, 0 no carrier, 0 PAUSE output
>      0 output buffer failures, 0 output buffers swapped out
> 
> 
> 
> From a win2K PC several hops away @ 16K:
> 
> Reply from  xx.xx.xx.xx: bytes=16000 time=78ms TTL=253
> Reply from  xx.xx.xx.xx: bytes=16000 time=76ms TTL=253
> Reply from  xx.xx.xx.xx: bytes=16000 time=77ms TTL=253
> Reply from  xx.xx.xx.xx: bytes=16000 time=77ms TTL=253
> Reply from  xx.xx.xx.xx: bytes=16000 time=76ms TTL=253
> 
> Ping statistics for  xx.xx.xx.xx:
>     Packets: Sent = 265, Received = 265, Lost = 0 (0% loss),
> Approximate round trip times in milli-seconds:
>     Minimum = 76ms, Maximum =  81ms, Average =  76ms
> Control-C
> 
> From a 2950c several hops away:
> 
> Sending 5, 100-byte ICMP Echos to xx.xx.xx.xx, timeout is 2 seconds:
> !!!!!
> Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/4 ms
> 
> Sending 5, 1000-byte ICMP Echos to  xx.xx.xx.xx, timeout is 2 seconds:
> .!!..
> Success rate is 40 percent (2/5), round-trip min/avg/max = 4/6/8 ms
> 
> Sending 5, 2000-byte ICMP Echos to  xx.xx.xx.xx, timeout is 2 seconds:
> ..!!!
> Success rate is 60 percent (3/5), round-trip min/avg/max = 12/12/12 ms
> 
> Sending 5, 2000-byte ICMP Echos to  xx.xx.xx.xx, timeout is 2 seconds:
> ....!
> Success rate is 20 percent (1/5), round-trip min/avg/max = 12/12/12 ms
> 
>> Hi Kevin,
>>
>> Do the switches show high cpu or high throughput on those ports?  If you
>> post the interface stats (e.g. show int switching, sh int, sh vlan)
>> perhaps it will help.
> 
>> Are the two limits exactly 500 & 1000? I wasn't clear from your email if
>> this was the case or if you were giving an approx range. Is there a very
>> clear cut off after 1000 bytes, with no replies at all after that?
> 
>> If the switch IP and PC were in the same vlan then the only differences
>> I can think of are that the extended ping may be sending at a faster
>> rate than a windows ping; the frame size is different if you enter 1000
>> for each (the frame sent by windows would be 28 bytes larger); and the
>> switch may perhaps(?) drop or rate limit locally generated/destined icmp
>> before cef switched transit traffic if it is under heavy load.  ICMP is
>> not normally considered high priority traffic.
> 
>> What kind of traffic would you expect on the network, much multicasts or
>> broadcasts for instance?
> 
>> Paul.
> 
> 
> **********************************************************************
> This transmission is confidential and must not be used or disclosed by
> anyone other than the intended recipient. Neither Corus Group Limited nor
> any of its subsidiaries can accept any responsibility for any use or
> misuse of the transmission by anyone.
> 
> For address and company registration details of certain entities
> within the Corus group of companies, please visit
> http://www.corusgroup.com/entities
> 
> **********************************************************************
> 
> 


-- 
Paul Cosgrove
HEAnet Limited, Ireland's Education and Research Network
1st Floor, 5 George's Dock, IFSC, Dublin 1
Registered in Ireland, no 275301
tel: +353-1-660 9040  fax: +353-1-660 3666
web: http://www.heanet.ie/