[c-nsp] Re: Dropped packets to a specific subnet(Not physical)
Kim Onnel
karim.adel at gmail.com
Sat Feb 25 05:55:54 EST 2006
I am suspecting its not layer 1 because its only happening to hosts in this
subnet (172.31.10.14), there are other devices on the switch as well, who do
not suffer from this, so i thought it maybe a routing problem, i submitted
below the interfaces on the routers and tommorow, i will be able to put the
switch and firewall.
WAN_ROUTER#sh int Fa2/1
FastEthernet2/1 is up, line protocol is up
Hardware is DEC21140A, address is 0007.ec79.5a90 (bia 0007.ec79.5a90)
Description: *** LINK TO MCE ***
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 21/255, rxload 14/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 1d22h
Input queue: 37/75/689/1592204 (size/max/drops/flushes); Total output
drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 5554000 bits/sec, 2037 packets/sec
30 second output rate 8617000 bits/sec, 2590 packets/sec
257175694 packets input, 556239078 bytes
Received 1469130 broadcasts, 0 runts, 92 giants, 368 throttles
1229 input errors, 1229 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
350477676 packets output, 1216085188 bytes, 255 underruns
255 output errors, 255 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
WAN_ROUTER#
Please note that WAN_ROUTER is an internet PE in an MPLS cloud of an ISP, so
there's alot behind it, i can see the underruns, errors, collisions,
throttles, giants :) but does that explain why certain hosts drop packets.
LAN_ROUTER#sh int Fas 2/1.1
FastEthernet2/1.1 is up, line protocol is up
Hardware is DEC21140A, address is 0007.ec79.5ade (bia 0007.ec79.5ade)
Description: *** Connection with WAN_ROUTER ***
Internet address is 172.31.10.50/30
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 16/255, rxload 20/255
Encapsulation ISL Virtual LAN, Color 10.
ARP type: ARPA, ARP Timeout 04:00:00
LAN_ROUTER#sh int Fas 2/1
FastEthernet2/1 is up, line protocol is up
Hardware is DEC21140A, address is 0007.ec79.5ade (bia 0007.ec79.5ade)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 16/255, rxload 20/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 205 drops; input queue 2/75, 24616 drops
30 second input rate 7944000 bits/sec, 2490 packets/sec
30 second output rate 6412000 bits/sec, 1953 packets/sec
2827954517 packets input, 385489239 bytes
Received 26651167 broadcasts, 0 runts, 46 giants, 16715 throttles
25317 input errors, 25316 CRC, 0 frame, 60 overrun, 9 ignored
0 watchdog
2 input packets with dribble condition detected
1509643285 packets output, 1620877593 bytes, 203179 underruns
203179 output errors, 0 collisions, 28 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
LAN_ROUTER#
LAN_ROUTER#sh int Fas 1/1
FastEthernet1/1 is up, line protocol is up
Hardware is DEC21140A, address is 0007.ec79.5ace (bia 0007.ec79.5ace)
Internet address is xx.xx.110.251/26
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 20/255, rxload 12/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 13594112 drops
30 second input rate 5014000 bits/sec, 1927 packets/sec
30 second output rate 8072000 bits/sec, 2445 packets/sec
1992291019 packets input, 2788861825 bytes
Received 8284521 broadcasts, 0 runts, 0 giants, 1042424 throttles
15 input errors, 0 CRC, 0 frame, 2631 overrun, 315098 ignored
0 watchdog
0 input packets with dribble condition detected
17971387 packets output, 1078666893 bytes, 4881184 underruns
4881184 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
5 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
LAN_ROUTER#
Also note that i got the above snapshots before clearing the counters, i
have cleared them and waited for 5 minutes, tried again and there were no
errors, CRC,..
Please check my problem description, as i said it was working without any
drops, until i changed the subnet, not the physical connectivity, i'll still
change the cabling tommorow and submit all interfaces counters all the way.
On 2/23/06, Kim Onnel <karim.adel at gmail.com> wrote:
>
> Hi,
>
> In our Datacenter, there is a couple of servers in the subnet 172.31.10.0,
> alot of packets to this subnet are dropped, i attached a network diagram,
> configurations of the devices in the diagram and ping snapshots and routing
> information, all other servers in different subnets reply normally.
>
> Pinging the Firewall outside interface
>
> WAN_ROUTER#ping
> Protocol [ip]:
> Target IP address: xx.xx.110.252
> Repeat count [5]: 100
> Datagram size [100]: 1500
> Timeout in seconds [2]:
> Extended commands [n]:
> Sweep range of sizes [n]:
> Type escape sequence to abort.
> Sending 100, 1500-byte ICMP Echos to xx.xx.110.252, timeout is 2 seconds:
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/4 ms
> WAN_ROUTER#
>
>
> Pinging DNS server
>
> WAN_ROUTER#ping
> Protocol [ip]:
> Target IP address: xx.xx.110.197
> Repeat count [5]: 100
> Datagram size [100]: 1500
> Timeout in seconds [2]:
> Extended commands [n]:
> Sweep range of sizes [n]:
> Type escape sequence to abort.
> Sending 100, 1500-byte ICMP Echos to xx.xx.110.197, timeout is 2 seconds:
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> Success rate is 100 percent (100/100), round-trip min/avg/max = 1/2/4 ms
> WAN_ROUTER#
>
>
>
>
> WAN_ROUTER#ping
> Protocol [ip]:
> Target IP address: 172.31.10.25
> Repeat count [5]: 10
> Datagram size [100]: 1500
> Timeout in seconds [2]:
> Extended commands [n]:
> Sweep range of sizes [n]:
> Type escape sequence to abort.
> Sending 10, 1500-byte ICMP Echos to 172.31.10.25, timeout is 2 seconds:
> !.!!!!!.!.
> Success rate is 70 percent (7/10), round-trip min/avg/max = 1/2/4 ms
>
>
>
> SSH_SERVER:~# tcpdump icmp
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
> 16:01:52.521633 IP 172.31.10.49 > 172.31.10.25: ICMP echo request, id
> 26363, seq 0, length 1480
> 16:01:52.536081 IP 172.31.10.25 > 172.31.10.49: ICMP echo reply, id 26363,
> seq 0, length 1480
> 16:01:54.522317 IP 172.31.10.49 > 172.31.10.25: ICMP echo request, id
> 26363, seq 2, length 1480
> 16:01:54.522422 IP 172.31.10.25 > 172.31.10.49: ICMP echo reply, id 26363,
> seq 2, length 1480
> 16:01:54.524919 IP 172.31.10.49 > 172.31.10.25: ICMP echo request, id
> 26363, seq 3, length 1480
> 16:01:54.525017 IP 172.31.10.25 > 172.31.10.49: ICMP echo reply, id 26363,
> seq 3, length 1480
> 16:01:54.527720 IP 172.31.10.49 > 172.31.10.25: ICMP echo request, id
> 26363, seq 4, length 1480
> 16:01:54.527819 IP 172.31.10.25 > 172.31.10.49: ICMP echo reply, id 26363,
> seq 4, length 1480
> 16:01:54.530296 IP 172.31.10.49 > 172.31.10.25: ICMP echo request, id
> 26363, seq 5, length 1480
> 16:01:54.530387 IP 172.31.10.25 > 172.31.10.49: ICMP echo reply, id 26363,
> seq 5, length 1480
> 16:01:54.533734 IP 172.31.10.49 > 172.31.10.25: ICMP echo request, id
> 26363, seq 6, length 1480
> 16:01:54.533832 IP 172.31.10.25 > 172.31.10.49: ICMP echo reply, id 26363,
> seq 6, length 1480
> 16:01:54.536519 IP 172.31.10.49 > 172.31.10.25: ICMP echo request, id
> 26363, seq 7, length 1480
> 16:01:56.534605 IP 172.31.10.49 > 172.31.10.25: ICMP echo request, id
> 26363, seq 8, length 1480
> 16:01:56.534710 IP 172.31.10.25 > 172.31.10.49: ICMP echo reply, id 26363,
> seq 8, length 1480
> 16:01:56.537562 IP 172.31.10.49 > 172.31.10.25: ICMP echo request, id
> 26363, seq 9, length 1480
> 16:01:56.537658 IP 172.31.10.25 > 172.31.10.49: ICMP echo reply, id 26363,
> seq 9, length 1480
>
>
>
> WAN_ROUTER#sh ip route 172.31.10.25
> Routing entry for 172.31.10.0/27
> Known via "static", distance 1, metric 0
> Redistributing via ospf 99, ospf 101, bgp 65000
> Advertised by ospf 101 subnets
> bgp 65000
> Routing Descriptor Blocks:
> * 172.31.10.50
> Route metric is 0, traffic share count is 1
>
>
> WAN_ROUTER#sh run | i ip route 172.31.10
> ip route 172.31.10.0 255.255.255.224 172.31.10.50
> ip route 172.31.10.29 255.255.255.255 172.31.15.130
> ip route 172.31.10.32 255.255.255.240 172.31.10.50
> WAN_ROUTER#
>
>
> WAN_ROUTER#sh ip route | i 172.31.10.
>
> C 172.31.10.48/30 is directly connected, FastEthernet2/1.1
> S 172.31.10.32/28 [1/0] via 172.31.10.50
> S 172.31.10.29/32 [1/0] via 172.31.15.130
> S 172.31.10.0/27 [1/0] via 172.31.10.50
>
>
> WAN_ROUTER#sh ip route 172.31.10.50
> Routing entry for 172.31.10.48/30
> Known via "connected", distance 0, metric 0 (connected, via interface)
> Redistributing via bgp 65000
> Advertised by bgp 65000
> Routing Descriptor Blocks:
> * directly connected, via FastEthernet2/1.1
> Route metric is 0, traffic share count is 1
>
>
>
> WAN_ROUTER#sh run int FastEthernet2/1
> Building configuration...
>
> Current configuration : 248 bytes
> !
> interface FastEthernet2/1
> description *** LINK TO LAN ROUTER ***
> no ip address
> ip access-group block-worms in
> ip access-group block-worms out
> ip route-cache flow
> load-interval 30
> duplex full
> no cdp enable
> end
> WAN_ROUTER#
>
> WAN_ROUTER#sh run int FastEthernet2/1.1
> !
> interface FastEthernet2/1.1
> description *** Connection with LAN ROUTER ***
> encapsulation isl 10
> ip address 172.31.10.49 255.255.255.252
> ip access-group block-worms in
> ip access-group block-worms out
> no ip redirects
> ip ospf hello-interval 5
> ip ospf dead-interval 15
> no cdp enable
> end
>
>
> Moving to LAN_ROUTER:
>
> Pinging the SSH server again:
>
> LAN_ROUTER#ping
> Protocol [ip]:
> Target IP address: 172.31.10.25
> Repeat count [5]: 10
> Datagram size [100]: 1500
> Timeout in seconds [2]:
> Extended commands [n]:
> Sweep range of sizes [n]:
> Type escape sequence to abort.
> Sending 10, 1500-byte ICMP Echos to 172.31.10.25, timeout is 2 seconds:
> !!!!!.!!.!
> Success rate is 80 percent (8/10), round-trip min/avg/max = 1/2/4 ms
> LAN_ROUTER#
>
> zazu:~# tcpdump icmp
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
> 16:10:16.681544 IP xx.xx.110.251 > 172.31.10.25: ICMP echo request, id
> 5359, seq 2409, length 1480
> 16:10:16.695414 IP 172.31.10.25 > xx.xx.110.251: ICMP echo reply, id 5359,
> seq 2409, length 1480
> 16:10:16.683176 IP xx.xx.110.251 > 172.31.10.25: ICMP echo request, id
> 5360, seq 2409, length 1480
> 16:10:16.683277 IP 172.31.10.25 > xx.xx.110.251: ICMP echo reply, id 5360,
> seq 2409, length 1480
> 16:10:16.684854 IP xx.xx.110.251 > 172.31.10.25: ICMP echo request, id
> 5361, seq 2409, length 1480
> 16:10:16.684985 IP 172.31.10.25 > xx.xx.110.251: ICMP echo reply, id 5361,
> seq 2409, length 1480
> 16:10:16.686815 IP xx.xx.110.251 > 172.31.10.25: ICMP echo request, id
> 5362, seq 2409, length 1480
> 16:10:16.686931 IP 172.31.10.25 > xx.xx.110.251: ICMP echo reply, id 5362,
> seq 2409, length 1480
> 16:10:16.688918 IP xx.xx.110.251 > 172.31.10.25: ICMP echo request, id
> 5363, seq 2409, length 1480
> 16:10:16.689004 IP 172.31.10.25 > xx.xx.110.251: ICMP echo reply, id 5363,
> seq 2409, length 1480
> 16:10:16.690580 IP xx.xx.110.251 > 172.31.10.25: ICMP echo request, id
> 5364, seq 2409, length 1480
> 16:10:18.689945 IP xx.xx.110.251 > 172.31.10.25: ICMP echo request, id
> 5365, seq 2409, length 1480
> 16:10:18.690052 IP 172.31.10.25 > xx.xx.110.251: ICMP echo reply, id 5365,
> seq 2409, length 1480
> 16:10:18.691900 IP xx.xx.110.251 > 172.31.10.25: ICMP echo request, id
> 5366, seq 2409, length 1480
> 16:10:18.691999 IP 172.31.10.25 > xx.xx.110.251: ICMP echo reply, id 5366,
> seq 2409, length 1480
> 16:10:18.693729 IP xx.xx.110.251 > 172.31.10.25: ICMP echo request, id
> 5367, seq 2409, length 1480
> 16:10:20.693596 IP xx.xx.110.251 > 172.31.10.25: ICMP echo request, id
> 5368, seq 2409, length 1480
> 16:10:20.693704 IP 172.31.10.25 > xx.xx.110.251: ICMP echo reply, id 5368,
> seq 2409, length 1480
>
> LAN_ROUTER#sh ip route 172.31.10.25
> Routing entry for 172.31.10.0/27
> Known via "static", distance 1, metric 0
> Routing Descriptor Blocks:
> * xx.xx.110.252
> Route metric is 0, traffic share count is 1
>
> LAN_ROUTER#sh ip route xx.xx.110.252
> Routing entry for xx.xx.110.192/26
> Known via "connected", distance 0, metric 0 (connected, via interface)
> Routing Descriptor Blocks:
> * directly connected, via FastEthernet1/1
> Route metric is 0, traffic share count is 1
>
> LAN_ROUTER#sh ip route | i 172.31.10.
> Gateway of last resort is 172.31.10.49 to network 0.0.0.0
>
> C 172.31.10.48/30 is directly connected, FastEthernet2/1.1
> S 172.31.10.29/32 is directly connected, FastEthernet2/1.1
> S 172.31.10.0/27 [1/0] via xx.xx.110.252
> S* 0.0.0.0/0 [1/0] via 172.31.10.49
> LAN_ROUTER#
>
>
> LAN_ROUTER#sh run int FastEthernet1/1
>
> interface FastEthernet1/1
> .....Alot of secondary IPs omitted....
> ip address xx.xx.110.3 255.255.255.192 secondary
> ip address xx.xx.110.251 255.255.255.192
> ip access-group block-worms in
> ip access-group block-worms out
> ip nat inside
> rate-limit input access-group rate-limit 100 2048000 5000 5000
> conform-action transmit exceed-action transmit
> rate-limit output access-group rate-limit 100 2048000 5000 5000
> conform-action transmit exceed-action transmit
> load-interval 30
> full-duplex
> no cdp enable
> end
>
>
> LAN_ROUTER#sh access-lists rate-limit 100
>
> Rate-limit access list 100
> 00B0.D064.8774
>
> LAN_ROUTER#sh arp | i .252
> Internet xx.xx.110.252 0 00d0.b782.3aa3 ARPA
> FastEthernet1/1
> LAN_ROUTER#sh arp | i 172.31.10.25
> LAN_ROUTER#sh arp | i 172.31.10.
> Internet 172.31.10.49 160 0007.ec79.5a90 ARPA
> FastEthernet2/1.1
> Internet 172.31.10.50 - 0007.ec79.5ade ARPA
> FastEthernet2/1.1
> Internet 172.31.10.29 160 0007.ec79.5a90 ARPA
> FastEthernet2/1.1
> LAN_ROUTER#
>
>
> LAN_ROUTER#sh ip access-lists block-worms
> Extended IP access list block-worms
> deny tcp any any eq 5554 (21672 matches)
> deny tcp any any range 135 139 (141805760 matches)
> deny udp any any range 135 netbios-ss (247404899 matches)
> deny tcp any any eq 445 (7593990 matches)
> deny udp any any eq 1026 (196450466 matches)
>
>
>
>
> Moving to PIX:
>
> : Saved
> : Written by enable_15 at 11:03:17.364 UTC Tue Feb 14 2006
>
> PIX Version 6.3(1)
>
> interface ethernet0 100basetx
> interface ethernet1 100basetx
> interface ethernet2 100basetx
> interface ethernet3 100basetx
> interface ethernet4 100basetx
> interface ethernet5 auto shutdown
> interface gb-ethernet0 1000auto
> interface gb-ethernet1 1000auto
>
> nameif ethernet0 collocated security50
> nameif ethernet1 NOC security60
> nameif ethernet2 dmz2 security40
> nameif ethernet3 DMZ3 security55
> nameif ethernet5 intf5 security10
> nameif gb-ethernet0 inside security100
> nameif gb-ethernet1 outside security0
>
> hostname PIX-OUTSIDE
> domain-name ciscopix.com
>
> fixup protocol ftp 21
> fixup protocol ftp 2009
> fixup protocol ftp 2100
> fixup protocol ftp 2200
> fixup protocol ftp 2201
> fixup protocol ftp 2202
> fixup protocol ftp 2203
> fixup protocol ftp 5000
> fixup protocol h323 h225 1720
> fixup protocol h323 ras 1718-1719
> fixup protocol http 80
> fixup protocol ils 389
> fixup protocol rsh 514
> fixup protocol rtsp 554
> fixup protocol sip 5060
> fixup protocol sip udp 5060
> fixup protocol skinny 2000
> no fixup protocol smtp 25
> fixup protocol sqlnet 1521
> names
>
>
>
> access-list external permit tcp any host 172.31.10.25 eq ssh
> access-list external permit udp any host 172.31.10.25 eq 2055
> access-list external permit udp any host 172.31.10.25 eq syslog
>
>
> mtu collocated 1500
> mtu NOC 1500
> mtu dmz2 1500
> mtu DMZ3 1500
>
> mtu intf5 1500
> mtu inside 1500
> mtu outside 1500
>
>
> ip address NOC 172.31.10.1 255.255.255.224
> ip address dmz2 10.0.9.1 255.255.255.0
> ip address DMZ3 10.0.11.1 255.255.255.0
>
> no ip address intf5
>
> ip address outside xx.xx.110.252 255.255.255.192
>
> ip audit info action alarm
> ip audit attack action alarm
>
> ip local pool ippool1 192.168.1.25-192.168.1.28
>
> pdm history enable
> arp timeout 14400
> nat (netcentrex) 0 access-list 110
>
> static (NOC,outside) 172.31.10.0 172.31.10.0 netmask 255.255.255.224 0 0
>
> access-group dmz2 in interface dmz2
> access-group dmz3 in interface DMZ3
>
> access-group external in interface outside
>
> route outside 0.0.0.0 0.0.0.0 xx.xx.110.251 1
>
> timeout xlate 3:00:00
> timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 rpc 0:10:00 h225
> 1:00:00
> timeout h323 0:05:00 mgcp 0:05:00 sip 0:30:00 sip_media 0:02:00
> timeout uauth 0:05:00 absolute
>
> floodguard enable
> telnet timeout 5
> ssh timeout 5
> console timeout 4
> terminal width 80
>
>
>
>
>
>
>
>
>
More information about the cisco-nsp
mailing list