[j-nsp] Strange Packet Loss Problem
Christian Koch
ckoch at globix.com
Mon Oct 23 08:38:21 EDT 2006
Hey Josef,
Real quick, here is a regular ping which looks fine, and a snapshot of
system icmp stats
ckoch at core2.lhr3> ping 212.71.229.25
PING 212.71.229.25 (212.71.229.25): 56 data bytes
64 bytes from 212.71.229.25: icmp_seq=0 ttl=251 time=1.113 ms
64 bytes from 212.71.229.25: icmp_seq=1 ttl=251 time=1.063 ms
64 bytes from 212.71.229.25: icmp_seq=2 ttl=251 time=0.957 ms
64 bytes from 212.71.229.25: icmp_seq=3 ttl=251 time=0.995 ms
64 bytes from 212.71.229.25: icmp_seq=4 ttl=251 time=0.947 ms
64 bytes from 212.71.229.25: icmp_seq=5 ttl=251 time=1.429 ms
64 bytes from 212.71.229.25: icmp_seq=6 ttl=251 time=0.934 ms
64 bytes from 212.71.229.25: icmp_seq=7 ttl=251 time=1.017 ms
64 bytes from 212.71.229.25: icmp_seq=8 ttl=251 time=0.935 ms
64 bytes from 212.71.229.25: icmp_seq=9 ttl=251 time=0.973 ms
64 bytes from 212.71.229.25: icmp_seq=10 ttl=251 time=1.138 ms
64 bytes from 212.71.229.25: icmp_seq=11 ttl=251 time=1.003 ms
64 bytes from 212.71.229.25: icmp_seq=12 ttl=251 time=0.946 ms
64 bytes from 212.71.229.25: icmp_seq=13 ttl=251 time=1.172 ms
64 bytes from 212.71.229.25: icmp_seq=14 ttl=251 time=1.002 ms
64 bytes from 212.71.229.25: icmp_seq=15 ttl=251 time=1.009 ms
64 bytes from 212.71.229.25: icmp_seq=16 ttl=251 time=0.921 ms
64 bytes from 212.71.229.25: icmp_seq=17 ttl=251 time=1.009 ms
64 bytes from 212.71.229.25: icmp_seq=18 ttl=251 time=0.959 ms
64 bytes from 212.71.229.25: icmp_seq=19 ttl=251 time=0.973 ms
64 bytes from 212.71.229.25: icmp_seq=20 ttl=251 time=2.362 ms
64 bytes from 212.71.229.25: icmp_seq=21 ttl=251 time=1.034 ms
64 bytes from 212.71.229.25: icmp_seq=22 ttl=251 time=1.007 ms
64 bytes from 212.71.229.25: icmp_seq=23 ttl=251 time=0.913 ms
64 bytes from 212.71.229.25: icmp_seq=24 ttl=251 time=0.959 ms
64 bytes from 212.71.229.25: icmp_seq=25 ttl=251 time=1.021 ms
64 bytes from 212.71.229.25: icmp_seq=26 ttl=251 time=0.996 ms
64 bytes from 212.71.229.25: icmp_seq=27 ttl=251 time=0.966 ms
64 bytes from 212.71.229.25: icmp_seq=28 ttl=251 time=3.044 ms
64 bytes from 212.71.229.25: icmp_seq=29 ttl=251 time=0.940 ms
64 bytes from 212.71.229.25: icmp_seq=30 ttl=251 time=0.946 ms
64 bytes from 212.71.229.25: icmp_seq=31 ttl=251 time=1.054 ms
64 bytes from 212.71.229.25: icmp_seq=32 ttl=251 time=0.989 ms
64 bytes from 212.71.229.25: icmp_seq=33 ttl=251 time=0.973 ms
64 bytes from 212.71.229.25: icmp_seq=34 ttl=251 time=0.963 ms
64 bytes from 212.71.229.25: icmp_seq=35 ttl=251 time=1.028 ms
64 bytes from 212.71.229.25: icmp_seq=36 ttl=251 time=0.954 ms
64 bytes from 212.71.229.25: icmp_seq=37 ttl=251 time=0.965 ms
64 bytes from 212.71.229.25: icmp_seq=38 ttl=251 time=0.999 ms
64 bytes from 212.71.229.25: icmp_seq=39 ttl=251 time=1.003 ms
64 bytes from 212.71.229.25: icmp_seq=40 ttl=251 time=0.897 ms
64 bytes from 212.71.229.25: icmp_seq=41 ttl=251 time=1.126 ms
64 bytes from 212.71.229.25: icmp_seq=42 ttl=251 time=0.913 ms
64 bytes from 212.71.229.25: icmp_seq=43 ttl=251 time=1.058 ms
64 bytes from 212.71.229.25: icmp_seq=44 ttl=251 time=0.942 ms
64 bytes from 212.71.229.25: icmp_seq=45 ttl=251 time=1.030 ms
64 bytes from 212.71.229.25: icmp_seq=46 ttl=251 time=0.941 ms
64 bytes from 212.71.229.25: icmp_seq=47 ttl=251 time=1.047 ms
64 bytes from 212.71.229.25: icmp_seq=48 ttl=251 time=0.938 ms
64 bytes from 212.71.229.25: icmp_seq=49 ttl=251 time=0.954 ms
64 bytes from 212.71.229.25: icmp_seq=50 ttl=251 time=0.964 ms
64 bytes from 212.71.229.25: icmp_seq=51 ttl=251 time=0.976 ms
64 bytes from 212.71.229.25: icmp_seq=52 ttl=251 time=0.944 ms
64 bytes from 212.71.229.25: icmp_seq=53 ttl=251 time=1.006 ms
64 bytes from 212.71.229.25: icmp_seq=54 ttl=251 time=0.972 ms
64 bytes from 212.71.229.25: icmp_seq=55 ttl=251 time=1.585 ms
64 bytes from 212.71.229.25: icmp_seq=56 ttl=251 time=0.990 ms
64 bytes from 212.71.229.25: icmp_seq=57 ttl=251 time=11.160 ms
64 bytes from 212.71.229.25: icmp_seq=58 ttl=251 time=0.948 ms
64 bytes from 212.71.229.25: icmp_seq=59 ttl=251 time=21.018 ms
64 bytes from 212.71.229.25: icmp_seq=60 ttl=251 time=0.996 ms
64 bytes from 212.71.229.25: icmp_seq=61 ttl=251 time=30.768 ms
^C
--- 212.71.229.25 ping statistics ---
62 packets transmitted, 62 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.897/2.029/30.768/4.634 ms
ckoch at core2.lhr3> show system statistics icmp
icmp:
0 drops due to rate limit
3747728 calls to icmp_error
0 errors not generated because old message was icmp
Output histogram:
echo reply: 9220380
destination unreachable: 3745300
time exceeded: 1577
time stamp reply: 472
2 messages with bad code fields
0 messages less than the minimum length
338 messages with bad checksum
2 messages with bad source address
28 messages with bad length
15 echo drops with broadcast or multicast destinaton address
0 timestamp drops with broadcast or multicast destination
address
Input histogram:
echo reply: 50251
destination unreachable: 5631
source quench: 6
routing redirect: 829
#7: 2
echo: 9220395
time exceeded: 1883
parameter problem: 1
time stamp: 472
time stamp reply: 1
information request reply: 20
address mask reply: 2
9220852 message responses generated
-----Original Message-----
From: Josef Buchsteiner [mailto:josefb at juniper.net]
Sent: Saturday, October 21, 2006 9:27 AM
To: Christian Koch
Cc: juniper-nsp at puck.nether.net
Subject: Re: [j-nsp] Strange Packet Loss Problem
Christian,
if it is a data corruption on certain bit patterns then
you should see this problem also without rapid pings.
In order to find out where this happen I would
certainly perform a regular ping ( you may need more
packets ) and watch the icmp statistics to confirm that
it is data corruption in case you get icmp chksum
errors. "show system statistics icmp"
To find out if this is local or remote ( you could make
some statements with the icmp statistics already
however I prefer solid results)and you need to put this
sonet link into local loop and encapsulation cisco-hdlc
with no-keepalive. Then you perform the ping again with
bypass routing and interface knob to make sure the
traffic goes out and comes back in and you watch the
packet result and icmp statistics. This way you will
find out if this is local M40 or remote M40.
Once you know this it becomes kinda bit tricky. It
could be the FPC in question that is the trigger but
also other FPCs are contributing in the shared data
buffer which means you would need to turn off one by
one and see if the error stops. If you are left with
the OC-48 then I would suggest to replace this
particular FPC. To sanity check also other links of the
router if possible.
Check also the message log if you ever have seen
anytime ECC error reported. Just one of them is already
enough to give you some hints as you would not need to
do above isolation work
hope this helps
Josef
PS: yes I do have seen such symptoms in the past
and not only on one vendor.
Friday, October 20, 2006, 8:36:09 PM, you wrote:
CK>
CK>
CK> Hi All,
CK>
CK> I am experiencing a strange issue between 2 m40 core routers
CK> connect through an oc48 link
CK>
CK> I experience packet loss only when sending packets between 293 and
CK> 599 bytes.
CK>
CK> I am also speaking to cold telecom about the issue to see if it's a
CK> problem with the sonet link, but as of now the link is not taking
CK> any errors or alarms.
CK>
CK> Anyone seen anything strange like this before?
CK>
CK> ckoch at core2.lhr3> ping rapid 212.71.229.25 count 10000 size 300
CK> PING 212.71.229.25 (212.71.229.25): 300 data bytes
CK>
CK>
!.!!.!!.!!!!.!.!....!!.!!!!!..!..!!.!!!..!!.!!....!!.!...!!!...!.!.!..!.
CK> .!!...!....!..!.!.!!.!!.!!....!.!!!!!!..!..!.!!.!..!!.!!^C
CK> --- 212.71.229.25 ping statistics ---
CK> 129 packets transmitted, 66 packets received, 48% packet loss
CK> round-trip min/avg/max/stddev = 1.023/1.640/34.095/4.035 ms
CK>
CK> ckoch at core2.lhr3> ping rapid 212.71.229.25 count 10000 size 400
CK> PING 212.71.229.25 (212.71.229.25): 400 data bytes
CK> !!!!!!!!.!!!!!...!!^C
CK> --- 212.71.229.25 ping statistics --- 20 packets transmitted, 15
CK> packets received, 25% packet loss round-trip min/avg/max/stddev =
CK> 1.074/1.111/1.348/0.071 ms
CK>
CK> ckoch at core2.lhr3> ping rapid 212.71.229.25 count 10000 size 500
CK> PING 212.71.229.25 (212.71.229.25): 500 data bytes
CK> !!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!^C
CK> --- 212.71.229.25 ping statistics ---
CK> 49 packets transmitted, 46 packets received, 6% packet loss
CK> round-trip min/avg/max/stddev = 1.141/1.202/2.514/0.210 ms
CK>
CK> ckoch at core2.lhr3> ping rapid 212.71.229.25 count 10000 size 600
CK> PING 212.71.229.25 (212.71.229.25): 600 data bytes
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!^C
CK> --- 212.71.229.25 ping statistics ---
CK> 955 packets transmitted, 954 packets received, 0% packet loss
CK> round-trip min/avg/max/stddev = 1.183/1.526/36.436/1.817 ms
CK>
CK> koch at core2.lhr3> ping rapid 212.71.229.25 count 10000 size 292
CK> PING 212.71.229.25 (212.71.229.25): 292 data bytes
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK>
CK>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
CK> !!!!!!!!^C!
CK> --- 212.71.229.25 ping statistics ---
CK> 916 packets transmitted, 916 packets received, 0% packet loss
CK> round-trip min/avg/max/stddev = 0.988/1.146/9.918/1.061 ms
CK>
CK> ckoch at core2.lhr3> ping rapid 212.71.229.25 count 10000 size 293
CK> PING 212.71.229.25 (212.71.229.25): 293 data bytes ..!!!.!^C
CK>
CK> Christian
CK> _______________________________________________
CK> juniper-nsp mailing list juniper-nsp at puck.nether.net
CK> https://puck.nether.net/mailman/listinfo/juniper-nsp
CK>
CK>
CK>
More information about the juniper-nsp
mailing list