[nsp] Strange ethernet behavior

[Aaron Howell]

Hi all, and welcome to this installment of crazy ethernet behavior.  Let
me start by apologizing for the length of this email, but this one is
odd.

  I've got a customer who is in the same office building as one of our
NOCs.  For the past few years they have had an ethernet connection
between our office and theirs, set up as follows:

   Us					  Them
Cisco        Catalyst          Cisco         Some
 7513 --->    3550    --->     2621 --->  switch


The cisco 7513 is our equipment, the cat3550 is our equipment, both
located in our NOC, the 2621 is the CUSTOMER'S equipment, also located
in our NOC, "Some switch" is the switch in the customer's suite.  (Yes,
the 2600 is massive overkill for a single 10MB ethernet feed, blame the
previous owners of our company, who sold it to them.)

  Up until recently, this has worked just fine, then all of a sudden,
they started having problems. They would lose connectivity several times
a day.  Initially, we tracked down several computers on their network
that were infected with blaster and nachi, took care of those, and
thought we'd be good to go.  A few days later, it happened again, but
this time no worm traffic, and their router itself was unreachable.  My
onsite tech couldn't find his console cable, so he just power cycled the
2600, everything came back up, looks fine.

  We get a console cable plugged into the 2600 so that I can see what's
going on when the router drops it's connectivity, customer calls say
it's down, and....I can't console into the router.  Ok, so it's a
hardware issue of some sort.  I recommend they run out and grab a cheap
DSL/Cable router (all they really needed in the first place, IMHO), and
we can get them fixed straight away.  My tech goes and buys one (an
Airlink AXOHO4P), installs it, everything looks good for about 15
minutes, then their connection goes down again.

(Insert many hours of pulling hair out and trying various things that
have no effect)

  So we come to this morning.  I find an unused ethernet interface on
the 7513, move customer to this, taking both the 2600 and the cat3550
out of the equation.  Customer now has their Airlink DSL router plugged
directly into our core.  Bring the interface up, everything looks good,
till 10-15 minutes later.  Can't ping their side of the connection.  sh
ip arp shows the correct MAC, on a whim I clear ip arp <cust. IP>,
magically it starts passing traffic again.  After some more
experimentation, I set the arp timeout to 5 minutes, and that seems to
be doing the trick at the moment.

  I've done some searching, but I can't find any information on a bug
similar to this, or anything really, that has simialr symptoms.  Has
anyone out there seen this sort of behavior, or have any ideas what may
be causing it?

Thanks,
Aaron Howell
--
Network/System Administrator
Amerion, LLC