[c-nsp] Switch Craziness and maybe STP or bad ARP's

[Joseph Jackson]

Hey all,

	Today I had the pleasure of having our network go insane.
Here's what happened at 1pm PST time I received alerts that I was unable
to ping any network eq in a secondary server room on our LAN.  When I
would ping address I would get back a TTL exceeded.  The pings would
show a reply address as coming from another interface on our router
(completely different subnet also)  example

My workstations setup
Ip address 10.8.69.10
Subnet mask 255.255.0.0
Default gateway 10.8.4.1

Ping 10.8.24.3 (a switch in a data closet that feeds half the building)

First I would get a time out and then I would get a reply that looked
like this

Reply from 10.64.6.1 TTL Exceeded, ttl=255, 170 ms (which is a interface
on our core 7206 that goes into another subnet)
Or close enough to that effect.  Weird I thought.. Why would
communication within the same broadcast domain be getting a reply back
from a interface on our core router.

After going around the entire building and shutting down switches we
finally traced it to a small work group switch that was coming off of
our core LAN switch once we brought that down
All communication come back up normally.  As its now late and I want to
go home I haven't had a chance to test out the switch on our test
network yet but I was wondering what you guys thoughts 
Were on this.  Why did that happen?  Is there any monitoring tools that
I could have used to track it down much faster than the hours it took
for us?  We don't use any STP and no redundant connections on the
switches.. Any thoughts?


Thanks

Joseph Jackson