[c-nsp] 6500 interface going administratively down

Fri May 2 14:34:08 EDT 2008

On Thu, May 01, 2008 at 02:20:10PM -0400, James Ashton wrote:
> About a dozen times a week the interface on the secondary 6509 goes
> admin down and all of my HSRP groups go live. The interface comes
> back up, HSRP re-negotiates and everything goes back to normal.

I'd say HSRP isn't your problem - unreliable interfaces are.  I've
seen repeated issues like this when using non-Cisco GBICs.  We had a
batch of bad transceivers from ebay caused a very similar issue in a
very similar configuration.

The fact that the secondary switch reports that the HSRP virtual MAC
is flapping further makes me think you've got a communication problem
between the two switches.  I'd debug transciever to see if there's
something fishy going on there.

I would also suggest that you convert the trunk between those
switches to a port-channel.  This will insulate from an individual
link failing, but there's nothing to say you won't see both go admin
down...

Ross

> 
> 
> This causes about 20 seconds of outage across the network.
> 
> Any thoughts?
> 
> 
> The logs for the primary for this event are:
> 
> May  1 11:03:31 10.10.5.100 23121: .May  1 11:01:50.314 EST:
> %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/1,
> changed state to down May  1 11:03:31 10.10.5.100 23122: .May  1
> 11:01:50.386 EST: %LINK-3-UPDOWN: Interface GigabitEthernet5/1,
> changed state to down May  1 11:03:32 10.10.5.100 23123: May  1
> 11:01:50.386 EST: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface
> GigabitEthernet5/1, changed state to down May  1 11:03:33
> 10.10.5.100 23124: May  1 11:01:50.386 EST: %LINK-SP-3-UPDOWN:
> Interface GigabitEthernet5/1, changed state to down May  1 11:03:33
> 10.10.5.100 23125: .May  1 11:01:52.034 EST: %LINK-3-UPDOWN:
> Interface GigabitEthernet5/1, changed state to up May  1 11:03:34
> 10.10.5.100 23126: May  1 11:01:52.035 EST: %LINK-SP-3-UPDOWN:
> Interface GigabitEthernet5/1, changed state to up May  1 11:03:36
> 10.10.5.100 23127: .May  1 11:01:54.998 EST: %LINEPROTO-5-UPDOWN:
> Line protocol on Interface GigabitEthernet5/1, changed state to up
> May  1 11:03:37 10.10.5.100 23128: May  1 11:01:55.012 EST:
> %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface
> GigabitEthernet5/1, changed state to up
> 
> 
> And for the secondary:
> 
> May  1 11:03:32 10.10.5.101 230399714: .May  1 11:01:43.460 EST:
> %LINK-5-CHANGED: Interface GigabitEthernet5/1, changed state to
> administratively down May  1 11:03:32 10.10.5.101 230399715: .May  1
> 11:01:43.464 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface
> GigabitEthernet5/1, changed state to down May  1 11:03:32
> 10.10.5.101 230399716: May  1 11:01:43.467 EST: %LINK-SP-5-CHANGED:
> Interface GigabitEthernet5/1, changed state to administratively down
> May  1 11:03:32 10.10.5.101 230399717: May  1 11:01:43.479 EST:
> %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface
> GigabitEthernet5/1, changed state to down May  1 11:03:33
> 10.10.5.101 230399718: .May  1 11:01:44.704 EST: %LINK-3-UPDOWN:
> Interface GigabitEthernet5/1, changed state to down May  1 11:03:33
> 10.10.5.101 230399719: .May  1 11:01:45.576 EST: %LINK-3-UPDOWN:
> Interface GigabitEthernet5/1, changed state to up May  1 11:03:34
> 10.10.5.101 230399720: May  1 11:01:44.702 EST: %LINK-SP-3-UPDOWN:
> Interface GigabitEthernet5/1, changed state to down May  1 11:03:34
> 10.10.5.101 230399721: May  1 11:01:45.574 EST: %LINK-SP-3-UPDOWN:
> Interface GigabitEthernet5/1, changed state to up May  1 11:03:37
> 10.10.5.101 230399722: .May  1 11:01:48.541 EST:
> %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/1,
> changed state to up May  1 11:03:38 10.10.5.101 230399723: May  1
> 11:01:48.561 EST: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface
> GigabitEthernet5/1, changed state to up May  1 11:04:06 10.10.5.101
> 230399724: .May  1 11:02:18.593 EST: %STANDBY-6-STATECHANGE: Vlan148
> Group 0 state Standby -> Active May  1 11:04:06 10.10.5.101
> 230399725: .May  1 11:02:18.617 EST: %STANDBY-6-STATECHANGE: Vlan404
> Group 0 state Standby -> Active
> 
> ~~~ For each of 100 vlans ~~~
> 
> May  1 11:04:12 10.10.5.101 230399783: .May  1 11:02:24.409 EST:
> %STANDBY-6-STATECHANGE: Vlan23 Group 0 state Active -> Speak May  1
> 11:04:18 10.10.5.101 230399784: May  1 11:02:30.865 EST:
> %MAC_MOVE-SP-4-NOTIF: Host 0000.0c07.ac00 in vlan 21 is flapping
> between port Gi5/1 and port Router May  1 11:04:18 10.10.5.101
> 230399785: May  1 11:02:30.865 EST: %MAC_MOVE-SP-4-NOTIF: Host
> 0000.0c07.ac00 in vlan 24 is flapping between port Gi5/1 and port
> Router May  1 11:04:18 10.10.5.101 230399786: May  1 11:02:30.865
> EST: %MAC_MOVE-SP-4-NOTIF: Host 0000.0c07.ac00 in vlan 215 is
> flapping between port Gi5/1 and port Router May  1 11:04:18
> 10.10.5.101 230399787: May  1 11:02:30.865 EST:
> %MAC_MOVE-SP-4-NOTIF: Host 0000.0c07.ac00 in vlan 111 is flapping
> between port Gi5/1 and port Router May  1 11:04:18 10.10.5.101
> 230399788: May  1 11:02:30.865 EST: %MAC_MOVE-SP-4-NOTIF: Host
> 0000.0c07.ac00 in vlan 8 is flapping between port Gi5/1 and port
> Router May  1 11:04:19 10.10.5.101 230399789: May  1 11:02:30.865
> EST: %MAC_MOVE-SP-4-NOTIF: Host 0000.0c07.ac00 in vlan 157 is
> flapping between port Gi5/1 and port Router May  1 11:04:19
> 10.10.5.101 230399790: May  1 11:02:30.865 EST:
> %MAC_MOVE-SP-4-NOTIF: Host 0000.0c07.ac00 in vlan 157 is flapping
> between port Gi5/1 and port Router May  1 11:04:19 10.10.5.101
> 230399791: May  1 11:02:30.865 EST: %MAC_MOVE-SP-4-NOTIF: Host
> 0000.0c07.ac00 in vlan 53 is flapping between port Gi5/1 and port
> Router May  1 11:04:19 10.10.5.101 230399792: May  1 11:02:30.865
> EST: %MAC_MOVE-SP-4-NOTIF: Host 0000.0c07.ac00 in vlan 46 is
> flapping between port Gi5/1 and port Router May  1 11:04:19
> 10.10.5.101 230399793: May  1 11:02:30.865 EST:
> %MAC_MOVE-SP-4-NOTIF: Host 0000.0c07.ac00 in vlan 51 is flapping
> between port Gi5/1 and port Router May  1 11:04:20 10.10.5.101
> 230399794: May  1 11:02:30.865 EST: %MAC_MOVE-SP-4-NOTIF: Host
> 0000.0c07.ac00 in vlan 51 is flapping between port Gi5/1 and port
> Router May  1 11:04:20 10.10.5.101 230399795: May  1 11:02:30.865
> EST: %MAC_MOVE-SP-4-NOTIF: Host 0000.0c07.ac00 in vlan 232 is
> flapping between port Gi5/1 and port Router May  1 11:04:20
> 10.10.5.101 230399796: May  1 11:02:30.865 EST:
> %MAC_MOVE-SP-4-NOTIF: Host 0000.0c07.ac00 in vlan 232 is flapping
> between port Gi5/1 and port Router May  1 11:04:20 10.10.5.101
> 230399797: May  1 11:02:30.865 EST: %MAC_MOVE-SP-4-NOTIF: Host
> 0000.0c07.ac00 in vlan 179 is flapping between port Gi5/1 and port
> Router May  1 11:04:20 10.10.5.101 230399798: May  1 11:02:30.865
> EST: %MAC_MOVE-SP-4-NOTIF: Host 0000.0c07.ac00 in vlan 64 is
> flapping between port Gi5/1 and port Router
> _______________________________________________ cisco-nsp mailing
> list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp archive at
> http://puck.nether.net/pipermail/cisco-nsp/

-- 
Ross Vandegrift
ross at kallisti.us

"The good Christian should beware of mathematicians, and all those who
make empty prophecies. The danger already exists that the mathematicians
have made a covenant with the devil to darken the spirit and to confine
man in the bonds of Hell."
	--St. Augustine, De Genesi ad Litteram, Book II, xviii, 37