[c-nsp] 6500 - FABRIC-SP-3-DISABLE_FAB

[Drew Weaver]

Hi,

We've been struggling with an issue on one of our 6500s about a week or so. It started out where the system would run on the supervisor in slot 8 for about 16-24 hours, then fail over to the secondary supervisor on slot 7 for "no" reason, then this error would be presented, and then it would immediately flip back to slot 8.

Originally the error message was:

Jan 12 09:36:03.144 EST: %FABRIC-SP-3-DISABLE_FAB: The fabric manager disabled active fabric in slot 7 due to the error (2) on this channel (FPOE 8) connected to slot 4

We re-seated the card in slot 4, and eventually replaced it and everything seemed to finally stabilize. This morning, as a test I forced it to switchover to the card in slot 7 to see if it would immediately switch back to the card in slot 8 it did not, and I was fairly pleased, however we now got this error: 

Jan 12 09:36:03.144 EST: %FABRIC-SP-3-DISABLE_FAB: The fabric manager disabled active fabric in slot 7 due to the error (2) on this channel (FPOE 8) connected to slot 13

We went to re-seat the completely unused 6548 card in slot 13 (this is a 6513) and it caused a failover again.

Jan 12 09:35:10.353 EST: %OIR-SP-6-REMCARD: Card removed from slot 13, interfaces disabled
Jan 12 09:36:03.144 EST: %SYS-SP-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.
Jan 12 09:36:03.144 EST: %FABRIC-SP-3-DISABLE_FAB: The fabric manager disabled active fabric in slot 7 due to the error (2) on this channel (FPOE 8) connected to slot 13
Jan 12 09:36:03.144 EST: %OIR-SP-3-PWRCYCLE: Card in module 7, is being power-cycled off (Fabric channel errors)

Anyone have any thoughts as to what might be occurring here? We can replace the card in slot 13 as well but we are concerned about the exciting game of musical fabric errors the switch is playing.

-Drew