[f-nsp] MCT issue on ICX6650's

[Frank Bulk]

I have pair of ICX6650's running ICXR08001.bin configured using MCT.  This
morning our NOC called me out of bed because a 10G link was down on our
transport.  After some poking around on the second ICX6650, trying to figure
out why it wouldn't' bring the 10G port up, I discover that CCP
communication were down between the two ICX6650's.  Bring a port down on an
MCLAG is normal behavior for the non-master shelf, to prevent spanning-tree
weirdness.

Not only was CCP communications down, I also couldn't SSH or telnet to the
first ICX6650, and the first ICX6650's BGP session was down.  Note that all
those are TCP-based protocols.  SNMP (which uses UDP) was fine.

I used SNMP to extract this from the first ICX6650:

"01:42:07 Central Fri Nov 15 2013: "CLUSTER FSM: Cluster R1 (Id: 1), client
Cyan (RBridge Id: 380) - Remote client CCEP state when CCP is down: up or
unknown "
 "01:42:07 Central Fri Nov 15 2013: "CLUSTER FSM: Cluster R1 (Id: 1), client
Cyan (RBridge Id: 380) - Remote client CCEP state when CCP is down: down "
 "01:44:48 Central Fri Nov 15 2013: "CLUSTER FSM: Cluster R1 (Id: 1), client
Cyan (RBridge Id: 380) - Remote client CCEP state when CCP is down: up or
unknown "
 "01:44:48 Central Fri Nov 15 2013: "CLUSTER FSM: Cluster R1 (Id: 1), client
Cyan (RBridge Id: 380) - Remote client CCEP state when CCP is down: down "
 "01:48:13 Central Fri Nov 15 2013: "CLUSTER FSM: Cluster R1 (Id: 1), client
Cyan (RBridge Id: 380) - Remote client CCEP state when CCP is down: up or
unknown "
 "01:48:13 Central Fri Nov 15 2013: "CLUSTER FSM: Cluster R1 (Id: 1), client
Cyan (RBridge Id: 380) - Remote client CCEP state when CCP is down: down "
 "01:50:08 Central Fri Nov 15 2013: "CLUSTER FSM: Cluster R1 (Id: 1), client
Cyan (RBridge Id: 380) - Remote client CCEP state when CCP is down: up or
unknown "
 "01:50:08 Central Fri Nov 15 2013: "CLUSTER FSM: Cluster R1 (Id: 1), client
Cyan (RBridge Id: 380) - Remote client CCEP state when CCP is down: down "

The second ICX6650 had its log buffer filled with these:
2013 Nov 15 02:35:40:A: TCP connection close message came, Closed the CCP
Session

I don't have serial-based OOB (don't me started on the pseudo-OOB IP mgmt
that's on Brocade devices), so I couldn't do any troubleshooting of the
first ICX6650.  I restored services by using our PDU to remotely power-cycle
it.

Anyone else have this kind of issue, where there's the management plane
seems to have lost its TCP functionality?  

Regards,

Frank