[c-nsp] ASR 9001 fan tray disappeared?

Vinny_Abello at Dell.com Vinny_Abello at Dell.com
Sun Oct 20 11:14:32 EDT 2013 

Hi all,

I had an odd event where the fan tray for one of my ASR 9001 routers suddenly disappeared. About 10 minutes later it returned and the alarm cleared. Apart from the obvious of someone having actually removed it, which I'm pretty certain did not happen, are there any known bugs that might cause this? I was going to open a TAC case, but I figured if someone has seen this before and knows this to be a known issue, I'll save the hassle. This device is running IOS XR 4.2.3. Relevant logs of the event below:

RP/0/RSP0/CPU0:Oct 20 07:42:19.759 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:BRINGDOWN
RP/0/RSP0/CPU0:Oct 20 07:42:19.767 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:NOT-PRESENT
RP/0/RSP0/CPU0:Oct 20 07:42:19.772 : invmgr[254]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/FT0/SP, state: BRINGDOWN
RP/0/RSP0/CPU0:Oct 20 07:42:22.497 : envmon[205]: %PLATFORM-ENVMON-2-FANTRAY_CLEAR : Fan tray failure cleared on slot 0/FT0/SP
RP/0/RSP0/CPU0:Oct 20 07:42:22.497 : shelfmgr[394]: %PLATFORM-SHELFMGR_HAL-7-CAN_INFO : Can API: slot 10 offline event.
RP/0/RSP0/CPU0:Oct 20 07:42:22.498 : eem_ed_hardware[188]: Messge received content : Event 9   NodeId: 0x4a0   Threshold FANTRAY_FAIL  Info Fantray Failure on 0/FT0/SP C  Severity 0x64  Frutype 0x20  Alarm_raised 0 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:42:22.775 : invmgr[254]: arb_lib_asic_get_ver: msg send problem: Invalid argument
RP/0/RSP0/CPU0:Oct 20 07:42:22.802 : invmgr[254]: %PLATFORM-INV-6-OIROUT : OIR: Node 0/FT0/SP removed
RP/0/RSP0/CPU0:Oct 20 07:42:22.803 : eem_ed_oir[190]: Messge received content : Event 1 NodeId: 0x4a0
RP/0/RSP0/CPU0:Oct 20 07:42:22.804 : eem_ed_hardware[188]: get_envmon_card_data : Bag found NULL, returning
RP/0/RSP0/CPU0:Oct 20 07:42:22.804 : envmon[205]: %PLATFORM-ENVMON-3-FANTRAY_ALARM_CLEAR : Fan tray alarm cleared on slot 0/FT0/SP
RP/0/RSP0/CPU0:Oct 20 07:42:22.805 : eem_ed_hardware[188]: Messge received content : Event 7   NodeId: 0x4a0   Threshold FANTRAY_ALARM  Info FANTRAY ALARM on 0/FT0/SP Cle  Severity 0x8  Frutype 0x20  Alarm_raised 0 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:42:22.809 : eem_ed_hardware[188]: get_envmon_card_data : Bag found NULL, returning
RP/0/RSP0/CPU0:Oct 20 07:42:22.810 : envmon[205]: %PLATFORM-ENVMON-2-ALL_FANTRAYS_MISSING : No fan trays present in the chassis, chassis cooling non-operational
RP/0/RSP0/CPU0:Oct 20 07:42:22.810 : eem_ed_hardware[188]: Messge received content : Event 1   NodeId: 0x1   Threshold All Fantrays Missing  Info All Fantrays Missing  Severity 0x64  Frutype 0x1  Alarm_raised 1 Alarm_type 1
RP/0/RSP0/CPU0:Oct 20 07:42:31.022 : envmon[205]: %PLATFORM-ENVMON-2-ALL_FANTRAYS_MISSING_CLEAR : One or more fan trays now plugged in, chassis cooling operational
RP/0/RSP0/CPU0:Oct 20 07:42:31.022 : eem_ed_hardware[188]: Messge received content : Event 1   NodeId: 0x1   Threshold ALL_FANTRAYS_MISSING  Info "All Fantrays Missing" Resolv  Severity 0x64  Frutype 0x1  Alarm_raised 0 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:42:31.459 : envmon[205]: %PLATFORM-ENVMON-2-ENV_CONDITION : Outstanding environmental condition cleared in the chassis
RP/0/RSP0/CPU0:Oct 20 07:42:34.761 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:PRESENT
RP/0/RSP0/CPU0:Oct 20 07:42:37.771 : invmgr[254]: arb_lib_asic_get_ver: msg send problem: Invalid argument
RP/0/RSP0/CPU0:Oct 20 07:42:37.782 : invmgr[254]: %PLATFORM-INV-6-OIRIN : OIR: Node 0/FT0/SP inserted
RP/0/RSP0/CPU0:Oct 20 07:42:37.782 : eem_ed_oir[190]: Messge received content : Event 0 NodeId: 0x4a0
RP/0/RSP0/CPU0:Oct 20 07:43:29.949 : envmon[205]: %PLATFORM-ENVMON-2-FANTRAY_FAIL : Fan tray failure on slot 0/FT0/SP
RP/0/RSP0/CPU0:Oct 20 07:43:29.949 : eem_ed_hardware[188]: Messge received content : Event 9   NodeId: 0x4a0   Threshold FANTRAY_FAIL  Info Fantray 0/FT0/SP Failed  Severity 0x64  Frutype 0x20  Alarm_raised 1 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:43:30.235 : canb-server[150]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC in slot 0/FT0/SP, alarm code CBC_ALRM_FT_LED_RED_FAILED
RP/0/RSP0/CPU0:Oct 20 07:43:30.265 : envmon[205]: %PLATFORM-ENVMON-2-FAN_FAIL : Fan failure on slot 0/FT0/SP, fan 1, RPM 0
RP/0/RSP0/CPU0:Oct 20 07:43:30.266 : eem_ed_hardware[188]: Messge received content : Event 6   NodeId: 0x4a0   Threshold FAN_FAIL  Info Fan: 1 RPM: 0  Severity 0x64  Frutype 0x20  Alarm_raised 1 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:43:30.590 : envmon[205]: %PLATFORM-ENVMON-2-ENV_CONDITION : Outstanding environmental condition exists in the chassis
RP/0/RSP0/CPU0:Oct 20 07:43:30.590 : envmon[205]: %PLATFORM-ENVMON-3-FANTRAY_ALARM : Fan tray alarm on slot 0/FT0/SP
RP/0/RSP0/CPU0:Oct 20 07:43:30.591 : eem_ed_hardware[188]: Messge received content : Event 7   NodeId: 0x4a0   Threshold FANTRAY_ALARM  Info FANTRAY ALARM on 0/FT0/SP  Severity 0x8  Frutype 0x20  Alarm_raised 1 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:43:30.950 : eem_ed_hardware[188]: Messge received content : Event 6   NodeId: 0x4a0   Threshold FAN_FAIL  Info Fan: 2 RPM: 0  Severity 0x64  Frutype 0x20  Alarm_raised 1 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:43:32.941 : envmon[205]: %PLATFORM-ENVMON-2-FAN_FAIL : Fan failure on slot 0/FT0/SP, fan 4, RPM 0
RP/0/RSP0/CPU0:Oct 20 07:43:32.941 : eem_ed_hardware[188]: Messge received content : Event 6   NodeId: 0x4a0   Threshold FAN_FAIL  Info Fan: 4 RPM: 0  Severity 0x64  Frutype 0x20  Alarm_raised 1 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:43:33.942 : envmon[205]: %PLATFORM-ENVMON-2-FAN_FAIL : Fan failure on slot 0/FT0/SP, fan 5, RPM 0
RP/0/RSP0/CPU0:Oct 20 07:43:33.942 : eem_ed_hardware[188]: Messge received content : Event 6   NodeId: 0x4a0   Threshold FAN_FAIL  Info Fan: 5 RPM: 0  Severity 0x64  Frutype 0x20  Alarm_raised 1 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:43:36.945 : envmon[205]: %PLATFORM-ENVMON-2-FAN_FAIL : Fan failure on slot 0/FT0/SP, fan 8, RPM 0
RP/0/RSP0/CPU0:Oct 20 07:43:36.945 : eem_ed_hardware[188]: Messge received content : Event 6   NodeId: 0x4a0   Threshold FAN_FAIL  Info Fan: 8 RPM: 0  Severity 0x64  Frutype 0x20  Alarm_raised 1 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:43:37.946 : envmon[205]: %PLATFORM-ENVMON-2-FAN_FAIL : Fan failure on slot 0/FT0/SP, fan 9, RPM 0
RP/0/RSP0/CPU0:Oct 20 07:43:37.946 : eem_ed_hardware[188]: Messge received content : Event 6   NodeId: 0x4a0   Threshold FAN_FAIL  Info Fan: 9 RPM: 0  Severity 0x64  Frutype 0x20  Alarm_raised 1 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:43:39.948 : envmon[205]: %PLATFORM-ENVMON-2-FAN_FAIL : Fan failure on slot 0/FT0/SP, fan 11, RPM 0
RP/0/RSP0/CPU0:Oct 20 07:43:39.948 : eem_ed_hardware[188]: Messge received content : Event 6   NodeId: 0x4a0   Threshold FAN_FAIL  Info Fan: 11 RPM: 0  Severity 0x64  Frutype 0x20  Alarm_raised 1 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:43:40.949 : envmon[205]: %PLATFORM-ENVMON-2-FAN_FAIL : Fan failure on slot 0/FT0/SP, fan 12, RPM 0
RP/0/RSP0/CPU0:Oct 20 07:43:40.949 : eem_ed_hardware[188]: Messge received content : Event 6   NodeId: 0x4a0   Threshold FAN_FAIL  Info Fan: 12 RPM: 0  Severity 0x64  Frutype 0x20  Alarm_raised 1 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:44:31.231 : canb-server[150]: %PLATFORM-CANB_SERVER-3-ALARM_CLEARED : Clear alarm from CBC in slot 0/FT0/SP, alarm code CBC_ALRM_FT_LED_RED_FAILED
RP/0/RSP0/CPU0:Oct 20 07:44:31.231 : canb-server[150]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC in slot 0/FT0/SP, alarm code CBC_ALRM_FT_LED_GREEN_FAILED
RP/0/RSP0/CPU0:Oct 20 07:50:24.763 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:BRINGDOWN
RP/0/RSP0/CPU0:Oct 20 07:50:24.769 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:NOT-PRESENT
RP/0/RSP0/CPU0:Oct 20 07:50:24.773 : invmgr[254]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/FT0/SP, state: BRINGDOWN
RP/0/RSP0/CPU0:Oct 20 07:50:26.850 : envmon[205]: %PLATFORM-ENVMON-2-FANTRAY_CLEAR : Fan tray failure cleared on slot 0/FT0/SP
RP/0/RSP0/CPU0:Oct 20 07:50:26.850 : shelfmgr[394]: %PLATFORM-SHELFMGR_HAL-7-CAN_INFO : Can API: slot 10 offline event.
RP/0/RSP0/CPU0:Oct 20 07:50:26.851 : eem_ed_hardware[188]: Messge received content : Event 9   NodeId: 0x4a0   Threshold FANTRAY_FAIL  Info Fantray Failure on 0/FT0/SP C  Severity 0x64  Frutype 0x20  Alarm_raised 0 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:50:27.777 : invmgr[254]: arb_lib_asic_get_ver: msg send problem: Invalid argument
RP/0/RSP0/CPU0:Oct 20 07:50:27.803 : invmgr[254]: %PLATFORM-INV-6-OIROUT : OIR: Node 0/FT0/SP removed
RP/0/RSP0/CPU0:Oct 20 07:50:27.803 : eem_ed_oir[190]: Messge received content : Event 1 NodeId: 0x4a0
RP/0/RSP0/CPU0:Oct 20 07:50:27.805 : eem_ed_hardware[188]: get_envmon_card_data : Bag found NULL, returning
RP/0/RSP0/CPU0:Oct 20 07:50:27.805 : envmon[205]: %PLATFORM-ENVMON-3-FANTRAY_ALARM_CLEAR : Fan tray alarm cleared on slot 0/FT0/SP
RP/0/RSP0/CPU0:Oct 20 07:50:27.806 : eem_ed_hardware[188]: Messge received content : Event 7   NodeId: 0x4a0   Threshold FANTRAY_ALARM  Info FANTRAY ALARM on 0/FT0/SP Cle  Severity 0x8  Frutype 0x20  Alarm_raised 0 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:50:27.811 : eem_ed_hardware[188]: get_envmon_card_data : Bag found NULL, returning
RP/0/RSP0/CPU0:Oct 20 07:50:27.811 : envmon[205]: %PLATFORM-ENVMON-2-ALL_FANTRAYS_MISSING : No fan trays present in the chassis, chassis cooling non-operational
RP/0/RSP0/CPU0:Oct 20 07:50:27.812 : eem_ed_hardware[188]: Messge received content : Event 1   NodeId: 0x1   Threshold All Fantrays Missing  Info All Fantrays Missing  Severity 0x64  Frutype 0x1  Alarm_raised 1 Alarm_type 1
RP/0/RSP0/CPU0:Oct 20 07:53:05.741 : envmon[205]: %PLATFORM-ENVMON-2-ALL_FANTRAYS_MISSING_CLEAR : One or more fan trays now plugged in, chassis cooling operational
RP/0/RSP0/CPU0:Oct 20 07:53:05.741 : shelfmgr[394]: %PLATFORM-SHELFMGR_HAL-7-CAN_INFO : Can API: slot 10 online event.
RP/0/RSP0/CPU0:Oct 20 07:53:05.742 : eem_ed_hardware[188]: Messge received content : Event 1   NodeId: 0x1   Threshold ALL_FANTRAYS_MISSING  Info "All Fantrays Missing" Resolv  Severity 0x64  Frutype 0x1  Alarm_raised 0 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:53:06.124 : envmon[205]: %PLATFORM-ENVMON-2-ENV_CONDITION : Outstanding environmental condition cleared in the chassis
RP/0/RSP0/CPU0:Oct 20 07:53:06.137 : envmon[205]: %PLATFORM-ENVMON-3-FANTRAY_ALARM : Fan tray alarm on slot 0/FT0/SP
RP/0/RSP0/CPU0:Oct 20 07:53:06.138 : eem_ed_hardware[188]: Messge received content : Event 7   NodeId: 0x4a0   Threshold FANTRAY_ALARM  Info FANTRAY ALARM on 0/FT0/SP  Severity 0x8  Frutype 0x20  Alarm_raised 1 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:53:06.140 : eem_ed_hardware[188]: get_envmon_card_data : Bag found NULL, returning
RP/0/RSP0/CPU0:Oct 20 07:53:06.141 : envmon[205]: %PLATFORM-ENVMON-2-ENV_CONDITION : Outstanding environmental condition exists in the chassis
RP/0/RSP0/CPU0:Oct 20 07:53:09.766 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:PRESENT
RP/0/RSP0/CPU0:Oct 20 07:53:12.776 : invmgr[254]: arb_lib_asic_get_ver: msg send problem: Invalid argument
RP/0/RSP0/CPU0:Oct 20 07:53:12.786 : invmgr[254]: %PLATFORM-INV-6-OIRIN : OIR: Node 0/FT0/SP inserted
RP/0/RSP0/CPU0:Oct 20 07:53:12.787 : eem_ed_oir[190]: Messge received content : Event 0 NodeId: 0x4a0
RP/0/RSP0/CPU0:Oct 20 07:53:34.764 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:BRINGDOWN
RP/0/RSP0/CPU0:Oct 20 07:53:34.770 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:NOT-PRESENT
RP/0/RSP0/CPU0:Oct 20 07:53:34.773 : invmgr[254]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/FT0/SP, state: BRINGDOWN
RP/0/RSP0/CPU0:Oct 20 07:53:37.779 : invmgr[254]: arb_lib_asic_get_ver: msg send problem: Invalid argument
RP/0/RSP0/CPU0:Oct 20 07:53:37.800 : invmgr[254]: %PLATFORM-INV-6-OIROUT : OIR: Node 0/FT0/SP removed
RP/0/RSP0/CPU0:Oct 20 07:53:37.801 : eem_ed_oir[190]: Messge received content : Event 1 NodeId: 0x4a0
RP/0/RSP0/CPU0:Oct 20 07:53:39.016 : envmon[205]: %PLATFORM-ENVMON-3-FANTRAY_ALARM_CLEAR : Fan tray alarm cleared on slot 0/FT0/SP
RP/0/RSP0/CPU0:Oct 20 07:53:39.016 : shelfmgr[394]: %PLATFORM-SHELFMGR_HAL-7-CAN_INFO : Can API: slot 10 offline event.
RP/0/RSP0/CPU0:Oct 20 07:53:39.017 : eem_ed_hardware[188]: Messge received content : Event 7   NodeId: 0x4a0   Threshold FANTRAY_ALARM  Info FANTRAY ALARM on 0/FT0/SP Cle  Severity 0x8  Frutype 0x20  Alarm_raised 0 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:53:39.020 : eem_ed_hardware[188]: get_envmon_card_data : Bag found NULL, returning
RP/0/RSP0/CPU0:Oct 20 07:53:39.020 : envmon[205]: %PLATFORM-ENVMON-2-ENV_CONDITION : Outstanding environmental condition cleared in the chassis
RP/0/RSP0/CPU0:Oct 20 07:53:39.020 : envmon[205]: %PLATFORM-ENVMON-2-ALL_FANTRAYS_MISSING : No fan trays present in the chassis, chassis cooling non-operational
RP/0/RSP0/CPU0:Oct 20 07:53:39.021 : eem_ed_hardware[188]: Messge received content : Event 1   NodeId: 0x1   Threshold All Fantrays Missing  Info All Fantrays Missing  Severity 0x64  Frutype 0x1  Alarm_raised 1 Alarm_type 1
RP/0/RSP0/CPU0:Oct 20 07:53:39.457 : envmon[205]: %PLATFORM-ENVMON-2-ENV_CONDITION : Outstanding environmental condition exists in the chassis
RP/0/RSP0/CPU0:Oct 20 07:53:42.239 : envmon[205]: %PLATFORM-ENVMON-2-ALL_FANTRAYS_MISSING_CLEAR : One or more fan trays now plugged in, chassis cooling operational
RP/0/RSP0/CPU0:Oct 20 07:53:42.239 : eem_ed_hardware[188]: Messge received content : Event 1   NodeId: 0x1   Threshold ALL_FANTRAYS_MISSING  Info "All Fantrays Missing" Resolv  Severity 0x64  Frutype 0x1  Alarm_raised 0 Alarm_type 2
RP/0/RSP0/CPU0:Oct 20 07:53:42.685 : envmon[205]: %PLATFORM-ENVMON-2-ENV_CONDITION : Outstanding environmental condition cleared in the chassis
RP/0/RSP0/CPU0:Oct 20 07:53:44.766 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:PRESENT
RP/0/RSP0/CPU0:Oct 20 07:53:47.779 : invmgr[254]: arb_lib_asic_get_ver: msg send problem: Invalid argument
RP/0/RSP0/CPU0:Oct 20 07:53:47.789 : invmgr[254]: %PLATFORM-INV-6-OIRIN : OIR: Node 0/FT0/SP inserted
RP/0/RSP0/CPU0:Oct 20 07:53:47.789 : eem_ed_oir[190]: Messge received content : Event 0 NodeId: 0x4a0
RP/0/RSP0/CPU0:Oct 20 07:54:41.444 : canb-server[150]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC in slot 0/FT0/SP, alarm code CBC_ALRM_FT_LED_RED_FAILED

Thanks!

-Vinny

More information about the cisco-nsp mailing list