[j-nsp] m10i Nastiness Friday night

Clue Store cluestore at gmail.com
Mon Aug 17 10:58:34 EDT 2009 

Hi All,

Last friday we had some nastiness on one of our m10i's. As I am not a
Juniper expert, I was wondering if someone could decipher the log messages
and determine if is possibly a CFEB issue, or just a fluke Junos issue and
whether I should do anything or let it be and see if it does it again. I
have another m10i running 8.5, so I am thinking of just upgrading this box
to the same as my other, but i'd like to hear what some of you on the list
think.

TIA,
Clue

Hostname: JuniperM10i-HMNDLAMA
Model: m10i
JUNOS Base OS boot [8.0R2.8]
JUNOS Base OS Software Suite [8.0R2.8]
JUNOS Kernel Software Suite [8.0R2.8]
JUNOS Packet Forwarding Engine Support (M7i/M10i) [8.0R2.8]
JUNOS Routing Software Suite [8.0R2.8]
JUNOS Online Documentation [8.0R2.8]


Aug 14 23:38:51  JuniperM10i-HMNDLAMA cfeb mpc106 machine check caused by
error on the Processor Bus
Aug 14 23:38:51  JuniperM10i-HMNDLAMA cfeb mpc106 PCI status register:
0x0020, error detect register 1: 0x00, 2: 0x08
Aug 14 23:38:51  JuniperM10i-HMNDLAMA cfeb mpc106 error ack count = 0
Aug 14 23:38:51  JuniperM10i-HMNDLAMA cfeb mpc106 error address: 0x0f3827f8
Aug 14 23:38:51  JuniperM10i-HMNDLAMA cfeb mpc106 Processor bus error status
register: 0x52
Aug 14 23:38:51  JuniperM10i-HMNDLAMA cfeb transfer type 0b01010, transfer
size 2
Aug 14 23:38:51  JuniperM10i-HMNDLAMA cfeb mpc106 error detection reg2: ECC
multibit
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb ^B
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb last message repeated 6 times
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Context: Interrupt Level (0)
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Registers:
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb R00: 0x00000446 R01: 0x00799450
R02: 0x00000000 R03: 0x4f3827fc
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb R04: 0x00000552 R05: 0x00000000
R06: 0x007994a0 R07: 0x00000004
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb R08: 0x00000548 R09: 0x0017f48b
R10: 0x00000002 R11: 0xb0c7d8ec
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb R12: 0x28002044 R13: 0x02420020
R14: 0xf1ae2100 R15: 0x82600020
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb R16: 0x442104c2 R17: 0x2248000b
R18: 0x00670000 R19: 0x00670000
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb R20: 0x00670000 R21: 0x006ce5a0
R22: 0x007902d0 R23: 0x00670000
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb R24: 0x00000002 R25: 0x00000004
R26: 0x0080bd40 R27: 0x0000ffff
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb R28: 0x00000001 R29: 0x00000001
R30: 0x4f38271c R31: 0x4f382714
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb MSR: 0x00089030 CTR: 0x00000239
Link:0x002e34c8 SP:  0x00799450
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb CCR: 0x48002028 XER: 0x20000000
PC:  0x00460320
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb DSISR: 0x00000000 DAR: 0x00000000
K_MSR: 0x00000030
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Stack Traceback:
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 01: sp = 0x00799450, pc =
0x0000c001
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 02: sp = 0x00799468, pc =
0x002e4d74
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 03: sp = 0x00799498, pc =
0x002e35e0
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 04: sp = 0x007994b8, pc =
0x002e3bb0
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 05: sp = 0x007994c0, pc =
0x00058818
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 06: sp = 0x007994d8, pc =
0x0003df34
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 07: sp = 0x00799500, pc =
0x003b4488
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 08: sp = 0x00799530, pc =
0x003b4660
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 09: sp = 0x00799548, pc =
0x003b3ed0
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 10: sp = 0x007995c8, pc =
0x003b3d30
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 11: sp = 0x007995e8, pc =
0x000b9f6c
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 12: sp = 0x00799610, pc =
0x000b8928
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 13: sp = 0x00799628, pc =
0x00448518
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 14: sp = 0x00799678, pc =
0x00442d00
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 15: sp = 0x00799698, pc =
0x0003a500
Aug 14 23:38:52  JuniperM10i-HMNDLAMA cfeb Frame 16: sp = 0x007996b0, pc =
0x0003b268
Aug 14 23:38:56  JuniperM10i-HMNDLAMA /kernel: rdp keepalive expired,
connection dropped - src 1:1021 dest 2:15360
Aug 14 23:38:56  JuniperM10i-HMNDLAMA craftd[2999]:  Major alarm set, CFEB
not online, the box is not forwarding
Aug 14 23:38:56  JuniperM10i-HMNDLAMA alarmd[2998]: Alarm set: CFEB
color=RED, class=CHASSIS, reason=CFEB not online, the box is not forwarding
Aug 14 23:38:56  JuniperM10i-HMNDLAMA craftd[2999]: forwarding display
request to chassisd: type = 4, subtype = 43
Aug 14 23:38:56  JuniperM10i-HMNDLAMA chassisd[2997]:
CHASSISD_SHUTDOWN_NOTICE: Shutdown reason: CFEB connection lost
Aug 14 23:38:56  JuniperM10i-HMNDLAMA chassisd[2997]:
CHASSISD_IFDEV_DETACH_FPC: ifdev_detach(0)
Aug 14 23:38:56  JuniperM10i-HMNDLAMA mib2d[3111]: SNMP_TRAP_LINK_DOWN:
ifIndex 77, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/0

(Lots of BGP notifications due to interface down issues)

Aug 14 23:38:56  JuniperM10i-HMNDLAMA snmpd[3132]: SNMPD_SEND_FAILURE:
trap_io_send_trap_now: send to (207.29.223.55) failure: Network is down
Aug 14 23:38:56  JuniperM10i-HMNDLAMA alarmd[2998]: shutting down chassisd
connection: chassisd ipc pipe read error
Aug 14 23:38:56  JuniperM10i-HMNDLAMA craftd[2999]:
craftd_user_conn_shutdown: socket 5, errno = 0
Aug 14 23:38:56  JuniperM10i-HMNDLAMA craftd[2999]: chassisd connection
succeeded after 0 retries
Aug 14 23:38:56  JuniperM10i-HMNDLAMA alarmd[2998]: chassisd connection
succeeded after 0 retries
Aug 14 23:38:56  JuniperM10i-HMNDLAMA mib2d[3111]: SNMP_TRAP_LINK_DOWN:
ifIndex 80, ifAdminStatus down(2), ifOperStatus down(2), ifName ge-1/0/0.462
Aug 14 23:38:56  JuniperM10i-HMNDLAMA alarmd[2998]: resending alarm state
Aug 14 23:38:56  JuniperM10i-HMNDLAMA craftd[2999]: forwarding display
request to chassisd: type = 4, subtype = 43
Aug 14 23:38:56  JuniperM10i-HMNDLAMA alarmd[2998]: Alarm set: CFEB
color=RED, class=CHASSIS, reason=CFEB not online, the box is not forwarding
Aug 14 23:38:56  JuniperM10i-HMNDLAMA alarmd[2998]: Alarm set: RE color=RED,
class=CHASSIS, reason=Host 0 fxp0: Ethernet Link Down
Aug 14 23:38:56  JuniperM10i-HMNDLAMA craftd[2999]: forwarding display
request to chassisd: type = 4, subtype = 43
Aug 14 23:38:56  JuniperM10i-HMNDLAMA alarmd[2998]: Alarm set: RE color=RED,
class=CHASSIS, reason=Host 1 fxp0: Ethernet Link Down
Aug 14 23:38:56  JuniperM10i-HMNDLAMA craftd[2999]: forwarding display
request to chassisd: type = 4, subtype = 43
Aug 14 23:38:56  JuniperM10i-HMNDLAMA /kernel: rdp keepalive expired,
connection dropped - src 1:1020 dest 2:15361
Aug 14 23:38:56  JuniperM10i-HMNDLAMA /kernel: pfe_listener_disconnect: conn
dropped: listener idx=0, tnpaddr=0x2, reason: socket error
Aug 14 23:39:41  JuniperM10i-HMNDLAMA chassisd[2997]:
CHASSISD_BLOWERS_SPEED_FULL: Fans and impellers being set to full speed
[system warm]
Aug 14 23:40:09  JuniperM10i-HMNDLAMA chassisd[2997]: CHASSISD_SNMP_TRAP10:
SNMP trap generated: FRU power on (jnxFruContentsIndex 6, jnxFruL1Index 1,
jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName CFEB 0, jnxFruType 4,
jnxFruSlot 1, jnxFruOfflineReason 2, jnxFruLastPowerOff 0, jnxFruLastPowerOn
0)
Aug 14 23:40:09  JuniperM10i-HMNDLAMA chassisd[2997]:
CHASSISD_IFDEV_DETACH_FPC: ifdev_detach(0)
Aug 14 23:40:09  JuniperM10i-HMNDLAMA chassisd[2997]:
CHASSISD_IFDEV_DETACH_FPC: ifdev_detach(1)
Aug 14 23:40:09  JuniperM10i-HMNDLAMA chassisd[2997]:
CHASSISD_IFDEV_DETACH_ALL_PSEUDO: ifdev_detach(pseudo devices: all)
Aug 14 23:40:09  JuniperM10i-HMNDLAMA craftd[2999]: Major alarm cleared,
Host 0 fxp0: Ethernet Link Down
Aug 14 23:40:09  JuniperM10i-HMNDLAMA alarmd[2998]: Alarm cleared: RE
color=RED, class=CHASSIS, reason=Host 0 fxp0: Ethernet Link Down
Aug 14 23:40:09  JuniperM10i-HMNDLAMA craftd[2999]: forwarding display
request to chassisd: type = 4, subtype = 44
Aug 14 23:40:09  JuniperM10i-HMNDLAMA cfeb CM: ALARM SET: (Major) Slot 0:
CFEB not online, the box is not forwarding
Aug 14 23:40:09  JuniperM10i-HMNDLAMA cfeb CM: ALARM SET: (Major) Slot 0:
Host 0 fxp0: Ethernet Link Down
Aug 14 23:40:09  JuniperM10i-HMNDLAMA cfeb CM: ALARM SET: (Major) Slot 1:
Host 1 fxp0: Ethernet Link Down
Aug 14 23:40:10  JuniperM10i-HMNDLAMA chassisd[2997]: CHASSISD_FRU_EVENT:
fpc_m40_recv_restart: restarted FPC 0
Aug 14 23:40:10  JuniperM10i-HMNDLAMA chassisd[2997]: CHASSISD_FRU_EVENT:
fpc_m40_recv_restart: restarted FPC 1
Aug 14 23:40:12  JuniperM10i-HMNDLAMA craftd[2999]:  Major alarm set, Host 0
fxp0: Ethernet Link Down
Aug 14 23:40:12  JuniperM10i-HMNDLAMA alarmd[2998]: Alarm set: RE color=RED,
class=CHASSIS, reason=Host 0 fxp0: Ethernet Link Down
Aug 14 23:40:12  JuniperM10i-HMNDLAMA craftd[2999]: forwarding display
request to chassisd: type = 4, subtype = 43
Aug 14 23:40:12  JuniperM10i-HMNDLAMA cfeb CM: ALARM CLEAR: Slot 0: Host 0
fxp0: Ethernet Link Down
Aug 14 23:40:17  JuniperM10i-HMNDLAMA craftd[2999]: Major alarm cleared,
CFEB not online, the box is not forwarding
Aug 14 23:40:17  JuniperM10i-HMNDLAMA alarmd[2998]: Alarm cleared: CFEB
color=RED, class=CHASSIS, reason=CFEB not online, the box is not forwarding
Aug 14 23:40:17  JuniperM10i-HMNDLAMA craftd[2999]: forwarding display
request to chassisd: type = 4, subtype = 44
Aug 14 23:40:17  JuniperM10i-HMNDLAMA cfeb CM: ALARM SET: (Major) Slot 0:
Host 0 fxp0: Ethernet Link Down
Aug 14 23:40:32  JuniperM10i-HMNDLAMA chassisd[2997]:
CHASSISD_BLOWERS_SPEED: Fans and impellers are now running at normal speed
Aug 14 23:40:33  JuniperM10i-HMNDLAMA chassisd[2997]: CHASSISD_FRU_EVENT:
scb_recv_slot_attach: attached FPC 0
Aug 14 23:40:55  JuniperM10i-HMNDLAMA chassisd[2997]: CHASSISD_FRU_EVENT:
scb_recv_slot_attach: attached FPC 1
Aug 14 23:40:57  JuniperM10i-HMNDLAMA chassisd[2997]: CHASSISD_SNMP_TRAP10:
SNMP trap generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index 1,
jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName PIC: 1x G/E, 1000 BASE-SX @
0/0/*, jnxFruType 11, jnxFruSlot 1, jnxFruOfflineReason 2,
jnxFruLastPowerOff 0, jnxFruLastPowerOn 0)
Aug 14 23:40:57  JuniperM10i-HMNDLAMA chassisd[2997]: CHASSISD_SNMP_TRAP10:
SNMP trap generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index 2,
jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName PIC: 1x G/E, 1000 BASE-SX @
1/0/*, jnxFruType 11, jnxFruSlot 2, jnxFruOfflineReason 2,
jnxFruLastPowerOff 0, jnxFruLastPowerOn 0)
Aug 14 23:40:57  JuniperM10i-HMNDLAMA chassisd[2997]:
CHASSISD_IFDEV_CREATE_NOTICE: create_pics: created interface device for
ge-0/0/0
Aug 14 23:40:58  JuniperM10i-HMNDLAMA chassisd[2997]:
CHASSISD_IFDEV_CREATE_NOTICE: create_pics: created interface device for
ge-1/0/0
Aug 14 23:40:58  JuniperM10i-HMNDLAMA chassisd[2997]: CHASSISD_SNMP_TRAP10:
SNMP trap generated: FRU power on (jnxFruContentsIndex 7, jnxFruL1Index 1,
jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC:  @ 0/*/*, jnxFruType 3,
jnxFruSlot 1, jnxFruOfflineReason 2, jnxFruLastPowerOff 0, jnxFruLastPowerOn
0)

(BGP notifications that peers are responding)


Aug 14 23:42:22  JuniperM10i-HMNDLAMA chassisd[2997]:
CHASSISD_BLOWERS_SPEED_FULL: Fans and impellers being set to full speed
[system warm]
Aug 14 23:43:22  JuniperM10i-HMNDLAMA chassisd[2997]:
CHASSISD_BLOWERS_SPEED: Fans and impellers are now running at normal speed
Aug 14 23:44:02  JuniperM10i-HMNDLAMA chassisd[2997]:
CHASSISD_BLOWERS_SPEED_FULL: Fans and impellers being set to full speed
[system warm]
Aug 14 23:44:37  JuniperM10i-HMNDLAMA chassisd[2997]:
CHASSISD_BLOWERS_SPEED: Fans and impellers are now running at normal speed

More information about the juniper-nsp mailing list