[j-nsp] M7i routing crash

Mon Apr 2 05:48:58 EDT 2007

Hello Guys,

Just trying to see if anyone has a better understanding about those logs
than me. Basically one of my major routers "M7i" failed last week ,
seems that the Forward Engine stops and brought all physical interfaces
down on the router, causing full loss of traffic, I have Junos 7.4 R2.6
running on all my M7i routers across the network. Below I have a few
logs from the crash time. Basically after a minute the CFEB did reboot
by itself and got the routing capability back. The router it self didn't
reboot like I saw system uptime 200 days, only the CFEB did a reboot.

In case anyone wants to share any thought I will appreciate.

Thanks

Rommel

Mar 28 09:43:11  Guinness /kernel: rdp keepalive expired, connection
dropped - src 1:1021 dest 2:11264

Mar 28 09:43:11  Guinness craftd[2684]:  Major alarm set, CFEB not
online, the box is not forwarding

Mar 28 09:43:11  Guinness chassisd[2682]: CHASSISD_SHUTDOWN_NOTICE:
Shutdown reason: CFEB connection lost

Mar 28 09:43:11  Guinness alarmd[2683]: Alarm set: CFEB color=RED,
class=CHASSIS, reason=CFEB not online, the box is not forwarding

Mar 28 09:43:11  Guinness craftd[2684]: forwarding display request to
chassisd: type = 4, subtype = 43

Mar 28 09:43:11  Guinness chassisd[2682]: CHASSISD_IFDEV_DETACH_FPC:
ifdev_detach(0)

Mar 28 09:43:11  Guinness /kernel: aggr_link_op: link so-0/0/0.0
(lidx=0) detached from bundle as0.0

Mar 28 09:43:11  Guinness syslogd: sendto: Network is down

Mar 28 09:43:11  Guinness /kernel: so-0/0/0.0 leaves as0.0

Mar 28 09:43:11  Guinness /kernel: cos_msg_sched_policy_def: ifd not
found (idx 0x88)

Mar 28 09:43:11  Guinness /kernel: bundle as0: bundle IFD state changed
to DOWN

Mar 28 09:43:11  Guinness /kernel: aggr_link_op: link so-0/0/1.0
(lidx=1) detached from bundle as0.0

Mar 28 09:43:11  Guinness /kernel: bundle as0.0: bundle IFL state
changed to DOWN

Mar 28 09:43:11  Guinness /kernel: so-0/0/1.0 leaves as0.0

Mar 28 09:43:11  Guinness /kernel: cos_msg_sched_policy_def: ifd not
found (idx 0x89)

Mar 28 09:43:11  Guinness snmpd[2687]: SNMPD_SEND_FAILURE:
trap_io_send_trap_now: send to (87.192.235.219) failure: Network is down

Mar 28 09:43:11  Guinness chassisd[2682]: CHASSISD_IFDEV_DETACH_FPC:
ifdev_detach(1)

Mar 28 09:43:11  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
87.192.235.219 to destination queue, 1 traps in queue

Mar 28 09:43:11  Guinness snmpd[2687]: SNMPD_SEND_FAILURE:
trap_io_send_trap_now: send to (62.231.52.94) failure: Network is down

Mar 28 09:43:11  Guinness /kernel: bundle as1: bundle IFD state changed
to DOWN

Mar 28 09:43:11  Guinness /kernel: aggr_link_op: link so-0/0/2.0
(lidx=0) detached from bundle as1.0

Mar 28 09:43:11  Guinness /kernel: bundle as1.0: bundle IFL state
changed to DOWN

Mar 28 09:43:11  Guinness /kernel: so-0/0/2.0 leaves as1.0

Mar 28 09:43:11  Guinness /kernel: cos_msg_sched_policy_def: ifd not
found (idx 0x8a)

Mar 28 09:43:11  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
62.231.52.94 to destination queue, 1 traps in queue

Mar 28 09:43:11  Guinness snmpd[2687]: SNMPD_SEND_FAILURE:
trap_io_send_trap_now: send to (10.10.1.43) failure: Network is down

Mar 28 09:43:12  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
10.10.1.43 to destination queue, 1 traps in queue

Mar 28 09:43:12  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
87.192.235.219 to destination queue, 2 traps in queue

Mar 28 09:43:12  Guinness /kernel: aggr_link_op: link so-0/0/3.0
(lidx=1) detached from bundle as1.0

Mar 28 09:43:12  Guinness /kernel: so-0/0/3.0 leaves as1.0

Mar 28 09:46:38  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
10.10.1.43 to destination queue, 266 traps in queue

Mar 28 09:46:38  Guinness /kernel: fxp1: media DOWN 100Mb / full-duplex

Mar 28 09:46:39  Guinness /kernel: fxp1: link UP 100Mb / full-duplex

Mar 28 09:46:39  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
87.192.235.219 to destination queue, 267 traps in queue

Mar 28 09:46:39  Guinness mib2d[2688]: SNMP_TRAP_LINK_UP: ifIndex 2,
ifAdminStatus up(1), ifOperStatus up(1), ifName fxp1

Mar 28 09:46:39  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
62.231.52.94 to destination queue, 267 traps in queue

Mar 28 09:46:39  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
10.10.1.43 to destination queue, 267 traps in queue

Mar 28 09:46:39  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
87.192.235.219 to destination queue, 268 traps in queue

Mar 28 09:46:39  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
62.231.52.94 to destination queue, 268 traps in queue

Mar 28 09:46:39  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
10.10.1.43 to destination queue, 268 traps in queue

Mar 28 09:46:39  Guinness cfeb COSMAN PFE RED prof_id_start 1000,
prof_id_end 3048

Mar 28 09:46:39  Guinness cfeb CSBR: Reset reason (0x8): RE Initiated
Reset

Mar 28 09:46:39  Guinness cfeb SNTP: Daemon created

Mar 28 09:46:39  Guinness cfeb Version 7.4R2.6 by builder on 2006-01-20
14:09:28 UTC

Mar 28 09:46:39  Guinness cfeb On-board NVRAM contains diagnostic
information.

Mar 28 09:46:39  Guinness cfeb CCHIP: Detected CF 2.0

Mar 28 09:46:39  Guinness cfeb SNTPD: Initial time of day set.

Mar 28 09:46:42  Guinness snmpd[2687]: SNMPD_SEND_FAILURE:
trap_io_send_trap_now: send to (10.10.1.43) failure: No route to host

Mar 28 09:46:42  Guinness snmpd[2687]: SNMPD_SEND_FAILURE:
trap_io_send_trap_now: send to (62.231.52.94) failure: No route to host

Mar 28 09:46:42  Guinness snmpd[2687]: SNMPD_SEND_FAILURE:
trap_io_send_trap_now: send to (87.192.235.219) failure: No route to
host

Mar 28 09:46:47  Guinness cfeb CM: Established connection to routing
engine

Mar 28 09:46:47  Guinness /kernel: pfe_peer_alloc: type 1, index 0, old
state Closed new state Opening mastership 1

Mar 28 09:46:47  Guinness /kernel: pfe_peer_set_timeout: set timeout
0xc03990a9

Mar 28 09:46:47  Guinness /kernel: pfe_listener_connect: conn
established: listener idx=0, tnpaddr=0x2

Mar 28 09:46:47  Guinness chassisd[2682]: CHASSISD_SNMP_TRAP10: SNMP
trap generated: FRU power on (jnxFruContentsIndex 6, jnxFruL1Index 1,
jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName CFEB, jnxFruType 4,
jnxFruSlot 1, jnxFruOfflineReason 2, jnxFruLastPowerOff 60104671,
jnxFruLastPowerOn 60104671)

Mar 28 09:46:47  Guinness chassisd[2682]: CHASSISD_TIMER_CLR_ERR:
timer_clear_fru: unable to clear timeout

Mar 28 09:46:47  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
87.192.235.219 to destination queue, 269 traps in queue

Mar 28 09:46:47  Guinness chassisd[2682]:
CHASSISD_IFDEV_DETACH_ALL_PSEUDO: ifdev_detach(pseudo devices: all)

Mar 28 09:46:47  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
62.231.52.94 to destination queue, 269 traps in queue

Mar 28 09:46:47  Guinness cfeb PFEMAN: Session manager active

Mar 28 09:46:47  Guinness cfeb PFEMAN: Established connection to Master

Mar 28 09:46:47  Guinness cfeb CM: ALARM SET: (Major) Slot 0: CFEB not
online, the box is not forwarding

Mar 28 09:46:47  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
10.10.1.43 to destination queue, 269 traps in queue

Mar 28 09:46:47  Guinness chassisd[2682]: CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as0

Mar 28 09:46:47  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
87.192.235.219 to destination queue, 270 traps in queue

Mar 28 09:46:47  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
62.231.52.94 to destination queue, 270 traps in queue

Mar 28 09:46:47  Guinness chassisd[2682]: CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as1

Mar 28 09:46:47  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
10.10.1.43 to destination queue, 270 traps in queue

Mar 28 09:46:47  Guinness chassisd[2682]: CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as2

Mar 28 09:46:47  Guinness chassisd[2682]: CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as3

Mar 28 09:46:47  Guinness chassisd[2682]: CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as4

Mar 28 09:46:47  Guinness cfeb CM: ALARM SET: (Major) Slot 0: Host 0
fxp0: Ethernet Link Down

Mar 28 09:46:47  Guinness chassisd[2682]: CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as5

Mar 28 09:46:47  Guinness chassisd[2682]: CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as6

Mar 28 09:46:47  Guinness chassisd[2682]: CHASSISD_IFDEV_CREATE_NOTICE:
create_pseudos: created pseudo interface device for as7

Mar 28 09:46:49  Guinness chassisd[2682]: CHASSISD_FRU_EVENT:
fpc_m40_recv_restart: restarted FPC 0

Mar 28 09:46:49  Guinness chassisd[2682]: CHASSISD_FRU_EVENT:
fpc_m40_recv_restart: restarted FPC 1

Mar 28 09:46:57  Guinness craftd[2684]: Major alarm cleared, CFEB not
online, the box is not forwarding

Mar 28 09:46:57  Guinness alarmd[2683]: Alarm cleared: CFEB color=RED,
class=CHASSIS, reason=CFEB not online, the box is not forwarding

Mar 28 09:46:57  Guinness craftd[2684]: forwarding display request to
chassisd: type = 4, subtype = 44

Mar 28 09:46:59  Guinness cfeb CM: ALARM CLEAR: Slot 0: CFEB not online,
the box is not forwarding

Got back up after reset the Cfeb

Mar 28 09:47:36  Guinness snmpd[2687]: SNMPD_TRAP_QUEUED: Adding trap to
10.10.1.43 to destination queue, 342 traps in queue

Mar 28 09:47:36  Guinness /kernel: bundle as0: bundle IFD state changed
to UP

Mar 28 09:47:36  Guinness /kernel: bundle as1: bundle IFD state changed
to UP

Note:
This message is for the named person's use only.  It may contain confidential, proprietary or legally privileged information.  No confidentiality or privilege is waived or lost by any mistransmission.  If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of it and notify the sender.  You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Irish Broadband and any of its subsidiaries each reserve the right to monitor all e-mail communications through its networks. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state them to be the views of any such entity.

Thank You. 