[j-nsp] sfm1 CCHIP: PIO error reply queue overflowed
peter brenner
peter5192 at hotmail.com
Wed Jan 10 20:24:23 EST 2007
Hello,
today I found the following in the logfiles.
The router seemed to have recovered completely but I'd like to know how to
prevent further outages like that.
The router is a M160 running JunOS 7.6R1.10, with 4 sfms (710-001228) and
the fpc0 is a fpc1 (710-001255).
The replaced IP a.b.c.d is the BGP neighbor on ge-0/0/0.
Jan 10 20:59:26 ham-cr2-re1 sfm1 CCHIP: PIO error reply queue overflowed
Jan 10 20:59:26 ham-cr2-re1 sfm1 A1: PIO read from 0x0021 failed: timeout
Jan 10 20:59:26 ham-cr2-re1 sfm1 PFE: Could not check or clear A1-chip
error status: A1: Cannot determine error state
Jan 10 20:59:26 ham-cr2-re1 sfm1 A1: PIO write to 0x0017 failed: timeout
Jan 10 20:59:26 ham-cr2-re1 sfm1 A2: PIO write to 0x0017 failed: timeout
Jan 10 20:59:28 ham-cr2-re1 sfm0 CCHIP: PIO error reply queue overflowed
Jan 10 20:59:28 ham-cr2-re1 sfm0 CCHIP: PIO error retry from Bm
Jan 10 20:59:28 ham-cr2-re1 sfm0 A1: PIO read from 0x0021 failed: timeout
Jan 10 20:59:28 ham-cr2-re1 sfm0 PFE: Could not check or clear A1-chip
error status: A1: Cannot determine error state
Jan 10 20:59:28 ham-cr2-re1 sfm0 A1: PIO write to 0x0017 failed: timeout
Jan 10 20:59:28 ham-cr2-re1 sfm0 A2: PIO write to 0x0017 failed: timeout
Jan 10 20:59:30 ham-cr2-re1 sfm2 CCHIP: PIO error reply queue overflowed
Jan 10 20:59:30 ham-cr2-re1 sfm2 A1: PIO read from 0x0021 failed: timeout
Jan 10 20:59:30 ham-cr2-re1 sfm2 PFE: Could not check or clear A1-chip
error status: A1: Cannot determine error state
Jan 10 20:59:30 ham-cr2-re1 sfm2 A1: PIO write to 0x0017 failed: timeout
Jan 10 20:59:30 ham-cr2-re1 sfm2 A2: PIO write to 0x0017 failed: timeout
Jan 10 20:59:32 ham-cr2-re1 /kernel: rdp keepalive expired, connection
dropped - src 1:1021 dest 16:35841
Jan 10 20:59:32 ham-cr2-re1 /kernel: rdp keepalive expired, connection
dropped - src 1:1020 dest 16:35840
Jan 10 20:59:32 ham-cr2-re1 /kernel: pfe_send_failed(index 0, type 4),
err=32
Jan 10 20:59:32 ham-cr2-re1 chassisd[2968]: CHASSISD_IFDEV_DETACH_FPC:
ifdev_detach(0)
Jan 10 20:59:32 ham-cr2-re1 mib2d[24656]: SNMP_TRAP_LINK_DOWN: ifIndex 144,
ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/0
Jan 10 20:59:32 ham-cr2-re1 mib2d[24656]: SNMP_TRAP_LINK_DOWN: ifIndex 71,
ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/2/0
Jan 10 20:59:32 ham-cr2-re1 chassisd[2968]:
CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 0
Jan 10 20:59:32 ham-cr2-re1 mib2d[24656]: SNMP_TRAP_LINK_DOWN: ifIndex 104,
ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/1/0
Jan 10 20:59:32 ham-cr2-re1 mib2d[24656]: SNMP_TRAP_LINK_DOWN: ifIndex 269,
ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/3/0
Jan 10 20:59:32 ham-cr2-re1 chassisd[2968]: CHASSISD_SNMP_TRAP10: SNMP trap
generated: Fru Offline (jnxFruContentsIndex 7, jnxFruL1Index 1,
jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: FPC Type 1 @ 0/*/*,
jnxFruType 3, jnxFruSlot 1, jnxFruOfflineReason 2, jnxFruLastPowerOff
127583, jnxFruLastPowerOn 130026)
Jan 10 20:59:32 ham-cr2-re1 sfm1 Slot 1: B-chip error flags present during
initialization; code 0x00020000
Jan 10 20:59:32 ham-cr2-re1 sfm2 Slot 1: B-chip error flags present during
initialization; code 0x00020000
Jan 10 20:59:32 ham-cr2-re1 sfm0 Slot 1: B-chip error flags present during
initialization; code 0x00020000
Jan 10 20:59:32 ham-cr2-re1 sfm0 Slot 6: B-chip error flags present during
initialization; code 0x00020000
Jan 10 20:59:32 ham-cr2-re1 sfm1 Slot 6: B-chip error flags present during
initialization; code 0x00020000
Jan 10 20:59:32 ham-cr2-re1 sfm2 Slot 6: B-chip error flags present during
initialization; code 0x00020000
Jan 10 20:59:32 ham-cr2-re1 /kernel: ae_link_op: link ge-0/1/0.2 (lidx=0)
detached from bundle ae0.2
Jan 10 20:59:32 ham-cr2-re1 /kernel: ge-0/1/0.2 leaves ae0.2
[...]
Jan 10 20:59:33 ham-cr2-re1 /kernel: ae_link_op: link ge-0/1/0.76 (lidx=0)
detached from bundle ae0.76
Jan 10 20:59:33 ham-cr2-re1 /kernel: ge-0/1/0.76 leaves ae0.76
Jan 10 20:59:33 ham-cr2-re1 /kernel: cos_msg_sched_policy_def: ifd not
found (idx 0x8d)
Jan 10 20:59:33 ham-cr2-re1 /kernel: cos_msg_sched_policy_def: ifd not
found (idx 0x8d)
Jan 10 20:59:33 ham-cr2-re1 /kernel: cos_msg_sched_policy_def: ifd not
found (idx 0x90)
Jan 10 20:59:33 ham-cr2-re1 /kernel: cos_msg_sched_policy_def: ifd not
found (idx 0x90)
Jan 10 20:59:34 ham-cr2-re1 /kernel: rdp keepalive expired, connection
dropped - src 1:1021 dest 11:14336
Jan 10 20:59:34 ham-cr2-re1 chassisd[2968]:
CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for SFM 3
Jan 10 20:59:34 ham-cr2-re1 chassisd[2968]: CHASSISD_SNMP_TRAP10: SNMP trap
generated: Fru Offline (jnxFruContentsIndex 6, jnxFruL1Index 4,
jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName SFM 3 , jnxFruType 4,
jnxFruSlot 4, jnxFruOfflineReason 2, jnxFruLastPowerOff 519141911,
jnxFruLastPowerOn 519261126)
Jan 10 20:59:34 ham-cr2-re1 craftd[2970]: Minor alarm set, Not all SFMs
are online
Jan 10 20:59:34 ham-cr2-re1 alarmd[2969]: Alarm set: SFM color=YELLOW,
class=CHASSIS, reason=Not all SFMs are online
Jan 10 20:59:34 ham-cr2-re1 /kernel: rdp keepalive expired, connection
dropped - src 1:1020 dest 11:14337
Jan 10 20:59:34 ham-cr2-re1 /kernel: pfe_send_failed(index 3, type 2),
err=32
Jan 10 20:59:35 ham-cr2-re1 chassisd[2968]: CHASSISD_IPC_ANNOUNCE_TIMEOUT:
fpc_m160_announce_offline_timeout: no ack received from SFMs for FPC 0 state
change (0x7, acks 0x7)
Jan 10 20:59:42 ham-cr2-re1 rpd[24657]: KRT ADD for 216.255.99.0/24 => {
ifl 213 addr a.b.c.d } failed, error "ENOENT -- Item not found".
Jan 10 20:59:42 ham-cr2-re1 rpd[24657]: KRT ADD for 141.149.72.0/23 => {
ifl 213 addr a.b.c.d } failed, error "ENOENT -- Item not found".
Jan 10 20:59:42 ham-cr2-re1 rpd[24657]: KRT ADD for 63.106.74.0/24 => { ifl
213 addr a.b.c.d } failed, error "ENOENT -- Item not found".
[...]
Jan 10 20:59:43 ffm1.cr1-re1 mib2d[24656]: MIB2D_RTSLIB_READ_FAILURE:
check_rtsock_rc: failed in reading lag_child, remote stats: 0 (Operation
timed out)
Jan 10 20:59:47 ham-cr2-re1 /kernel: pfe_listener_disconnect: conn dropped:
listener idx=2, tnpaddr=0x10, reason: reconnect timeout
Jan 10 20:59:49 ham-cr2-re1 /kernel: pfe_listener_disconnect: conn dropped:
listener idx=5, tnpaddr=0xb, reason: reconnect timeout
Jan 10 21:00:37 ham-cr2-re1 craftd[2970]: Minor alarm cleared, Not all SFMs
are online
Jan 10 21:00:37 ham-cr2-re1 alarmd[2969]: Alarm cleared: SFM color=YELLOW,
class=CHASSIS, reason=Not all SFMs are online
Jan 10 21:00:37 ham-cr2-re1 chassisd[2968]: CHASSISD_SNMP_TRAP7: SNMP trap
generated: Fru Online (jnxFruContentsIndex 6, jnxFruL1Index 4, jnxFruL2Index
0, jnxFruL3Index 0, jnxFruName SFM 3 , jnxFruType 4, jnxFruSlot 4)
Jan 10 21:00:44 ham-cr2-re1 sfm3 PFEMAN: Couldn't write "NHDB" msg to
master pipe
Jan 10 21:02:45 ham-cr2-re1 chassisd[2968]: CHASSISD_SNMP_TRAP10: SNMP trap
generated: FRU power on (jnxFruContentsIndex 7, jnxFruL1Index 1,
jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: FPC Type 1 @ 0/*/*,
jnxFruType 3, jnxFruSlot 1, jnxFruOfflineReason 2, jnxFruLastPowerOff
127583, jnxFruLastPowerOn 628957365)
Jan 10 21:03:39 ham-cr2-re1 chassisd[2968]: CHASSISD_SNMP_TRAP7: SNMP trap
generated: Fru Online (jnxFruContentsIndex 7, jnxFruL1Index 1, jnxFruL2Index
0, jnxFruL3Index 0, jnxFruName FPC: FPC Type 1 @ 0/*/*, jnxFruType 3,
jnxFruSlot 1)
Jan 10 21:03:39 ham-cr2-re1 chassisd[2968]: CHASSISD_SNMP_TRAP10: SNMP trap
generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index 1,
jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName PIC: 1x G/E, 1000 BASE-LX @
0/0/*, jnxFruType 11, jnxFruSlot 1, jnxFruOfflineReason 2,
jnxFruLastPowerOff 0, jnxFruLastPowerOn 628962801)
Jan 10 21:03:39 ham-cr2-re1 chassisd[2968]: CHASSISD_SNMP_TRAP10: SNMP trap
generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index 1,
jnxFruL2Index 2, jnxFruL3Index 0, jnxFruName PIC: 1x G/E, 1000 BASE-SX @
0/1/*, jnxFruType 11, jnxFruSlot 1, jnxFruOfflineReason 2,
jnxFruLastPowerOff 31637617, jnxFruLastPowerOn 628962812)
Jan 10 21:03:39 ham-cr2-re1 chassisd[2968]: CHASSISD_SNMP_TRAP10: SNMP trap
generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index 1,
jnxFruL2Index 3, jnxFruL3Index 0, jnxFruName PIC: 1x G/E, 1000 BASE-SX @
0/2/*, jnxFruType 11, jnxFruSlot 1, jnxFruOfflineReason 2,
jnxFruLastPowerOff 0, jnxFruLastPowerOn 628962825)
Jan 10 21:03:39 ham-cr2-re1 chassisd[2968]: CHASSISD_SNMP_TRAP10: SNMP trap
generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index 1,
jnxFruL2Index 4, jnxFruL3Index 0, jnxFruName PIC: 1x G/E, 1000 BASE-SX @
0/3/*, jnxFruType 11, jnxFruSlot 1, jnxFruOfflineReason 2,
jnxFruLastPowerOff 517771326, jnxFruLastPowerOn 628962837)
Jan 10 21:03:51 ham-cr2-re1 chassisd[2968]: CHASSISD_IFDEV_CREATE_NOTICE:
create_pics: created interface device for ge-0/0/0
Jan 10 21:03:51 ham-cr2-re1 chassisd[2968]: CHASSISD_IFDEV_CREATE_NOTICE:
create_pics: created interface device for ge-0/1/0
Jan 10 21:03:51 ham-cr2-re1 chassisd[2968]: CHASSISD_IFDEV_CREATE_NOTICE:
create_pics: created interface device for ge-0/2/0
Jan 10 21:03:51 ham-cr2-re1 chassisd[2968]: CHASSISD_IFDEV_CREATE_NOTICE:
create_pics: created interface device for ge-0/3/0
Best Regards, Peter
_________________________________________________________________
Don't just search. Find. Check out the new MSN Search!
http://search.msn.click-url.com/go/onm00200636ave/direct/01/
More information about the juniper-nsp
mailing list