[c-nsp] problem with newly installed vip and pa-fe-tx on 7507

Scott Granados sgranados at jeteye.com
Wed Mar 8 10:33:19 EST 2006 

Hi, last evening, I installed a new vip (I believe 2-50) and pa-fe-tx on
a 7507 running 12.2-21 on an rsp4.  Presently I have one bgp session
attached and this card was going to be the provider facing card for a
second.

 

After installing, the router booted fine, I could do a show int fast
1/0/0 and it appeared fine.

 

Once the bgp session came up the newly installed vip crashed and I've
included the logging output below.  It looks like some sort of memory
issue but I'm not sure if this is bad memory, no memory or what I'm
finding.  Any pointers would be greatly appreciated.

 

Thanks

 

show logging

Syslog logging: enabled (0 messages dropped, 1 messages rate-limited, 0
flushes, 0 overruns)

    Console logging: level debugging, 47 messages logged

    Monitor logging: level debugging, 0 messages logged

    Buffer logging: level debugging, 48 messages logged

    Logging Exception size (4096 bytes)

    Trap logging: level informational, 93 message lines logged

        Logging to 71.5.15.4, 93 message lines logged

 --More--         

Log Buffer (65536 bytes):

 

00:00:59: %SYS-5-CONFIG_I: Configured from memory by console

00:01:01: %SYS-5-RESTART: System restarted --

Cisco Internetwork Operating System Software 

IOS (tm) RSP Software (RSP-JSV-M), Version 12.2(21), RELEASE SOFTWARE
(fc3)

Copyright (c) 1986-2003 by cisco Systems, Inc.

Compiled Tue 02-Dec-03 12:16 by kellmill

00:01:01: %SNMP-5-COLDSTART: SNMP agent on host core1.sfo is undergoing
a cold start

00:01:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface
FastEthernet0/0/0, changed state to up

00:01:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface
GigabitEthernet4/0/0, changed state to up

00:01:01: %SYS-6-BOOTTIME: Time taken to reboot after reload =
1141798166 seconds

00:01:02: %LINK-5-CHANGED: Interface FastEthernet1/0/0, changed state to
administratively down

00:01:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface
FastEthernet1/0/0, changed state to down

00:01:06: %BGP-5-ADJCHANGE: neighbor 67.109.27.57 Up 

00:01:58: %FIB-3-FIBDISABLE: Fatal error, slot 1: no memory

 --More--         00:02:00: %IPC-5-SLAVELOG:
VIP-SLOT1:

 00:01:09: %SYS-2-MALLOCFAIL: Memory allocation of 65556 bytes failed
from 0x600DF9CC, alignment 16 

Pool: Processor  Free: 12816  Cause: Not enough free memory 

Alternate Pool: None  Free: 0  Cause: No Alternate pool 

 

-Process= "CEF IPC Background", ipl= 0, pid= 28

-Traceback= 600E3EE0 600E56FC 600DF9D4 6031ED38 60300E24 60305DD8
6030DCA4 6031AEE4 60315078 60315304 60315680 60315D18

00:04:09: %FIB-3-FIBDISABLE: Fatal error, slot 0: no memory

00:04:12: %IPC-5-SLAVELOG: VIP-SLOT0:

 00:03:20: %SYS-2-MALLOCFAIL: Memory allocation of 65556 bytes failed
from 0x600DF9CC, alignment 16 

Pool: Processor  Free: 43456  Cause: Not enough free memory 

Alternate Pool: None  Free: 0  Cause: No Alternate pool 

 

-Process= "CEF IPC Background", ipl= 0, pid= 28

-Traceback= 600E3EE0 600E56FC 600DF9D4 6031ED38 60300E24 60305DD8
6030DCA4 6031AEE4 60315078 60315304 60315680 60315D18

04:36:27: %CBUS-3-CMDTIMEOUT: Cmd timed out, CCB 0x5800FF30, slot 1, cmd
code 2

-Traceback= 603BC170 603BC61C 603B48B4 603AD540 603ADE68

04:36:38: %CBUS-3-CCBPTIMEOUT: CCB handover timed out, CCB 0x5800FF30,
slot 1

-Traceback= 603BCC2C 603BA2E8 603BA924 603ADFB0

 --More--         04:36:38: %SYS-3-CPUHOG: Task ran
for 7004 msec (990/0), process = OIR Handler, PC = 603ADDDC.

-Traceback= 603ADDE4

04:36:49: %CBUS-3-CCBPTIMEOUT: CCB handover timed out, CCB 0x5800FF30,
slot 1

-Traceback= 603BCC2C 603BA2E8 603BA924 603ADFB0

04:36:49: %SYS-3-CPUHOG: Task ran for 10700 msec (3/0), process = OIR
Handler, PC = 603ADDDC.

-Traceback= 603ADDE4

04:37:05: %OIR-3-LONGSTALL: Long bus stall (5000 ms), check for
improperly seated cards

04:37:20: %CBUS-3-CCBPTIMEOUT: CCB handover timed out, CCB 0x5800FF30,
slot 1

-Traceback= 603BCC2C 603BA2E8 603BA924 603ADFB0

04:37:20: %SYS-3-CPUHOG: Task ran for 11532 msec (85/83), process = OIR
Handler, PC = 603ADDDC.

-Traceback= 603ADDE4

04:39:42: %DBUS-3-SW_NOTRDY: DBUS software not ready after
dbus_slot_enable(), elapsed 13044, status 0x40

-Traceback= 603A7108 603A7B88 603BA850 603ADFB0

04:39:53: %CBUS-3-CCBPTIMEOUT: CCB handover timed out, CCB 0x5800FF30,
slot 1

-Traceback= 603BCC2C 603BA2E8 603BA924 603ADFB0

04:39:53: %SYS-3-CPUHOG: Task ran for 15896 msec (2127/163), process =
OIR Handler, PC = 603ADDDC.

-Traceback= 603ADDE4

 --More--         04:40:06: %DBUS-3-SW_NOTRDY: DBUS
software not ready after dbus_slot_enable(), elapsed 13036, status 0x40

-Traceback= 603A7108 603A7B88 603BA850 603ADFB0

04:40:17: %CBUS-3-CCBPTIMEOUT: CCB handover timed out, CCB 0x5800FF30,
slot 1

-Traceback= 603BCC2C 603BA2E8 603BA924 603ADFB0

04:40:17: %SYS-3-CPUHOG: Task ran for 22556 msec (298/0), process = OIR
Handler, PC = 603ADDDC.

-Traceback= 603ADDE4

04:40:33: %CBUS-3-CCBPTIMEOUT: CCB handover timed out, CCB 0x5800FF30,
slot 1

-Traceback= 603BCC2C 603BA2E8 603BA924 603ADFB0

04:40:33: %SYS-3-CPUHOG: Task ran for 16656 msec (3/1), process = OIR
Handler, PC = 603ADDDC.

-Traceback= 603ADDE4

04:40:40: %BGP-5-ADJCHANGE: neighbor 67.109.27.57 Down BGP Notification
sent

04:40:40: %BGP-3-NOTIFICATION: sent to neighbor 67.109.27.57 4/0 (hold
time expired) 0 bytes 

04:40:57: %BGP-5-ADJCHANGE: neighbor 67.109.27.57 Up 

04:44:02: %CBUS-3-CCBPTIMEOUT: CCB handover timed out, CCB 0x5800FF30,
slot 1

-Traceback= 603BCC2C 603BA2E8 603BA924 603ADFB0

04:44:02: %SYS-3-CPUHOG: Task ran for 9808 msec (238/13), process = OIR
Handler, PC = 603ADDDC.

-Traceback= 603ADDE4

04:44:15: %CBUS-3-CCBPTIMEOUT: CCB handover timed out, CCB 0x5800FF30,
slot 1

 --More--         -Traceback= 603BCC2C 603BA2E8
603BA924 603ADFB0

04:44:15: %SYS-3-CPUHOG: Task ran for 10700 msec (9/7), process = OIR
Handler, PC = 603ADDDC.

-Traceback= 603ADDE4

04:44:35: %OIR-3-LONGSTALL: Long bus stall (5000 ms), check for
improperly seated cards

04:44:51: %CBUS-3-CCBPTIMEOUT: CCB handover timed out, CCB 0x5800FF30,
slot 1

-Traceback= 603BCC2C 603BA2E8 603BA924 603ADFB0

04:44:51: %SYS-3-CPUHOG: Task ran for 10764 msec (6/4), process = OIR
Handler, PC = 603ADDDC.

-Traceback= 603ADDE4

08:01:20: %CBUS-3-CCBPTIMEOUT: CCB handover timed out, CCB 0x5800FF30,
slot 1

-Traceback= 603BCC2C 603BA2E8 603BA924 603ADFB0

08:01:20: %SYS-3-CPUHOG: Task ran for 17292 msec (2/0), process = OIR
Handler, PC = 603ADDDC.

-Traceback= 603ADDE4

08:39:27: %OIR-3-LONGSTALL: Long bus stall (5000 ms), check for
improperly seated cards

08:39:43: %CBUS-3-CCBPTIMEOUT: CCB handover timed out, CCB 0x5800FF30,
slot 1

-Traceback= 603BCC2C 603BA2E8 603BA924 603ADFB0

08:39:43: %SYS-3-CPUHOG: Task ran for 12684 msec (41/39), process = OIR
Handler, PC = 603ADDDC.

-Traceback= 603ADDE4

 --More--         08:40:22: %CBUS-3-CCBPTIMEOUT: CCB
handover timed out, CCB 0x5800FF30, slot 1

-Traceback= 603BCC2C 603BA2E8 603BA924 603ADFB0

08:40:22: %SYS-3-CPUHOG: Task ran for 18908 msec (148/0), process = OIR
Handler, PC = 603ADDDC.

-Traceback= 603ADDE4

08:40:33: %CBUS-3-CCBPTIMEOUT: CCB handover timed out, CCB 0x5800FF30,
slot 1

-Traceback= 603BCC2C 603BA2E8 603BA924 603ADFB0

08:40:33: %SYS-3-CPUHOG: Task ran for 10704 msec (2/0), process = OIR
Handler, PC = 603ADDDC.

-Traceback= 603ADDE4

core1.sfo#\ 

More information about the cisco-nsp mailing list