[c-nsp] memory allocation bug again?
Jeff Chan
cisco-nsp at jeffchan.com
Mon Jul 3 09:36:11 EDT 2006
More complete logs are:
Jul 3 06:21:12 mdsnwi-router-01 7476: Jul 3 06:20:31: %LINEPROTO-5-UPDOWN: Line protocol on Interface Multilink1, changed state to down
Jul 3 06:21:12 mdsnwi-router-01 7477: Jul 3 06:20:32: %LINK-3-UPDOWN: Interface Multilink4, changed state to down
Jul 3 06:21:12 mdsnwi-router-01 7478: Jul 3 06:20:33: %LINEPROTO-5-UPDOWN: Line protocol on Interface Multilink4, changed state to down
Jul 3 06:21:12 mdsnwi-router-01 7479: Jul 3 06:20:33: %LINK-3-UPDOWN: Interface Multilink5, changed state to down
Jul 3 06:21:12 mdsnwi-router-01 7480: Jul 3 06:20:35: %LINEPROTO-5-UPDOWN: Line protocol on Interface Multilink5, changed state to down
Jul 3 06:21:12 mdsnwi-router-01 7481: Jul 3 06:20:35: %LINK-3-UPDOWN: Interface Multilink2, changed state to down
Jul 3 06:21:12 mdsnwi-router-01 7482: Jul 3 06:20:35: %FIB-3-NOMEM: Malloc Failure, disabling DCEF
Jul 3 06:21:12 mdsnwi-router-01 7483: -Traceback= 415E75DC 402DB9FC 407974D8 4076EFE0 40471AFC 40471AE0
Jul 3 06:21:12 mdsnwi-router-01 7484: Jul 3 06:20:36: %LINEPROTO-5-UPDOWN: Line protocol on Interface Multilink2, changed state to down
Jul 3 06:21:12 mdsnwi-router-01 7485: Jul 3 06:20:54: %DBUS-3-SW_NOTRDY: DBUS software not ready for slot 10 after dbus_slot_enable(), elapsed 13056, status 0x0
Jul 3 06:21:12 mdsnwi-router-01 7486: -Traceback= 405456D0 405465F0 4055B568 4054D89C
Jul 3 06:21:12 mdsnwi-router-01 7487: Jul 3 06:21:04: %CBUS-3-CCBPTIMEOUT: CCB handover timed out, CCB 0xF800FFB0, slot 10
Jul 3 06:21:12 mdsnwi-router-01 7488: -Traceback= 4055DAE8 4055B084 4055B63C 4054D89C
Jul 3 06:21:12 mdsnwi-router-01 7489: Jul 3 06:21:04: %CBUS-3-CCBCMDFAIL1: Controller 10, cmd (61 0x00000008) failed (0x8010)
Jul 3 06:21:12 mdsnwi-router-01 7490: Jul 3 06:21:04: %CBUS-3-CCBCMDFAIL1: Controller 10, cmd (36 0x00000000) failed (0x8010)
Jul 3 06:21:12 mdsnwi-router-01 7491: Jul 3 06:21:04: %CBUS-3-ADDRFILTR: Interface FastEthernet10/0/0, address filter write command failed, code 0x8010
Jul 3 06:21:12 mdsnwi-router-01 7492: -Traceback= 4055EF9C 4055F804 4055FBAC 405F3214 405F17A8 40560564 40559F04 4055A3A0 4054D8B8
Jul 3 06:21:12 mdsnwi-router-01 7493: Jul 3 06:21:04: %CBUS-3-CCBCMDFAIL1: Controller 10, cmd (36 0x0000FFFF) failed (0x8010)
Jul 3 06:21:12 mdsnwi-router-01 7494: Jul 3 06:21:04: %CBUS-3-CCBCMDFAIL1: Controller 10, cmd (36 0x00000000) failed (0x8010)
Jul 3 06:21:12 mdsnwi-router-01 7495: Jul 3 06:21:04: %CBUS-3-CCBCMDFAIL1: Controller 10, cmd (36 0x0000FFFF) failed (0x8010)
Jul 3 06:21:12 mdsnwi-router-01 7496: Jul 3 06:21:04: %CBUS-3-CCBCMDFAIL1: Controller 10, cmd (61 0x00000008) failed (0x8010)
Jul 3 06:21:12 mdsnwi-router-01 7497: Jul 3 06:21:04: %CBUS-3-CCBCMDFAIL1: Controller 10, cmd (36 0x00000000) failed (0x8010)
Jul 3 06:21:12 mdsnwi-router-01 7498: Jul 3 06:21:04: %CBUS-3-CCBCMDFAIL1: Controller 10, cmd (36 0x0000FFFF) failed (0x8010)
Jul 3 06:21:12 mdsnwi-router-01 7499: Jul 3 06:21:04: %CBUS-3-CCBCMDFAIL1: Controller 10, cmd (36 0x00000000) failed (0x8010)
Jul 3 06:21:12 mdsnwi-router-01 7500: Jul 3 06:21:04: %CBUS-3-CCBCMDFAIL1: Controller 10, cmd (36 0x0000FFFF) failed (0x8010)
Jul 3 06:21:12 mdsnwi-router-01 7501: Jul 3 06:21:07: %SYS-3-CPUHOG: Task ran for 31172 msec (11/6), process = OIR Handler, PC = 4054D6BC.
Jul 3 06:21:12 mdsnwi-router-01 7502: -Traceback= 4054D6C4
Jul 3 06:21:12 mdsnwi-router-01 7503: Jul 3 06:21:07: %LINK-3-UPDOWN: Interface FastEthernet10/0/0, changed state to down
Jul 3 06:21:19 mdsnwi-router-01 7504: Jul 3 06:21:09: %LINEPROTO-5-UPDOWN: Line protocol on Interface Hssi8/1/0, changed state to up
Jul 3 06:21:19 mdsnwi-router-01 7505: Jul 3 06:21:09: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial10/1/1/19:0, changed state to down
Jul 3 06:21:19 mdsnwi-router-01 7506: Jul 3 06:21:09: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet10/0/0, changed state to down
Jul 3 06:21:20 mdsnwi-router-01 7507: Jul 3 06:21:18: %LINEPROTO-5-UPDOWN: Line protocol on Interface Hssi12/0/0, changed state to up
Jul 3 06:21:30 mdsnwi-router-01 7508: Jul 3 06:21:29: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Down Peer closed the session
Jul 3 06:21:39 mdsnwi-router-01 7509: Jul 3 06:21:38: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Up
Jul 3 06:21:48 mdsnwi-router-01 7510: Jul 3 06:21:47: %BGP-5-ADJCHANGE: neighbor 144.228.52.181 Up
Jul 3 06:21:59 mdsnwi-router-01 7511: Jul 3 06:21:58: %BGP-5-ADJCHANGE: neighbor 12.118.243.165 Up
Jul 3 06:24:42 mdsnwi-router-01 7512: Jul 3 06:24:42: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Down BGP Notification sent
Jul 3 06:24:43 mdsnwi-router-01 7513: Jul 3 06:24:42: %BGP-3-NOTIFICATION: sent to neighbor 24.177.132.109 4/0 (hold time expired) 0 bytes
Jul 3 06:25:25 mdsnwi-router-01 7514: Jul 3 06:25:24: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Up
Jul 3 06:28:26 mdsnwi-router-01 7515: Jul 3 06:28:26: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Down BGP Notification sent
Jul 3 06:28:27 mdsnwi-router-01 7516: Jul 3 06:28:26: %BGP-3-NOTIFICATION: sent to neighbor 24.177.132.109 4/0 (hold time expired) 0 bytes
Jul 3 06:29:10 mdsnwi-router-01 7517: Jul 3 06:29:09: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Up
Jul 3 06:32:15 mdsnwi-router-01 7518: Jul 3 06:32:15: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Down BGP Notification sent
Jul 3 06:32:16 mdsnwi-router-01 7519: Jul 3 06:32:15: %BGP-3-NOTIFICATION: sent to neighbor 24.177.132.109 4/0 (hold time expired) 0 bytes
Jul 3 06:33:02 mdsnwi-router-01 7520: Jul 3 06:33:00: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Up
Jul 3 06:36:37 mdsnwi-router-01 7521: Jul 3 06:36:02: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Down BGP Notification sent
Jul 3 06:36:38 mdsnwi-router-01 7522: Jul 3 06:36:02: %BGP-3-NOTIFICATION: sent to neighbor 24.177.132.109 4/0 (hold time expired) 0 bytes
Jul 3 06:36:56 mdsnwi-router-01 7523: Jul 3 06:36:55: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Up
Jul 3 06:39:57 mdsnwi-router-01 7524: Jul 3 06:39:57: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Down BGP Notification sent
Jul 3 06:39:58 mdsnwi-router-01 7525: Jul 3 06:39:57: %BGP-3-NOTIFICATION: sent to neighbor 24.177.132.109 4/0 (hold time expired) 0 bytes
Jul 3 06:40:40 mdsnwi-router-01 7526: Jul 3 06:40:39: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Up
Jul 3 06:43:46 mdsnwi-router-01 7527: Jul 3 06:43:46: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Down BGP Notification sent
Jul 3 06:43:47 mdsnwi-router-01 7528: Jul 3 06:43:46: %BGP-3-NOTIFICATION: sent to neighbor 24.177.132.109 4/0 (hold time expired) 0 bytes
Jul 3 06:44:28 mdsnwi-router-01 7529: Jul 3 06:44:26: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Up
Jul 3 06:47:29 mdsnwi-router-01 7530: Jul 3 06:47:29: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Down BGP Notification sent
Jul 3 06:47:30 mdsnwi-router-01 7531: Jul 3 06:47:29: %BGP-3-NOTIFICATION: sent to neighbor 24.177.132.109 4/0 (hold time expired) 0 bytes
Jul 3 06:48:14 mdsnwi-router-01 7532: Jul 3 06:48:13: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Up
Jul 3 06:48:25 mdsnwi-router-01 7533: Jul 3 06:48:25: %BGP-5-ADJCHANGE: neighbor 66.170.0.3 Down BGP Notification sent
Jul 3 06:48:26 mdsnwi-router-01 7534: Jul 3 06:48:25: %BGP-3-NOTIFICATION: sent to neighbor 66.170.0.3 4/0 (hold time expired) 0 bytes
Jul 3 06:49:04 mdsnwi-router-01 7535: Jul 3 06:49:03: %BGP-5-ADJCHANGE: neighbor 66.170.0.3 Up
Jul 3 06:51:16 mdsnwi-router-01 7536: Jul 3 06:51:16: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Down BGP Notification sent
Jul 3 06:51:17 mdsnwi-router-01 7537: Jul 3 06:51:16: %BGP-3-NOTIFICATION: sent to neighbor 24.177.132.109 4/0 (hold time expired) 0 bytes
Jul 3 06:52:04 mdsnwi-router-01 7538: Jul 3 06:52:03: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Up
Jul 3 06:52:04 mdsnwi-router-01 7539: Jul 3 06:52:03: %BGP-5-ADJCHANGE: neighbor 66.170.0.3 Down BGP Notification sent
Jul 3 06:52:04 mdsnwi-router-01 7540: Jul 3 06:52:03: %BGP-3-NOTIFICATION: sent to neighbor 66.170.0.3 4/0 (hold time expired) 0 bytes
Jul 3 06:52:46 mdsnwi-router-01 7541: Jul 3 06:52:45: %BGP-5-ADJCHANGE: neighbor 66.170.0.3 Up
Jul 3 06:55:05 mdsnwi-router-01 7542: Jul 3 06:55:05: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Down BGP Notification sent
Jul 3 06:55:06 mdsnwi-router-01 7543: Jul 3 06:55:05: %BGP-3-NOTIFICATION: sent to neighbor 24.177.132.109 4/0 (hold time expired) 0 bytes
Jul 3 06:55:46 mdsnwi-router-01 7544: Jul 3 06:55:46: %BGP-5-ADJCHANGE: neighbor 66.170.0.3 Down BGP Notification sent
Jul 3 06:57:06 mdsnwi-router-01 7545: Jul 3 06:55:46: %BGP-3-NOTIFICATION: sent to neighbor 66.170.0.3 4/0 (hold time expired) 0 bytes
Jul 3 06:57:06 mdsnwi-router-01 7546: Jul 3 06:56:00: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Up
Jul 3 06:57:06 mdsnwi-router-01 7547: Jul 3 06:56:21: %BGP-5-ADJCHANGE: neighbor 66.170.0.3 Up
Jul 3 06:59:58 mdsnwi-router-01 7548: Jul 3 06:59:58: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Down BGP Notification sent
Jul 3 06:59:59 mdsnwi-router-01 7549: Jul 3 06:59:58: %BGP-3-NOTIFICATION: sent to neighbor 24.177.132.109 4/0 (hold time expired) 0 bytes
Jul 3 07:00:45 mdsnwi-router-01 7550: Jul 3 07:00:44: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Up
Jul 3 07:01:13 mdsnwi-router-01 7551: Jul 3 07:01:13: %BGP-5-ADJCHANGE: neighbor 66.170.0.3 Down BGP Notification sent
Jul 3 07:01:14 mdsnwi-router-01 7552: Jul 3 07:01:13: %BGP-3-NOTIFICATION: sent to neighbor 66.170.0.3 4/0 (hold time expired) 0 bytes
Jul 3 07:01:50 mdsnwi-router-01 7553: Jul 3 07:01:49: %BGP-5-ADJCHANGE: neighbor 66.170.0.3 Up
Jul 3 07:11:57 mdsnwi-router-01 302: *Jul 3 07:05:58: %BGP-5-ADJCHANGE: neighbor 144.228.52.181 Up
Jul 3 07:11:57 mdsnwi-router-01 303: *Jul 3 07:06:02: %BGP-5-ADJCHANGE: neighbor 24.177.132.109 Up
Jul 3 07:11:57 mdsnwi-router-01 304: *Jul 3 07:06:08: %BGP-5-ADJCHANGE: neighbor 66.170.0.3 Up
Jul 3 07:11:57 mdsnwi-router-01 305: *Jul 3 07:06:08: %BGP-5-ADJCHANGE: neighbor 12.118.243.165 Up
Then the router appeared to lock up and possibly not respond on
the console so we had to power cycle it.
Jeff C.
--
Jeff Chan
mailto:cisco-nsp at jeffchan.com
http://www.supranet.net/
More information about the cisco-nsp
mailing list