[c-nsp] GSR: Failed to Allocate MBUS Channel / Fabric Handing Faield: Invalid bandwidth mode.
bharath kondi
bluffmaster4hearts at gmail.com
Mon Mar 8 08:48:21 EST 2010
HI Everyone,
Good day to you.
I have some issues with our GSR 12012 router, kindly check the below log.
This log is taken just before i lost connection to my router. Can anyone
tell me what is wrong with our router.
I am also attaching the log after I hard reboot the GSR. Once i hard reboot,
everything back to normal for some time only, after 45 mins again i lost
connectivity and all the BGP sessions went down, but the physical interfaces
are UP connecting to other switches directly. The second time i hard reset
and i got connectivity but after 20 mins i lost connection. This time i
didn't hard reboot, i am waited for more than 30mins, then automatically i
got the connection back and untill now it is working fine. I am attaching
the logg after i got the connection automatically without reboot the router.
Kindly please help me, i will be very much thankful to you guys.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*This is the first log before i loose connection my router:*
SEC 1:*Mar 8 00:00:38.447: %RP-3-FABRIC_UNI: Unicast send timed out (0)
*Mar 8 00:00:39.887: %LINK-3-UPDOWN: Interface Ethernet0, changed state to
up
*Mar 8 00:00:43.683: %RP-3-FABRIC_UNI: Unicast send timed out (0)
*Mar 8 00:01:09.799: %RP-3-FABRIC_UNI: Unicast send timed out (1)
*Mar 8 00:01:10.619: %FABRIC-3-ERR_HANDLE: Reconfigure all fabric cards due
to RP SWITCHOVER PING ERROR error from slot 1
*Mar 8 00:01:21.507: %FABRIC-3-ERR_HANDLE: Fabric not up even after
reconfig! slot -1
*Mar 8 00:01:37.183: %LINK-3-UPDOWN: Interface Loopback0, changed state to
up
*Mar 8 00:02:01.191: %MBUS_SYS-3-NOCHANNEL: Failed to allocate MBUS channel
for over 10 secs
*Mar 8 00:02:01.191: %MBUSFLASH-3-SEND: Error 8 sending cmd 167 to slot 1
*Mar 8 00:02:04.351: %MBUSFLASH-3-SEND: Error 8 sending cmd 4 to slot 1
*Mar 8 00:02:09.735: %MBUS_SYS-3-NOCHANNEL: Failed to allocate MBUS channel
for over 10 secs
*Mar 8 00:02:21.883: %MBUS_SYS-3-NOCHANNEL: Failed to allocate MBUS channel
for over 10 secs
*Mar 8 00:02:26.187: %MBUS_SYS-3-TXERR: Failed to transmit MBUS message for
over 10 secs
*Mar 8 00:02:41.247: %MBUS_SYS-3-NOCHANNEL: Failed to allocate MBUS channel
for over 10 secs
*Mar 8 00:02:41.247: %MBUSFLASH-3-SEND: Error 8 sending cmd 167 to slot 1
*Mar 8 00:02:46.415: %MBUS_SYS-3-NOCHANNEL: Failed to allocate MBUS channel
for over 10 secs
*Mar 8 00:02:46.415: %MBUSFLASH-3-SEND: Error 8 sending cmd 2 to slot 1
*Mar 8 00:02:50.435: %MBUS_SYS-3-TXERR: Failed to transmit MBUS message for
over 10 secs
*Mar 8 00:03:09.007: %MBUS_SYS-3-TXERR: Failed to transmit MBUS message for
over 10 secs
*Mar 8 00:03:13.311: %MBUS_SYS-3-NOCHANNEL: Failed to allocate MBUS channel
for over 10 secs
*Mar 8 00:03:21.107: %FIA-3-LOS: LOS for slot 20 was detected.
*Mar 8 00:03:26.011: %FABRIC-0-OPERATIONAL: Fabric handling failed: Invalid
bandwidth mode
*Mar 8 00:03:32.107: %FIA-3-LOS: LOS for slot 20 was cleared.
*Mar 8 00:03:40.591: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0; state
UNKNSTAT
-Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
*Mar 8 00:03:46.639: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0; state
UNKNSTAT
-Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
*Mar 8 00:03:52.687: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0; state
RP RDY
-Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
*Mar 8 00:03:58.735: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0; state
RP RDY
-Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
*Mar 8 00:04:04.783: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0; state
RP RDY
-Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
*Mar 8 00:04:11.183: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0; state
RP RDY
-Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
*Mar 8 00:04:14.487: %LINK-3-UPDOWN: Interface GigabitEthernet7/0, changed
state to down
*Mar 8 00:04:14.507: %LINK-3-UPDOWN: Interface GigabitEthernet10/0, changed
state to down
*Mar 8 00:04:17.231: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0; state
RP RDY
-Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
*Mar 8 00:04:23.279: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0; state
RP RDY
-Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
*Mar 8 00:04:29.327: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0; state
RP RDY
-Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
SLOT 7:*Mar 8 00:04:28.855: %SYS-2-FREEBAD: Attempted to free memory at
415DAD20, not part of buffer pool
-Traceback= 40031140 400F2AB4 400F64DC 4015FDEC 4015F59C 4016018C
*Mar 8 00:04:30.979: %LINK-3-UPDOWN: Interface GigabitEthernet10/0, changed
state to up
*Mar 8 00:04:31.007: %LINK-3-UPDOWN: Interface GigabitEthernet7/0, changed
state to up
*Mar 8 00:04:35.375: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0; state
RP RDY
-Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
*Mar 8 00:04:39.579: %LINK-3-UPDOWN: Interface GigabitEthernet8/0, changed
state to down
*Mar 8 00:04:40.523: %LINK-3-UPDOWN: Interface GigabitEthernet5/0, changed
state to down
*Mar 8 00:04:41.423: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0; state
RP RDY
-Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
*Mar 8 00:04:41.875: %LINK-3-UPDOWN: Interface FastEthernet3/0, changed
state to down
*Mar 8 00:04:41.879: %LINK-3-UPDOWN: Interface FastEthernet3/1, changed
state to down
*Mar 8 00:04:41.879: %LINK-3-UPDOWN: Interface FastEthernet3/2, changed
state to down
*Mar 8 00:04:41.879: %LINK-3-UPDOWN: Interface FastEthernet3/3, changed
state to down
*Mar 8 00:04:41.879: %LINK-3-UPDOWN: Interface FastEthernet3/4, changed
state to down
*Mar 8 00:04:41.879: %LINK-3-UPDOWN: Interface FastEthernet3/5, changed
state to down
*Mar 8 00:04:41.879: %LINK-3-UPDOWN: Interface FastEthernet3/6, changed
state to down
*Mar 8 00:04:41.891: %LINK-3-UPDOWN: Interface FastEthernet3/7, changed
state to down
*Mar 8 00:04:47.471: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0; state
RP RDY
-Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
*Mar 8 00:04:53.279: %LCCOREDUMP-3-BADOPEN: Can't open slave core socket
for slot 0
*Mar 8 00:04:53.519: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0; state
RP RDY
-Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
*Mar 8 00:04:55.103: %RP-3-RESTART: Linecard packet buffers recarved
*Mar 8 00:04:56.719: %LINK-3-UPDOWN: Interface GigabitEthernet8/0, changed
state to up
*Mar 8 00:04:57.135: %RP-3-CARVE_FAIL: ToFab BMA, slot 5
*Mar 8 00:04:57.135: %RP-3-COREDUMP: Core dump incident on slot 5, error:
ToFab buffer carve failure
*Mar 8 00:04:58.179: %LINK-3-UPDOWN: Interface GigabitEthernet5/0, changed
state to up
*Mar 8 00:04:58.551: %COSGRP-3-UNEXP: retry queue flush
-Traceback= 501F246C 50345544 5034C77C 5034F91C
*Mar 8 00:04:58.571: %LCINFO-3-CRASH: Line card in slot 5 crashed
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*
*
*This is the log after i hard reboot the router:*
*
*
*Mar 8 01:29:17.808: %FABRIC-0-OPERATIONAL: Fabric handling failed: Invalid
bandwidth mode
*Mar 8 01:32:09.572: %FABRIC-0-OPERATIONAL: Fabric handling failed: Self
ping failed after fab reconfiguration
*Mar 8 01:33:06.996: %FABRIC-0-OPERATIONAL: Fabric handling failed: Invalid
bandwidth mode
*Mar 8 01:36:47.979: %FABRIC-0-OPERATIONAL: Fabric handling failed: Invalid
bandwidth mode
*Mar 8 01:38:33.023: %FABRIC-0-OPERATIONAL: Fabric handling failed: Self
ping failed after fab reconfiguration
*Mar 8 01:38:44.035: %FABRIC-0-OPERATIONAL: Fabric handling failed: Invalid
bandwidth mode
*Mar 8 01:41:30.399: %FABRIC-0-OPERATIONAL: Fabric handling failed: Invalid
bandwidth mode
*Mar 8 01:42:04.091: %FABRIC-0-OPERATIONAL: Fabric handling failed: Invalid
bandwidth mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*This is the log after i got connection automatically:*
*Mar 8 01:48:27.698: %FABRIC-0-OPERATIONAL: Fabric handling failed: Invalid
bandwidth mode
*Mar 8 01:51:32.238: %FABRIC-0-OPERATIONAL: Fabric handling failed: Self
ping failed after fab reconfiguration
*Mar 8 01:53:56.426: %FABRIC-0-OPERATIONAL: Fabric handling failed: Invalid
bandwidth mode
*Mar 8 01:59:06.061: %FABRIC-0-OPERATIONAL: Fabric handling failed: Invalid
bandwidth mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*This is some of the information from my router:*
#sh cef linecard detail
CEF linecard slot 5, status up, fibhwidb sync, fibidb sync, adj sync
Sequence number 23543, Maximum sequence number expected 26038, Seq Epoch 2
Send failed 0, Out Of Sequence 0, drops 0
Linecard CEF reset 2, reloaded 1
546533 elements packed in 24136 messages(21964719 bytes) sent
0 elements cleared
0/0/0 xdr elements in LowQ/MediumQ/HighQ
1419/98/1514 peak elements on LowQ/MediumQ/HighQ
Keepalive timer will expire in 00:05:59
Init window timer is not running
Input packets 1892483742, bytes 1196878280288
Output packets 1581187348, bytes 468461207571, drops 0
UTI input packets 0, bytes 0
UTI output packets 0, bytes 0
0 proc sw, 0 proc sw fail, 0 hdr error
CEF Table statistics:
Table name Version Prefix-xdr Status
Default 914875 541165 Active, sync, table-up
CEF linecard slot 3, status up, fibhwidb sync, fibidb sync, adj sync
Sequence number 23504, Maximum sequence number expected 25999, Seq Epoch 2
Send failed 0, Out Of Sequence 0, drops 0
Linecard CEF reset 1, reloaded 1
545877 elements packed in 24097 messages(21940002 bytes) sent
0 elements cleared
0/0/0 xdr elements in LowQ/MediumQ/HighQ
10201/384/564 peak elements on LowQ/MediumQ/HighQ
Keepalive timer will expire in 00:05:59
Init window timer is not running
Input packets 620035311, bytes 285338977123
Output packets 2242795, bytes 142721123, drops 7581117
UTI input packets 0, bytes 0
UTI output packets 0, bytes 0
0 proc sw, 0 proc sw fail, 0 hdr error
CEF Table statistics:
Table name Version Prefix-xdr Status
Default 914875 541163 Active, sync, table-up
CEF linecard slot 10, status up, fibhwidb sync, fibidb sync, adj sync
Sequence number 23499, Maximum sequence number expected 25994, Seq Epoch 3
Send failed 0, Out Of Sequence 0, drops 0
Linecard CEF reset 1, reloaded 1
545400 elements packed in 24092 messages(21924913 bytes) sent
0 elements cleared
0/0/0 xdr elements in LowQ/MediumQ/HighQ
10201/98/663 peak elements on LowQ/MediumQ/HighQ
Keepalive timer will expire in 00:05:59
Init window timer is not running
Input packets 1198515706, bytes 888383104966
Output packets 636902739, bytes 180880757134, drops 0
UTI input packets 0, bytes 0
UTI output packets 0, bytes 0
0 proc sw, 0 proc sw fail, 0 hdr error
CEF Table statistics:
Table name Version Prefix-xdr Status
Default 914875 541163 Active, sync, table-up
CEF linecard slot 8, status up, fibhwidb sync, fibidb sync, adj sync
Sequence number 23493, Maximum sequence number expected 25988, Seq Epoch 1
Send failed 0, Out Of Sequence 0, drops 0
Linecard CEF reset 1, reloaded 1
545104 elements packed in 24086 messages(21916792 bytes) sent
0 elements cleared
0/0/0 xdr elements in LowQ/MediumQ/HighQ
11718/100/565 peak elements on LowQ/MediumQ/HighQ
Keepalive timer will expire in 00:05:59
Init window timer is not running
Input packets 2444665602, bytes 700700988650
Output packets 2824179964, bytes 1820085577053, drops 0
UTI input packets 0, bytes 0
UTI output packets 0, bytes 0
0 proc sw, 0 proc sw fail, 0 hdr error
CEF Table statistics:
Table name Version Prefix-xdr Status
Default 914875 541163 Active, sync, table-up
CEF linecard slot 7, status up, fibhwidb sync, fibidb sync, adj sync
Sequence number 22883, Maximum sequence number expected 25378, Seq Epoch 4
Send failed 0, Out Of Sequence 0, drops 0
Linecard CEF reset 2, reloaded 2
504934 elements packed in 23490 messages(20388441 bytes) sent
4 elements cleared
0/0/0 xdr elements in LowQ/MediumQ/HighQ
10000/98/651 peak elements on LowQ/MediumQ/HighQ
Keepalive timer will expire in 00:05:59
Init window timer is not running
Input packets 183903517, bytes 103969609979
Output packets 264694307, bytes 206059685347, drops 0
UTI input packets 0, bytes 0
UTI output packets 0, bytes 0
0 proc sw, 0 proc sw fail, 0 hdr error
CEF Table statistics:
Table name Version Prefix-xdr Status
Default 914881 501088 Active, sync, table-up
CEF RRP slot 1, status up, fibhwidb sync, fibidb sync, adj sync
Sequence number 22836, Maximum sequence number expected 25331, Seq Epoch 1
Send failed 0, Out Of Sequence 0, drops 0
Linecard CEF reset 1, reloaded 1
501081 elements packed in 23428 messages(20239920 bytes) sent
0 elements cleared
0/0/0 xdr elements in LowQ/MediumQ/HighQ
10000/99/654 peak elements on LowQ/MediumQ/HighQ
Keepalive timer will expire in 00:05:59
Init window timer is not running
Input packets 0, bytes 0
Output packets 0, bytes 0, drops 0
UTI input packets 0, bytes 0
UTI output packets 0, bytes 0
0 proc sw, 0 proc sw fail, 0 hdr error
CEF Table statistics:
Table name Version Prefix-xdr Status
Default 914881 497471 Active, sync, table-up
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#sh gsr
Slot 0 type = Route Processor
state = ACTV RP IOS Running ACTIVE
Slot 1 type = Route Processor
state = STBY RP IOS Running STANDBY
Slot 3 type = 8 Port Fast Ethernet
state = IOS RUN Line Card Enabled
Slot 5 type = 1 Port Gigabit Ethernet
state = IOS RUN Line Card Enabled
Slot 7 type = 1 Port Gigabit Ethernet
state = IOS RUN Line Card Enabled
Slot 8 type = 1 Port Gigabit Ethernet
state = IOS RUN Line Card Enabled
Slot 10 type = 1 Port Gigabit Ethernet
state = IOS RUN Line Card Enabled
Slot 17 type = Clock Scheduler Card
state = Card Powered PRIMARY CLOCK
Slot 18 type = Switch Fabric Card
state = Card Powered
Slot 19 type = Switch Fabric Card
state = Card Powered
Slot 20 type = Switch Fabric Card
state = Card Powered
Slot 24 type = AC Power Supply
state = Card Powered
Slot 25 type = AC Power Supply
state = Card Powered
Slot 27 type = AC Power Supply
state = Card Powered
Slot 28 type = Blower Module(12)
state = Card Powered
Slot 29 type = Blower Module(12)
state = Card Powered
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#show fab
Dest ToFab FrFab Bad Unexpected Lost
Slot Pkts Pkts Seq Pkts Pkts
----------------------------------------------------------
Slot0 11911 11874 0 0 37
Slot1 11836 11833 0 0 3
Slot3 11830 11830 0 0 0
Slot5 11837 11837 0 0 0
Slot7 11810 11810 0 0 0
Slot8 11829 11829 0 0 0
Slot10 11829 11829 0 0 0
Round-trip delay profile (mS)
00 100 200 300 500 1000 2000 5000
10000
99 ms 199 ms 299 ms 499 ms 999 ms 1999 ms 4999 9999 and
up
---------------------------------------------------------------------------
0 11874 0 0 0 0 0 0 0 0
1 11833 0 0 0 0 0 0 0 0
3 11829 1 0 0 0 0 0 0 0
5 11837 0 0 0 0 0 0 0 0
7 11810 0 0 0 0 0 0 0 0
8 11829 0 0 0 0 0 0 0 0
10 11829 0 0 0 0 0 0 0 0
Max RTT profile (mS)
Round-trip
Slot Maximum
---- -------
0 4 02:01:02 GMT-MY Mar 8 2010
1 4 04:47:43 GMT-MY Mar 8 2010
3 192 02:00:56 GMT-MY Mar 8 2010
5 4 02:06:59 GMT-MY Mar 8 2010
7 4 02:47:09 GMT-MY Mar 8 2010
8 4 02:02:26 GMT-MY Mar 8 2010
10 4 02:02:56 GMT-MY Mar 8 2010
Last clearing of "show fabric" counters never
multicast timeout 0
failed pak 0
Current fabric timeout is 6000
fabric send fails 3
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#sh context summary
CRASH INFO SUMMARY
Slot 0 : 0 crashes
Slot 1 : 0 crashes
Slot 2 : 0 crashes
Slot 3 : 0 crashes
Slot 4 : 0 crashes
Slot 5 : 0 crashes
Slot 6 : 0 crashes
Slot 7 : 0 crashes
Slot 8 : 0 crashes
Slot 9 : 0 crashes
Slot 10: 0 crashes
Slot 11: 0 crashes
RESET INFO SUMMARY
Slot 0 : 0 resets
Slot 1 : 0 resets
Slot 2 : 0 resets
Slot 3 : 0 resets
Slot 4 : 0 resets
Slot 5 : 0 resets
Slot 6 : 0 resets
Slot 7 : 0 resets
Slot 8 : 0 resets
Slot 9 : 0 resets
Slot 10: 0 resets
Slot 11: 0 resets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
More information about the cisco-nsp
mailing list