[c-nsp] GSR: Failed to Allocate MBUS Channel / Fabric Handing Faield: Invalid bandwidth mode.
Eninja
eninja at gmail.com
Mon Mar 8 16:06:51 EST 2010
Bharath,
Your logs below suggest that these chain of events started with fabric/
active RP problems.
What does a 'sh contr fia' look like from the RP and an attach session
to all LCs? Also, what do the logs say now?
Finally, what changes were made to this device prior to all the errors?
/Eninja
On Mar 8, 2010, at 2:48 PM, bharath kondi
<bluffmaster4hearts at gmail.com> wrote:
> HI Everyone,
>
> Good day to you.
>
> I have some issues with our GSR 12012 router, kindly check the below
> log.
> This log is taken just before i lost connection to my router. Can
> anyone
> tell me what is wrong with our router.
>
> I am also attaching the log after I hard reboot the GSR. Once i hard
> reboot,
> everything back to normal for some time only, after 45 mins again i
> lost
> connectivity and all the BGP sessions went down, but the physical
> interfaces
> are UP connecting to other switches directly. The second time i hard
> reset
> and i got connectivity but after 20 mins i lost connection. This
> time i
> didn't hard reboot, i am waited for more than 30mins, then
> automatically i
> got the connection back and untill now it is working fine. I am
> attaching
> the logg after i got the connection automatically without reboot the
> router.
>
> Kindly please help me, i will be very much thankful to you guys.
>
>
> ~~~
> ~~~
> ~~~
> ~~~
> ~~~
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> *This is the first log before i loose connection my router:*
>
> SEC 1:*Mar 8 00:00:38.447: %RP-3-FABRIC_UNI: Unicast send timed
> out (0)
> *Mar 8 00:00:39.887: %LINK-3-UPDOWN: Interface Ethernet0, changed
> state to
> up
> *Mar 8 00:00:43.683: %RP-3-FABRIC_UNI: Unicast send timed out (0)
> *Mar 8 00:01:09.799: %RP-3-FABRIC_UNI: Unicast send timed out (1)
> *Mar 8 00:01:10.619: %FABRIC-3-ERR_HANDLE: Reconfigure all fabric
> cards due
> to RP SWITCHOVER PING ERROR error from slot 1
> *Mar 8 00:01:21.507: %FABRIC-3-ERR_HANDLE: Fabric not up even after
> reconfig! slot -1
> *Mar 8 00:01:37.183: %LINK-3-UPDOWN: Interface Loopback0, changed
> state to
> up
> *Mar 8 00:02:01.191: %MBUS_SYS-3-NOCHANNEL: Failed to allocate MBUS
> channel
> for over 10 secs
> *Mar 8 00:02:01.191: %MBUSFLASH-3-SEND: Error 8 sending cmd 167 to
> slot 1
> *Mar 8 00:02:04.351: %MBUSFLASH-3-SEND: Error 8 sending cmd 4 to
> slot 1
> *Mar 8 00:02:09.735: %MBUS_SYS-3-NOCHANNEL: Failed to allocate MBUS
> channel
> for over 10 secs
> *Mar 8 00:02:21.883: %MBUS_SYS-3-NOCHANNEL: Failed to allocate MBUS
> channel
> for over 10 secs
> *Mar 8 00:02:26.187: %MBUS_SYS-3-TXERR: Failed to transmit MBUS
> message for
> over 10 secs
> *Mar 8 00:02:41.247: %MBUS_SYS-3-NOCHANNEL: Failed to allocate MBUS
> channel
> for over 10 secs
> *Mar 8 00:02:41.247: %MBUSFLASH-3-SEND: Error 8 sending cmd 167 to
> slot 1
> *Mar 8 00:02:46.415: %MBUS_SYS-3-NOCHANNEL: Failed to allocate MBUS
> channel
> for over 10 secs
> *Mar 8 00:02:46.415: %MBUSFLASH-3-SEND: Error 8 sending cmd 2 to
> slot 1
> *Mar 8 00:02:50.435: %MBUS_SYS-3-TXERR: Failed to transmit MBUS
> message for
> over 10 secs
> *Mar 8 00:03:09.007: %MBUS_SYS-3-TXERR: Failed to transmit MBUS
> message for
> over 10 secs
> *Mar 8 00:03:13.311: %MBUS_SYS-3-NOCHANNEL: Failed to allocate MBUS
> channel
> for over 10 secs
> *Mar 8 00:03:21.107: %FIA-3-LOS: LOS for slot 20 was detected.
>
> *Mar 8 00:03:26.011: %FABRIC-0-OPERATIONAL: Fabric handling failed:
> Invalid
> bandwidth mode
> *Mar 8 00:03:32.107: %FIA-3-LOS: LOS for slot 20 was cleared.
>
> *Mar 8 00:03:40.591: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0;
> state
> UNKNSTAT
> -Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
> *Mar 8 00:03:46.639: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0;
> state
> UNKNSTAT
> -Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
> *Mar 8 00:03:52.687: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0;
> state
> RP RDY
> -Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
> *Mar 8 00:03:58.735: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0;
> state
> RP RDY
> -Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
> *Mar 8 00:04:04.783: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0;
> state
> RP RDY
> -Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
> *Mar 8 00:04:11.183: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0;
> state
> RP RDY
> -Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
> *Mar 8 00:04:14.487: %LINK-3-UPDOWN: Interface GigabitEthernet7/0,
> changed
> state to down
> *Mar 8 00:04:14.507: %LINK-3-UPDOWN: Interface GigabitEthernet10/0,
> changed
> state to down
> *Mar 8 00:04:17.231: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0;
> state
> RP RDY
> -Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
> *Mar 8 00:04:23.279: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0;
> state
> RP RDY
> -Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
> *Mar 8 00:04:29.327: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0;
> state
> RP RDY
> -Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
> SLOT 7:*Mar 8 00:04:28.855: %SYS-2-FREEBAD: Attempted to free
> memory at
> 415DAD20, not part of buffer pool
> -Traceback= 40031140 400F2AB4 400F64DC 4015FDEC 4015F59C 4016018C
> *Mar 8 00:04:30.979: %LINK-3-UPDOWN: Interface GigabitEthernet10/0,
> changed
> state to up
> *Mar 8 00:04:31.007: %LINK-3-UPDOWN: Interface GigabitEthernet7/0,
> changed
> state to up
> *Mar 8 00:04:35.375: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0;
> state
> RP RDY
> -Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
> *Mar 8 00:04:39.579: %LINK-3-UPDOWN: Interface GigabitEthernet8/0,
> changed
> state to down
> *Mar 8 00:04:40.523: %LINK-3-UPDOWN: Interface GigabitEthernet5/0,
> changed
> state to down
> *Mar 8 00:04:41.423: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0;
> state
> RP RDY
> -Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
> *Mar 8 00:04:41.875: %LINK-3-UPDOWN: Interface FastEthernet3/0,
> changed
> state to down
> *Mar 8 00:04:41.879: %LINK-3-UPDOWN: Interface FastEthernet3/1,
> changed
> state to down
> *Mar 8 00:04:41.879: %LINK-3-UPDOWN: Interface FastEthernet3/2,
> changed
> state to down
> *Mar 8 00:04:41.879: %LINK-3-UPDOWN: Interface FastEthernet3/3,
> changed
> state to down
> *Mar 8 00:04:41.879: %LINK-3-UPDOWN: Interface FastEthernet3/4,
> changed
> state to down
> *Mar 8 00:04:41.879: %LINK-3-UPDOWN: Interface FastEthernet3/5,
> changed
> state to down
> *Mar 8 00:04:41.879: %LINK-3-UPDOWN: Interface FastEthernet3/6,
> changed
> state to down
> *Mar 8 00:04:41.891: %LINK-3-UPDOWN: Interface FastEthernet3/7,
> changed
> state to down
> *Mar 8 00:04:47.471: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0;
> state
> RP RDY
> -Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
> *Mar 8 00:04:53.279: %LCCOREDUMP-3-BADOPEN: Can't open slave core
> socket
> for slot 0
> *Mar 8 00:04:53.519: %RP-3-BAD_LC_STATE: Cannot send pkt to slot 0;
> state
> RP RDY
> -Traceback= 501F246C 5050BB28 5050BD44 503A56D0 503A5FB0 503A64D4
> *Mar 8 00:04:55.103: %RP-3-RESTART: Linecard packet buffers recarved
> *Mar 8 00:04:56.719: %LINK-3-UPDOWN: Interface GigabitEthernet8/0,
> changed
> state to up
> *Mar 8 00:04:57.135: %RP-3-CARVE_FAIL: ToFab BMA, slot 5
> *Mar 8 00:04:57.135: %RP-3-COREDUMP: Core dump incident on slot 5,
> error:
> ToFab buffer carve failure
> *Mar 8 00:04:58.179: %LINK-3-UPDOWN: Interface GigabitEthernet5/0,
> changed
> state to up
> *Mar 8 00:04:58.551: %COSGRP-3-UNEXP: retry queue flush
> -Traceback= 501F246C 50345544 5034C77C 5034F91C
> *Mar 8 00:04:58.571: %LCINFO-3-CRASH: Line card in slot 5 crashed
> ~~~
> ~~~
> ~~~
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> *
> *
> *This is the log after i hard reboot the router:*
> *
> *
> *Mar 8 01:29:17.808: %FABRIC-0-OPERATIONAL: Fabric handling failed:
> Invalid
> bandwidth mode
> *Mar 8 01:32:09.572: %FABRIC-0-OPERATIONAL: Fabric handling failed:
> Self
> ping failed after fab reconfiguration
> *Mar 8 01:33:06.996: %FABRIC-0-OPERATIONAL: Fabric handling failed:
> Invalid
> bandwidth mode
> *Mar 8 01:36:47.979: %FABRIC-0-OPERATIONAL: Fabric handling failed:
> Invalid
> bandwidth mode
> *Mar 8 01:38:33.023: %FABRIC-0-OPERATIONAL: Fabric handling failed:
> Self
> ping failed after fab reconfiguration
> *Mar 8 01:38:44.035: %FABRIC-0-OPERATIONAL: Fabric handling failed:
> Invalid
> bandwidth mode
> *Mar 8 01:41:30.399: %FABRIC-0-OPERATIONAL: Fabric handling failed:
> Invalid
> bandwidth mode
> *Mar 8 01:42:04.091: %FABRIC-0-OPERATIONAL: Fabric handling failed:
> Invalid
> bandwidth mode
> ~~~
> ~~~
> ~~~
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> *This is the log after i got connection automatically:*
>
>
> *Mar 8 01:48:27.698: %FABRIC-0-OPERATIONAL: Fabric handling failed:
> Invalid
> bandwidth mode
> *Mar 8 01:51:32.238: %FABRIC-0-OPERATIONAL: Fabric handling failed:
> Self
> ping failed after fab reconfiguration
> *Mar 8 01:53:56.426: %FABRIC-0-OPERATIONAL: Fabric handling failed:
> Invalid
> bandwidth mode
> *Mar 8 01:59:06.061: %FABRIC-0-OPERATIONAL: Fabric handling failed:
> Invalid
> bandwidth mode
> ~~~
> ~~~
> ~~~
> ~~~
> ~~~
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> *This is some of the information from my router:*
>
> #sh cef linecard detail
> CEF linecard slot 5, status up, fibhwidb sync, fibidb sync, adj sync
> Sequence number 23543, Maximum sequence number expected 26038, Seq
> Epoch 2
> Send failed 0, Out Of Sequence 0, drops 0
> Linecard CEF reset 2, reloaded 1
> 546533 elements packed in 24136 messages(21964719 bytes) sent
> 0 elements cleared
> 0/0/0 xdr elements in LowQ/MediumQ/HighQ
> 1419/98/1514 peak elements on LowQ/MediumQ/HighQ
> Keepalive timer will expire in 00:05:59
> Init window timer is not running
> Input packets 1892483742, bytes 1196878280288
> Output packets 1581187348, bytes 468461207571, drops 0
> UTI input packets 0, bytes 0
> UTI output packets 0, bytes 0
> 0 proc sw, 0 proc sw fail, 0 hdr error
>
> CEF Table statistics:
> Table name Version Prefix-xdr Status
> Default 914875 541165 Active, sync, table-up
>
> CEF linecard slot 3, status up, fibhwidb sync, fibidb sync, adj sync
> Sequence number 23504, Maximum sequence number expected 25999, Seq
> Epoch 2
> Send failed 0, Out Of Sequence 0, drops 0
> Linecard CEF reset 1, reloaded 1
> 545877 elements packed in 24097 messages(21940002 bytes) sent
> 0 elements cleared
> 0/0/0 xdr elements in LowQ/MediumQ/HighQ
> 10201/384/564 peak elements on LowQ/MediumQ/HighQ
> Keepalive timer will expire in 00:05:59
> Init window timer is not running
> Input packets 620035311, bytes 285338977123
> Output packets 2242795, bytes 142721123, drops 7581117
> UTI input packets 0, bytes 0
> UTI output packets 0, bytes 0
> 0 proc sw, 0 proc sw fail, 0 hdr error
>
> CEF Table statistics:
> Table name Version Prefix-xdr Status
> Default 914875 541163 Active, sync, table-up
>
> CEF linecard slot 10, status up, fibhwidb sync, fibidb sync, adj sync
> Sequence number 23499, Maximum sequence number expected 25994, Seq
> Epoch 3
> Send failed 0, Out Of Sequence 0, drops 0
> Linecard CEF reset 1, reloaded 1
> 545400 elements packed in 24092 messages(21924913 bytes) sent
> 0 elements cleared
> 0/0/0 xdr elements in LowQ/MediumQ/HighQ
> 10201/98/663 peak elements on LowQ/MediumQ/HighQ
> Keepalive timer will expire in 00:05:59
> Init window timer is not running
> Input packets 1198515706, bytes 888383104966
> Output packets 636902739, bytes 180880757134, drops 0
> UTI input packets 0, bytes 0
> UTI output packets 0, bytes 0
> 0 proc sw, 0 proc sw fail, 0 hdr error
>
> CEF Table statistics:
> Table name Version Prefix-xdr Status
> Default 914875 541163 Active, sync, table-up
>
> CEF linecard slot 8, status up, fibhwidb sync, fibidb sync, adj sync
> Sequence number 23493, Maximum sequence number expected 25988, Seq
> Epoch 1
> Send failed 0, Out Of Sequence 0, drops 0
> Linecard CEF reset 1, reloaded 1
> 545104 elements packed in 24086 messages(21916792 bytes) sent
> 0 elements cleared
> 0/0/0 xdr elements in LowQ/MediumQ/HighQ
> 11718/100/565 peak elements on LowQ/MediumQ/HighQ
> Keepalive timer will expire in 00:05:59
> Init window timer is not running
> Input packets 2444665602, bytes 700700988650
> Output packets 2824179964, bytes 1820085577053, drops 0
> UTI input packets 0, bytes 0
> UTI output packets 0, bytes 0
> 0 proc sw, 0 proc sw fail, 0 hdr error
>
> CEF Table statistics:
> Table name Version Prefix-xdr Status
> Default 914875 541163 Active, sync, table-up
>
> CEF linecard slot 7, status up, fibhwidb sync, fibidb sync, adj sync
> Sequence number 22883, Maximum sequence number expected 25378, Seq
> Epoch 4
> Send failed 0, Out Of Sequence 0, drops 0
> Linecard CEF reset 2, reloaded 2
> 504934 elements packed in 23490 messages(20388441 bytes) sent
> 4 elements cleared
> 0/0/0 xdr elements in LowQ/MediumQ/HighQ
> 10000/98/651 peak elements on LowQ/MediumQ/HighQ
> Keepalive timer will expire in 00:05:59
> Init window timer is not running
> Input packets 183903517, bytes 103969609979
> Output packets 264694307, bytes 206059685347, drops 0
> UTI input packets 0, bytes 0
> UTI output packets 0, bytes 0
> 0 proc sw, 0 proc sw fail, 0 hdr error
>
> CEF Table statistics:
> Table name Version Prefix-xdr Status
> Default 914881 501088 Active, sync, table-up
>
> CEF RRP slot 1, status up, fibhwidb sync, fibidb sync, adj sync
> Sequence number 22836, Maximum sequence number expected 25331, Seq
> Epoch 1
> Send failed 0, Out Of Sequence 0, drops 0
> Linecard CEF reset 1, reloaded 1
> 501081 elements packed in 23428 messages(20239920 bytes) sent
> 0 elements cleared
> 0/0/0 xdr elements in LowQ/MediumQ/HighQ
> 10000/99/654 peak elements on LowQ/MediumQ/HighQ
> Keepalive timer will expire in 00:05:59
> Init window timer is not running
> Input packets 0, bytes 0
> Output packets 0, bytes 0, drops 0
> UTI input packets 0, bytes 0
> UTI output packets 0, bytes 0
> 0 proc sw, 0 proc sw fail, 0 hdr error
>
> CEF Table statistics:
> Table name Version Prefix-xdr Status
> Default 914881 497471 Active, sync, table-up
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> #sh gsr
> Slot 0 type = Route Processor
> state = ACTV RP IOS Running ACTIVE
> Slot 1 type = Route Processor
> state = STBY RP IOS Running STANDBY
> Slot 3 type = 8 Port Fast Ethernet
> state = IOS RUN Line Card Enabled
> Slot 5 type = 1 Port Gigabit Ethernet
> state = IOS RUN Line Card Enabled
> Slot 7 type = 1 Port Gigabit Ethernet
> state = IOS RUN Line Card Enabled
> Slot 8 type = 1 Port Gigabit Ethernet
> state = IOS RUN Line Card Enabled
> Slot 10 type = 1 Port Gigabit Ethernet
> state = IOS RUN Line Card Enabled
> Slot 17 type = Clock Scheduler Card
> state = Card Powered PRIMARY CLOCK
> Slot 18 type = Switch Fabric Card
> state = Card Powered
> Slot 19 type = Switch Fabric Card
> state = Card Powered
> Slot 20 type = Switch Fabric Card
> state = Card Powered
> Slot 24 type = AC Power Supply
> state = Card Powered
> Slot 25 type = AC Power Supply
> state = Card Powered
> Slot 27 type = AC Power Supply
> state = Card Powered
> Slot 28 type = Blower Module(12)
> state = Card Powered
> Slot 29 type = Blower Module(12)
> state = Card Powered
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> #show fab
> Dest ToFab FrFab Bad Unexpected Lost
> Slot Pkts Pkts Seq Pkts Pkts
> ----------------------------------------------------------
> Slot0 11911 11874 0 0 37
> Slot1 11836 11833 0 0 3
> Slot3 11830 11830 0 0 0
> Slot5 11837 11837 0 0 0
> Slot7 11810 11810 0 0 0
> Slot8 11829 11829 0 0 0
> Slot10 11829 11829 0 0 0
>
> Round-trip delay profile (mS)
> 00 100 200 300 500 1000 2000 5000
> 10000
> 99 ms 199 ms 299 ms 499 ms 999 ms 1999 ms 4999
> 9999 and
> up
>
> ---
> ---
> ---------------------------------------------------------------------
> 0 11874 0 0 0 0 0 0
> 0 0
>
> 1 11833 0 0 0 0 0 0
> 0 0
>
> 3 11829 1 0 0 0 0 0
> 0 0
>
> 5 11837 0 0 0 0 0 0
> 0 0
>
> 7 11810 0 0 0 0 0 0
> 0 0
>
> 8 11829 0 0 0 0 0 0
> 0 0
>
> 10 11829 0 0 0 0 0 0
> 0 0
>
>
> Max RTT profile (mS)
> Round-trip
> Slot Maximum
> ---- -------
> 0 4 02:01:02 GMT-MY Mar 8 2010
> 1 4 04:47:43 GMT-MY Mar 8 2010
> 3 192 02:00:56 GMT-MY Mar 8 2010
> 5 4 02:06:59 GMT-MY Mar 8 2010
> 7 4 02:47:09 GMT-MY Mar 8 2010
> 8 4 02:02:26 GMT-MY Mar 8 2010
> 10 4 02:02:56 GMT-MY Mar 8 2010
>
> Last clearing of "show fabric" counters never
> multicast timeout 0
> failed pak 0
> Current fabric timeout is 6000
>
> fabric send fails 3
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> #sh context summary
> CRASH INFO SUMMARY
> Slot 0 : 0 crashes
> Slot 1 : 0 crashes
> Slot 2 : 0 crashes
> Slot 3 : 0 crashes
> Slot 4 : 0 crashes
> Slot 5 : 0 crashes
> Slot 6 : 0 crashes
> Slot 7 : 0 crashes
> Slot 8 : 0 crashes
> Slot 9 : 0 crashes
> Slot 10: 0 crashes
> Slot 11: 0 crashes
> RESET INFO SUMMARY
> Slot 0 : 0 resets
> Slot 1 : 0 resets
> Slot 2 : 0 resets
> Slot 3 : 0 resets
> Slot 4 : 0 resets
> Slot 5 : 0 resets
> Slot 6 : 0 resets
> Slot 7 : 0 resets
> Slot 8 : 0 resets
> Slot 9 : 0 resets
> Slot 10: 0 resets
> Slot 11: 0 resets
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
More information about the cisco-nsp
mailing list