[c-nsp] Troubleshooting Lag between GigE interfaces

Paul Stewart pauls at nexicom.net
Thu Sep 23 12:36:22 EDT 2004


Thanks Rodney...

I appreciate your help on this and hope the list has benefit from this
discussion.... this issue crept on me to be honest...

Here's some followup to prove memory as the issue:

gw-7513#if-con 8
Console or Debug [C]:
Entering CONSOLE for GEIP 8
Type "^C^C^C" or "if-quit" to end this session
 
 
%% Low on memory; try again later
 
%% Low on memory; try again later
 
%% Low on memory; try again later
 
%% Low on memory; try again later
 
%% Low on memory; try again later
 
%% Low on memory; try again later
 
Terminate DBUS IO session? [confirm]
 
Disconnecting from slot 8 CONSOLE after 00:00:21
 
gw-7513#if-con 9
Console or Debug [C]:
Entering CONSOLE for GEIP 9
Type "^C^C^C" or "if-quit" to end this session
 
 
%% Low on memory; try again later
 
%% Low on memory; try again later
 
%% Low on memory; try again later
 
%% Low on memory; try again later
 
%% Low on memory; try again later
 
%% Low on memory; try again later
 
%% Low on memory; try again later
 
Terminate DBUS IO session? [confirm]
 
Disconnecting from slot 9 CONSOLE after 00:00:09


Time to upgrade..:)

Paul


On Thu, 2004-09-23 at 12:22, Rodney Dunn wrote:
> If you have any significant number of routes
> this isn't enough memory.
> 
> You can check what's free on the GEIP by
> doing:
> 
> 
> if-con <slot>
> sh proc mem
> 
> ie:
> 
> W2N-4.4-7513b#if-con 0
> Console or Debug [C]: 
> Entering CONSOLE for VIP4-80 RM7000 0
> Type "^C^C^C" or "if-quit" to end this session
> 
> 
> VIP-Slot0>
> VIP-Slot0>en
> VIP-Slot0#sh proc mem
> Processor Pool Total:  256802816 Used:   27282600 Free:  229520216
>       PCI Pool Total:   67108608 Used:   67092376 Free:      16232
> 
> 
> You can even get to the point where the image that has to
> be loaded on the VIP can't even load if the memory is too low.
> As you upgrade, the image gets larger because new features are
> added and bugfixes.
> 
> One other thing, if anyone ever wants to run dCEF on a 75xx
> they should make sure ALL cards support it and that means
> getting rid of the old legacy ip boards (ie: FSIP).
> You need to get a VIP/serial PA combination for those links.
> 
> Now that we have figured this out, even with the packets 
> being switched at interrupt level by the RSP you shouldn't
> see 60 msec of latency.
> 
> If you turn off dCEF on a 75xx you basically have a 72xx
> with more slots.  Everything is handled by the RSP.
> 
> Rodney
> 
> 
> 
> On Thu, Sep 23, 2004 at 12:14:21PM -0400, Paul Stewart wrote:
> > Hmm... I think we're getting warmer..:)
> > 
> > Here's the "sh diag":
> > 
> > gw-7513#sh diagbus
> > Slot 3:
> >         Physical slot 3, ~physical slot 0xC, logical slot 3, CBus 0
> >         Microcode Status 0x0
> >         Master Enable, LED, WCS Loaded
> >         Board is analyzed
> >         EEPROM format version 1
> >         FSIP controller, HW rev 1.01, board revision C0
> >         Serial number: 03496276  Part number: 73-1126-05
> >         Test history: 0x00        RMA number: 00-00-00
> >         Flags: cisco 7000 board; 7500 compatible
> >  
> >         EEPROM contents (hex):
> >           0x20: 01 07 01 01 00 35 59 54 49 04 66 05 00 00 00 00
> >           0x30: 60 00 00 00 FF 00 00 00 00 00 00 00 00 00 00 00
> >  
> >         Slot database information:
> >         Flags: 0x4      Insertion time: 0x4380 (1w1d ago)
> > Slot 6:
> >         EEPROM format version 1
> >         Route/Switch Processor 4, HW rev 1.03, board revision C0
> >         Serial number: 15308129  Part number: 73-1689-05
> >         Test history: 0x00        RMA number: 00-00-00
> >         Flags: cisco 7000 board; 7500 compatible
> >  
> >         EEPROM contents (hex):
> >           0x20: 01 1A 01 03 00 E9 95 61 49 06 99 05 00 00 00 00
> >           0x30: 60 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00
> > Slot 7:
> >         EEPROM format version 1
> >         Route/Switch Processor 4, HW rev 1.03, board revision C0
> >         Serial number: 15308150  Part number: 73-1689-05
> >         Test history: 0x00        RMA number: 00-00-00
> >         Flags: cisco 7000 board; 7500 compatible
> >  
> >         EEPROM contents (hex):
> >           0x20: 01 1A 01 03 00 E9 95 76 49 06 99 05 00 00 00 00
> >           0x30: 60 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> > Slot 8:
> >         Physical slot 8, ~physical slot 0x7, logical slot 8, CBus 1
> >         Microcode Status 0x4
> >         Master Enable, LED, WCS Loaded
> >         Board is analyzed
> >         Pending I/O Status: None
> >         EEPROM format version 1
> >         GEIP controller, HW rev 2.03, board revision A0
> >         Serial number: 18950511  Part number: 73-2167-06
> >         Test history: 0x00        RMA number: 00-00-00
> >         Flags: cisco 7000 board; 7500 compatible
> >  
> >         EEPROM contents (hex):
> >           0x20: 01 21 02 03 01 21 29 6F 49 08 77 06 00 00 00 00
> >           0x30: 50 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> >  
> >         Slot database information:
> >         Flags: 0x4      Insertion time: 0x4380 (1w1d ago)
> >  
> >         Controller Memory Size: 32 MBytes DRAM, 4096 KBytes SRAM
> >  
> >         PA Bay 1 Information:
> >                 Gigabit-Ethernet PA, 1 ports
> >                 EEPROM format version 1
> >                 HW rev 1.00, Board revision A1
> >                 Serial number: 17941328  Part number: 73-3144-03
> >  
> >         --Boot log begin--
> >  
> > Cisco Internetwork Operating System Software
> > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10), RELEASE SOFTWARE
> > (fc3)
> > Copyright (c) 1986-2004 by cisco Systems, Inc.
> > Compiled Mon 16-Aug-04 21:50 by kellythw
> > Image text-base: 0x60010AE4, data-base: 0x60560000
> >  
> >  
> >         --Boot log end--
> >  
> > Slot 9:
> >         Physical slot 9, ~physical slot 0x6, logical slot 9, CBus 1
> >         Microcode Status 0x4
> >         Master Enable, LED, WCS Loaded
> >         Board is analyzed
> >         Pending I/O Status: None
> >         EEPROM format version 1
> >         GEIP controller, HW rev 2.02, board revision C0
> >         Serial number: 13943816  Part number: 73-2167-05
> >         Test history: 0x00        RMA number: 00-00-00
> >         Flags: cisco 7000 board; 7500 compatible
> >  
> >         EEPROM contents (hex):
> >           0x20: 01 21 02 02 00 D4 C4 08 49 08 77 05 00 00 00 00
> >           0x30: 60 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> >  
> >         Slot database information:
> >         Flags: 0x4      Insertion time: 0x4380 (1w1d ago)
> >  
> >         Controller Memory Size: 32 MBytes DRAM, 4096 KBytes SRAM
> >  
> >         PA Bay 1 Information:
> >                 Gigabit-Ethernet PA, 1 ports
> >                 EEPROM format version 1
> >                 HW rev 1.00, Board revision A1
> >                 Serial number: 22598189  Part number: 73-3144-03
> >  
> >         --Boot log begin--
> >  
> > Cisco Internetwork Operating System Software
> > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10), RELEASE SOFTWARE
> > (fc3)
> > Copyright (c) 1986-2004 by cisco Systems, Inc.
> > Compiled Mon 16-Aug-04 21:50 by kellythw
> > Image text-base: 0x60010AE4, data-base: 0x60560000
> >  
> >  
> >         --Boot log end--
> >            
> > Slot 10:
> >         Physical slot 10, ~physical slot 0x5, logical slot 10, CBus 1
> >         Microcode Status 0x4
> >         Master Enable, LED, WCS Loaded
> >         Board is analyzed
> >         Pending I/O Status: None
> >         EEPROM format version 1
> >         VIP2 R5K controller, HW rev 2.03, board revision A0
> >         Serial number: 17944577  Part number: 73-2167-06
> >         Test history: 0x00        RMA number: 00-00-00
> >         Flags: cisco 7000 board; 7500 compatible
> >  
> >         EEPROM contents (hex):
> >           0x20: 01 1E 02 03 01 11 D0 01 49 08 77 06 00 00 00 00
> >           0x30: 50 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> >  
> >         Slot database information:
> >         Flags: 0x4      Insertion time: 0x4380 (1w1d ago)
> >  
> >         Controller Memory Size: 128 MBytes DRAM, 8192 KBytes SRAM
> >  
> >         PA Bay 0 Information:
> >                 Fast-Ethernet PA, 2 ports, 100BaseTX-ISL
> >                 EEPROM format version 1
> >                 HW rev 2.00, Board revision B0
> >                 Serial number: 16413067  Part number: 73-2618-03
> >  
> >         PA Bay 1 Information:
> >                 Fast-Ethernet PA, 2 ports, 100BaseTX-ISL
> >                 EEPROM format version 1
> >                 HW rev 1.01, Board revision A0
> >                 Serial number: 14073635  Part number: 73-2618-02
> >  
> >         --Boot log begin--
> >  
> > Cisco Internetwork Operating System Software
> > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10), RELEASE SOFTWARE
> > (fc3)
> > Copyright (c) 1986-2004 by cisco Systems, Inc.
> > Compiled Mon 16-Aug-04 21:50 by kellythw
> > Image text-base: 0x60010AE4, data-base: 0x60560000
> >  
> >  
> >         --Boot log end--
> >  
> > Slot 11:
> >         Physical slot 11, ~physical slot 0x4, logical slot 11, CBus 1
> >         Microcode Status 0x4
> >         Master Enable, LED, WCS Loaded
> >         Board is analyzed
> >         Pending I/O Status: None
> >         EEPROM format version 1
> >         VIP2 R5K controller, HW rev 2.03, board revision A0
> >         Serial number: 19425523  Part number: 73-2167-06
> >         Test history: 0x00        RMA number: 00-00-00
> >         Flags: cisco 7000 board; 7500 compatible
> >            
> >         EEPROM contents (hex):
> >           0x20: 01 1E 02 03 01 28 68 F3 49 08 77 06 00 00 00 00
> >           0x30: 50 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> >  
> >         Slot database information:
> >         Flags: 0x4      Insertion time: 0x4380 (1w1d ago)
> >  
> >         Controller Memory Size: 128 MBytes DRAM, 8192 KBytes SRAM
> >  
> >         PA Bay 0 Information:
> >                 Fast-Ethernet PA, 2 ports, 100BaseTX-ISL
> >                 EEPROM format version 1
> >                 HW rev 2.00, Board revision B0
> >                 Serial number: 16778522  Part number: 73-2618-03
> >  
> >         --Boot log begin--
> >  
> > Cisco Internetwork Operating System Software
> > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10), RELEASE SOFTWARE
> > (fc3)
> > Copyright (c) 1986-2004 by cisco Systems, Inc.
> > Compiled Mon 16-Aug-04 21:50 by kellythw
> > Image text-base: 0x60010AE4, data-base: 0x60560000
> >  
> >  
> >         --Boot log end--
> >  
> > Slot 31 (virtual):
> >         EEPROM format version 1
> >         Chassis Interface, HW rev 1.01, board revision C0
> >         Serial number: 18002258  Part number: 73-1306-02
> >         Test history: 0x00        RMA number: 00-00-00
> >         Flags: cisco 7000 board
> >  
> >         EEPROM contents (hex):
> >           0x20: 01 10 01 01 01 12 B1 52 49 05 1A 02 00 00 00 00
> >           0x30: 60 00 00 00 FF 00 00 00 00 00 00 00 00 00 00 00
> > 
> > 
> > On Thu, 2004-09-23 at 12:03, Rodney Dunn wrote:
> > > How many routes do you have on the box.
> > > 
> > > 32M isn't nearly enough to hold the full BGP
> > > table in distributed mode.
> > > 
> > > Can you post a 'sh diag' for the box here?
> > > 
> > > I can't remember, do we even ship a GEIP with
> > > 32M of memory?
> > > 
> > > Here is the one in my lab:
> > > 
> > > Slot 0:
> > >         Physical slot 0, ~physical slot 0xF, logical slot 0, CBus 0
> > >         Microcode Status 0x4
> > >         Master Enable, LED, WCS Loaded
> > >         Board is analyzed 
> > >         Pending I/O Status: None
> > >         EEPROM format version 1
> > >         VIP4-80 RM7000 controller, FRU: VIP4-80=, HW rev 2.04, board revision A0
> > >         Serial number: 18629786  Part number: 73-3143-10
> > >         Test history: 0x00        RMA number: 00-00-00
> > >         Flags: cisco 7000 board; 7500 compatible
> > > 
> > >         EEPROM contents (hex):
> > >           0x20: 01 22 02 04 01 1C 44 9A 49 0C 47 0A 00 00 00 00
> > >           0x30: 50 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > > 
> > >         Slot database information:
> > >         Flags: 0x4      Insertion time: 0x32EC (1w3d ago)
> > > 
> > >         Controller Memory Size: 256 MBytes CPU SDRAM, 64 MBytes Packet SDRAM
> > > 
> > >         PA Bay 0 Information:
> > >                 Gigabit-Ethernet PA(Dual-Wide), 1 ports, FRU: PA-GE=
> > >                 EEPROM format version 4
> > >                 HW rev 0.02, Board revision A0
> > >                 Serial number: FAA045107SV  Part number: 73-4520-03 
> > > 
> > > 
> > > it's 256 Meg of DRAM on the GEIP (aka: vip/pa fixed combo).
> > > 
> > > On Thu, Sep 23, 2004 at 11:58:59AM -0400, Paul Stewart wrote:
> > > > > Nope.  We do ACL's in the distributed path.
> > > > > Only caveat there is named ACL's in 12.0S are not supported
> > > > > in the dCEF path.
> > > > > 
> > > > 
> > > > Ah right.. sorry my mistake...
> > > > > 
> > > > > > would that be causing it to punt down?
> > > > > 
> > > > > This doesn't have anything to do with punts.  Here dCEF isn't
> > > > > even up to the VIP in slot 8 and 9.
> > > > > 
> > > > > For some reason dCEF got disabled to those slots.
> > > > > You would have to have the logs from when it got disabled
> > > > > to know why.
> > > > > 
> > > > > How much memory are on those cards?
> > > > > 'sh diag'
> > > > > 
> > > > 32 Meg on each card..
> > > > 
> > > > > 
> > > > > You can try to re-enable it via: clear cef linecard
> > > > > 
> > > > > or do:
> > > > > 
> > > > > 
> > > > > confi t
> > > > > ip cef  <that converts everything to RSP based CEF switching
> > > > > ip cef distributed <that will turn dCEF back on for the box.
> > > > > 
> > > > > You have to get dCEF sync'd up to the cards first.
> > > > > That should say "up and sync, table-up" in sh cef linecard.
> > > > > 
> > > > 
> > > > Tried this... 
> > > > 
> > > > They still show as disabled.. weird.. must be my configuration I'm
> > > > guessing.. but not sure why...
> > > > 
> > > > Paul
> > > > 



More information about the cisco-nsp mailing list