[c-nsp] Troubleshooting Lag between GigE interfaces
Paul Stewart
pauls at nexicom.net
Thu Sep 23 12:36:22 EDT 2004
Thanks Rodney...
I appreciate your help on this and hope the list has benefit from this
discussion.... this issue crept on me to be honest...
Here's some followup to prove memory as the issue:
gw-7513#if-con 8
Console or Debug [C]:
Entering CONSOLE for GEIP 8
Type "^C^C^C" or "if-quit" to end this session
%% Low on memory; try again later
%% Low on memory; try again later
%% Low on memory; try again later
%% Low on memory; try again later
%% Low on memory; try again later
%% Low on memory; try again later
Terminate DBUS IO session? [confirm]
Disconnecting from slot 8 CONSOLE after 00:00:21
gw-7513#if-con 9
Console or Debug [C]:
Entering CONSOLE for GEIP 9
Type "^C^C^C" or "if-quit" to end this session
%% Low on memory; try again later
%% Low on memory; try again later
%% Low on memory; try again later
%% Low on memory; try again later
%% Low on memory; try again later
%% Low on memory; try again later
%% Low on memory; try again later
Terminate DBUS IO session? [confirm]
Disconnecting from slot 9 CONSOLE after 00:00:09
Time to upgrade..:)
Paul
On Thu, 2004-09-23 at 12:22, Rodney Dunn wrote:
> If you have any significant number of routes
> this isn't enough memory.
>
> You can check what's free on the GEIP by
> doing:
>
>
> if-con <slot>
> sh proc mem
>
> ie:
>
> W2N-4.4-7513b#if-con 0
> Console or Debug [C]:
> Entering CONSOLE for VIP4-80 RM7000 0
> Type "^C^C^C" or "if-quit" to end this session
>
>
> VIP-Slot0>
> VIP-Slot0>en
> VIP-Slot0#sh proc mem
> Processor Pool Total: 256802816 Used: 27282600 Free: 229520216
> PCI Pool Total: 67108608 Used: 67092376 Free: 16232
>
>
> You can even get to the point where the image that has to
> be loaded on the VIP can't even load if the memory is too low.
> As you upgrade, the image gets larger because new features are
> added and bugfixes.
>
> One other thing, if anyone ever wants to run dCEF on a 75xx
> they should make sure ALL cards support it and that means
> getting rid of the old legacy ip boards (ie: FSIP).
> You need to get a VIP/serial PA combination for those links.
>
> Now that we have figured this out, even with the packets
> being switched at interrupt level by the RSP you shouldn't
> see 60 msec of latency.
>
> If you turn off dCEF on a 75xx you basically have a 72xx
> with more slots. Everything is handled by the RSP.
>
> Rodney
>
>
>
> On Thu, Sep 23, 2004 at 12:14:21PM -0400, Paul Stewart wrote:
> > Hmm... I think we're getting warmer..:)
> >
> > Here's the "sh diag":
> >
> > gw-7513#sh diagbus
> > Slot 3:
> > Physical slot 3, ~physical slot 0xC, logical slot 3, CBus 0
> > Microcode Status 0x0
> > Master Enable, LED, WCS Loaded
> > Board is analyzed
> > EEPROM format version 1
> > FSIP controller, HW rev 1.01, board revision C0
> > Serial number: 03496276 Part number: 73-1126-05
> > Test history: 0x00 RMA number: 00-00-00
> > Flags: cisco 7000 board; 7500 compatible
> >
> > EEPROM contents (hex):
> > 0x20: 01 07 01 01 00 35 59 54 49 04 66 05 00 00 00 00
> > 0x30: 60 00 00 00 FF 00 00 00 00 00 00 00 00 00 00 00
> >
> > Slot database information:
> > Flags: 0x4 Insertion time: 0x4380 (1w1d ago)
> > Slot 6:
> > EEPROM format version 1
> > Route/Switch Processor 4, HW rev 1.03, board revision C0
> > Serial number: 15308129 Part number: 73-1689-05
> > Test history: 0x00 RMA number: 00-00-00
> > Flags: cisco 7000 board; 7500 compatible
> >
> > EEPROM contents (hex):
> > 0x20: 01 1A 01 03 00 E9 95 61 49 06 99 05 00 00 00 00
> > 0x30: 60 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00
> > Slot 7:
> > EEPROM format version 1
> > Route/Switch Processor 4, HW rev 1.03, board revision C0
> > Serial number: 15308150 Part number: 73-1689-05
> > Test history: 0x00 RMA number: 00-00-00
> > Flags: cisco 7000 board; 7500 compatible
> >
> > EEPROM contents (hex):
> > 0x20: 01 1A 01 03 00 E9 95 76 49 06 99 05 00 00 00 00
> > 0x30: 60 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> > Slot 8:
> > Physical slot 8, ~physical slot 0x7, logical slot 8, CBus 1
> > Microcode Status 0x4
> > Master Enable, LED, WCS Loaded
> > Board is analyzed
> > Pending I/O Status: None
> > EEPROM format version 1
> > GEIP controller, HW rev 2.03, board revision A0
> > Serial number: 18950511 Part number: 73-2167-06
> > Test history: 0x00 RMA number: 00-00-00
> > Flags: cisco 7000 board; 7500 compatible
> >
> > EEPROM contents (hex):
> > 0x20: 01 21 02 03 01 21 29 6F 49 08 77 06 00 00 00 00
> > 0x30: 50 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> >
> > Slot database information:
> > Flags: 0x4 Insertion time: 0x4380 (1w1d ago)
> >
> > Controller Memory Size: 32 MBytes DRAM, 4096 KBytes SRAM
> >
> > PA Bay 1 Information:
> > Gigabit-Ethernet PA, 1 ports
> > EEPROM format version 1
> > HW rev 1.00, Board revision A1
> > Serial number: 17941328 Part number: 73-3144-03
> >
> > --Boot log begin--
> >
> > Cisco Internetwork Operating System Software
> > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10), RELEASE SOFTWARE
> > (fc3)
> > Copyright (c) 1986-2004 by cisco Systems, Inc.
> > Compiled Mon 16-Aug-04 21:50 by kellythw
> > Image text-base: 0x60010AE4, data-base: 0x60560000
> >
> >
> > --Boot log end--
> >
> > Slot 9:
> > Physical slot 9, ~physical slot 0x6, logical slot 9, CBus 1
> > Microcode Status 0x4
> > Master Enable, LED, WCS Loaded
> > Board is analyzed
> > Pending I/O Status: None
> > EEPROM format version 1
> > GEIP controller, HW rev 2.02, board revision C0
> > Serial number: 13943816 Part number: 73-2167-05
> > Test history: 0x00 RMA number: 00-00-00
> > Flags: cisco 7000 board; 7500 compatible
> >
> > EEPROM contents (hex):
> > 0x20: 01 21 02 02 00 D4 C4 08 49 08 77 05 00 00 00 00
> > 0x30: 60 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> >
> > Slot database information:
> > Flags: 0x4 Insertion time: 0x4380 (1w1d ago)
> >
> > Controller Memory Size: 32 MBytes DRAM, 4096 KBytes SRAM
> >
> > PA Bay 1 Information:
> > Gigabit-Ethernet PA, 1 ports
> > EEPROM format version 1
> > HW rev 1.00, Board revision A1
> > Serial number: 22598189 Part number: 73-3144-03
> >
> > --Boot log begin--
> >
> > Cisco Internetwork Operating System Software
> > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10), RELEASE SOFTWARE
> > (fc3)
> > Copyright (c) 1986-2004 by cisco Systems, Inc.
> > Compiled Mon 16-Aug-04 21:50 by kellythw
> > Image text-base: 0x60010AE4, data-base: 0x60560000
> >
> >
> > --Boot log end--
> >
> > Slot 10:
> > Physical slot 10, ~physical slot 0x5, logical slot 10, CBus 1
> > Microcode Status 0x4
> > Master Enable, LED, WCS Loaded
> > Board is analyzed
> > Pending I/O Status: None
> > EEPROM format version 1
> > VIP2 R5K controller, HW rev 2.03, board revision A0
> > Serial number: 17944577 Part number: 73-2167-06
> > Test history: 0x00 RMA number: 00-00-00
> > Flags: cisco 7000 board; 7500 compatible
> >
> > EEPROM contents (hex):
> > 0x20: 01 1E 02 03 01 11 D0 01 49 08 77 06 00 00 00 00
> > 0x30: 50 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> >
> > Slot database information:
> > Flags: 0x4 Insertion time: 0x4380 (1w1d ago)
> >
> > Controller Memory Size: 128 MBytes DRAM, 8192 KBytes SRAM
> >
> > PA Bay 0 Information:
> > Fast-Ethernet PA, 2 ports, 100BaseTX-ISL
> > EEPROM format version 1
> > HW rev 2.00, Board revision B0
> > Serial number: 16413067 Part number: 73-2618-03
> >
> > PA Bay 1 Information:
> > Fast-Ethernet PA, 2 ports, 100BaseTX-ISL
> > EEPROM format version 1
> > HW rev 1.01, Board revision A0
> > Serial number: 14073635 Part number: 73-2618-02
> >
> > --Boot log begin--
> >
> > Cisco Internetwork Operating System Software
> > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10), RELEASE SOFTWARE
> > (fc3)
> > Copyright (c) 1986-2004 by cisco Systems, Inc.
> > Compiled Mon 16-Aug-04 21:50 by kellythw
> > Image text-base: 0x60010AE4, data-base: 0x60560000
> >
> >
> > --Boot log end--
> >
> > Slot 11:
> > Physical slot 11, ~physical slot 0x4, logical slot 11, CBus 1
> > Microcode Status 0x4
> > Master Enable, LED, WCS Loaded
> > Board is analyzed
> > Pending I/O Status: None
> > EEPROM format version 1
> > VIP2 R5K controller, HW rev 2.03, board revision A0
> > Serial number: 19425523 Part number: 73-2167-06
> > Test history: 0x00 RMA number: 00-00-00
> > Flags: cisco 7000 board; 7500 compatible
> >
> > EEPROM contents (hex):
> > 0x20: 01 1E 02 03 01 28 68 F3 49 08 77 06 00 00 00 00
> > 0x30: 50 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> >
> > Slot database information:
> > Flags: 0x4 Insertion time: 0x4380 (1w1d ago)
> >
> > Controller Memory Size: 128 MBytes DRAM, 8192 KBytes SRAM
> >
> > PA Bay 0 Information:
> > Fast-Ethernet PA, 2 ports, 100BaseTX-ISL
> > EEPROM format version 1
> > HW rev 2.00, Board revision B0
> > Serial number: 16778522 Part number: 73-2618-03
> >
> > --Boot log begin--
> >
> > Cisco Internetwork Operating System Software
> > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10), RELEASE SOFTWARE
> > (fc3)
> > Copyright (c) 1986-2004 by cisco Systems, Inc.
> > Compiled Mon 16-Aug-04 21:50 by kellythw
> > Image text-base: 0x60010AE4, data-base: 0x60560000
> >
> >
> > --Boot log end--
> >
> > Slot 31 (virtual):
> > EEPROM format version 1
> > Chassis Interface, HW rev 1.01, board revision C0
> > Serial number: 18002258 Part number: 73-1306-02
> > Test history: 0x00 RMA number: 00-00-00
> > Flags: cisco 7000 board
> >
> > EEPROM contents (hex):
> > 0x20: 01 10 01 01 01 12 B1 52 49 05 1A 02 00 00 00 00
> > 0x30: 60 00 00 00 FF 00 00 00 00 00 00 00 00 00 00 00
> >
> >
> > On Thu, 2004-09-23 at 12:03, Rodney Dunn wrote:
> > > How many routes do you have on the box.
> > >
> > > 32M isn't nearly enough to hold the full BGP
> > > table in distributed mode.
> > >
> > > Can you post a 'sh diag' for the box here?
> > >
> > > I can't remember, do we even ship a GEIP with
> > > 32M of memory?
> > >
> > > Here is the one in my lab:
> > >
> > > Slot 0:
> > > Physical slot 0, ~physical slot 0xF, logical slot 0, CBus 0
> > > Microcode Status 0x4
> > > Master Enable, LED, WCS Loaded
> > > Board is analyzed
> > > Pending I/O Status: None
> > > EEPROM format version 1
> > > VIP4-80 RM7000 controller, FRU: VIP4-80=, HW rev 2.04, board revision A0
> > > Serial number: 18629786 Part number: 73-3143-10
> > > Test history: 0x00 RMA number: 00-00-00
> > > Flags: cisco 7000 board; 7500 compatible
> > >
> > > EEPROM contents (hex):
> > > 0x20: 01 22 02 04 01 1C 44 9A 49 0C 47 0A 00 00 00 00
> > > 0x30: 50 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > >
> > > Slot database information:
> > > Flags: 0x4 Insertion time: 0x32EC (1w3d ago)
> > >
> > > Controller Memory Size: 256 MBytes CPU SDRAM, 64 MBytes Packet SDRAM
> > >
> > > PA Bay 0 Information:
> > > Gigabit-Ethernet PA(Dual-Wide), 1 ports, FRU: PA-GE=
> > > EEPROM format version 4
> > > HW rev 0.02, Board revision A0
> > > Serial number: FAA045107SV Part number: 73-4520-03
> > >
> > >
> > > it's 256 Meg of DRAM on the GEIP (aka: vip/pa fixed combo).
> > >
> > > On Thu, Sep 23, 2004 at 11:58:59AM -0400, Paul Stewart wrote:
> > > > > Nope. We do ACL's in the distributed path.
> > > > > Only caveat there is named ACL's in 12.0S are not supported
> > > > > in the dCEF path.
> > > > >
> > > >
> > > > Ah right.. sorry my mistake...
> > > > >
> > > > > > would that be causing it to punt down?
> > > > >
> > > > > This doesn't have anything to do with punts. Here dCEF isn't
> > > > > even up to the VIP in slot 8 and 9.
> > > > >
> > > > > For some reason dCEF got disabled to those slots.
> > > > > You would have to have the logs from when it got disabled
> > > > > to know why.
> > > > >
> > > > > How much memory are on those cards?
> > > > > 'sh diag'
> > > > >
> > > > 32 Meg on each card..
> > > >
> > > > >
> > > > > You can try to re-enable it via: clear cef linecard
> > > > >
> > > > > or do:
> > > > >
> > > > >
> > > > > confi t
> > > > > ip cef <that converts everything to RSP based CEF switching
> > > > > ip cef distributed <that will turn dCEF back on for the box.
> > > > >
> > > > > You have to get dCEF sync'd up to the cards first.
> > > > > That should say "up and sync, table-up" in sh cef linecard.
> > > > >
> > > >
> > > > Tried this...
> > > >
> > > > They still show as disabled.. weird.. must be my configuration I'm
> > > > guessing.. but not sure why...
> > > >
> > > > Paul
> > > >
More information about the cisco-nsp
mailing list