[c-nsp] Troubleshooting Lag between GigE interfaces
Rodney Dunn
rodunn at cisco.com
Thu Sep 23 12:39:13 EDT 2004
:)
I'd start there and get that FSIP replaced with
a VIP/serial PA.
See if those two things fix the latency problem.
On Thu, Sep 23, 2004 at 12:36:22PM -0400, Paul Stewart wrote:
> Thanks Rodney...
>
> I appreciate your help on this and hope the list has benefit from this
> discussion.... this issue crept on me to be honest...
>
> Here's some followup to prove memory as the issue:
>
> gw-7513#if-con 8
> Console or Debug [C]:
> Entering CONSOLE for GEIP 8
> Type "^C^C^C" or "if-quit" to end this session
>
>
> %% Low on memory; try again later
>
> %% Low on memory; try again later
>
> %% Low on memory; try again later
>
> %% Low on memory; try again later
>
> %% Low on memory; try again later
>
> %% Low on memory; try again later
>
> Terminate DBUS IO session? [confirm]
>
> Disconnecting from slot 8 CONSOLE after 00:00:21
>
> gw-7513#if-con 9
> Console or Debug [C]:
> Entering CONSOLE for GEIP 9
> Type "^C^C^C" or "if-quit" to end this session
>
>
> %% Low on memory; try again later
>
> %% Low on memory; try again later
>
> %% Low on memory; try again later
>
> %% Low on memory; try again later
>
> %% Low on memory; try again later
>
> %% Low on memory; try again later
>
> %% Low on memory; try again later
>
> Terminate DBUS IO session? [confirm]
>
> Disconnecting from slot 9 CONSOLE after 00:00:09
>
>
> Time to upgrade..:)
>
> Paul
>
>
> On Thu, 2004-09-23 at 12:22, Rodney Dunn wrote:
> > If you have any significant number of routes
> > this isn't enough memory.
> >
> > You can check what's free on the GEIP by
> > doing:
> >
> >
> > if-con <slot>
> > sh proc mem
> >
> > ie:
> >
> > W2N-4.4-7513b#if-con 0
> > Console or Debug [C]:
> > Entering CONSOLE for VIP4-80 RM7000 0
> > Type "^C^C^C" or "if-quit" to end this session
> >
> >
> > VIP-Slot0>
> > VIP-Slot0>en
> > VIP-Slot0#sh proc mem
> > Processor Pool Total: 256802816 Used: 27282600 Free: 229520216
> > PCI Pool Total: 67108608 Used: 67092376 Free: 16232
> >
> >
> > You can even get to the point where the image that has to
> > be loaded on the VIP can't even load if the memory is too low.
> > As you upgrade, the image gets larger because new features are
> > added and bugfixes.
> >
> > One other thing, if anyone ever wants to run dCEF on a 75xx
> > they should make sure ALL cards support it and that means
> > getting rid of the old legacy ip boards (ie: FSIP).
> > You need to get a VIP/serial PA combination for those links.
> >
> > Now that we have figured this out, even with the packets
> > being switched at interrupt level by the RSP you shouldn't
> > see 60 msec of latency.
> >
> > If you turn off dCEF on a 75xx you basically have a 72xx
> > with more slots. Everything is handled by the RSP.
> >
> > Rodney
> >
> >
> >
> > On Thu, Sep 23, 2004 at 12:14:21PM -0400, Paul Stewart wrote:
> > > Hmm... I think we're getting warmer..:)
> > >
> > > Here's the "sh diag":
> > >
> > > gw-7513#sh diagbus
> > > Slot 3:
> > > Physical slot 3, ~physical slot 0xC, logical slot 3, CBus 0
> > > Microcode Status 0x0
> > > Master Enable, LED, WCS Loaded
> > > Board is analyzed
> > > EEPROM format version 1
> > > FSIP controller, HW rev 1.01, board revision C0
> > > Serial number: 03496276 Part number: 73-1126-05
> > > Test history: 0x00 RMA number: 00-00-00
> > > Flags: cisco 7000 board; 7500 compatible
> > >
> > > EEPROM contents (hex):
> > > 0x20: 01 07 01 01 00 35 59 54 49 04 66 05 00 00 00 00
> > > 0x30: 60 00 00 00 FF 00 00 00 00 00 00 00 00 00 00 00
> > >
> > > Slot database information:
> > > Flags: 0x4 Insertion time: 0x4380 (1w1d ago)
> > > Slot 6:
> > > EEPROM format version 1
> > > Route/Switch Processor 4, HW rev 1.03, board revision C0
> > > Serial number: 15308129 Part number: 73-1689-05
> > > Test history: 0x00 RMA number: 00-00-00
> > > Flags: cisco 7000 board; 7500 compatible
> > >
> > > EEPROM contents (hex):
> > > 0x20: 01 1A 01 03 00 E9 95 61 49 06 99 05 00 00 00 00
> > > 0x30: 60 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00
> > > Slot 7:
> > > EEPROM format version 1
> > > Route/Switch Processor 4, HW rev 1.03, board revision C0
> > > Serial number: 15308150 Part number: 73-1689-05
> > > Test history: 0x00 RMA number: 00-00-00
> > > Flags: cisco 7000 board; 7500 compatible
> > >
> > > EEPROM contents (hex):
> > > 0x20: 01 1A 01 03 00 E9 95 76 49 06 99 05 00 00 00 00
> > > 0x30: 60 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> > > Slot 8:
> > > Physical slot 8, ~physical slot 0x7, logical slot 8, CBus 1
> > > Microcode Status 0x4
> > > Master Enable, LED, WCS Loaded
> > > Board is analyzed
> > > Pending I/O Status: None
> > > EEPROM format version 1
> > > GEIP controller, HW rev 2.03, board revision A0
> > > Serial number: 18950511 Part number: 73-2167-06
> > > Test history: 0x00 RMA number: 00-00-00
> > > Flags: cisco 7000 board; 7500 compatible
> > >
> > > EEPROM contents (hex):
> > > 0x20: 01 21 02 03 01 21 29 6F 49 08 77 06 00 00 00 00
> > > 0x30: 50 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> > >
> > > Slot database information:
> > > Flags: 0x4 Insertion time: 0x4380 (1w1d ago)
> > >
> > > Controller Memory Size: 32 MBytes DRAM, 4096 KBytes SRAM
> > >
> > > PA Bay 1 Information:
> > > Gigabit-Ethernet PA, 1 ports
> > > EEPROM format version 1
> > > HW rev 1.00, Board revision A1
> > > Serial number: 17941328 Part number: 73-3144-03
> > >
> > > --Boot log begin--
> > >
> > > Cisco Internetwork Operating System Software
> > > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10), RELEASE SOFTWARE
> > > (fc3)
> > > Copyright (c) 1986-2004 by cisco Systems, Inc.
> > > Compiled Mon 16-Aug-04 21:50 by kellythw
> > > Image text-base: 0x60010AE4, data-base: 0x60560000
> > >
> > >
> > > --Boot log end--
> > >
> > > Slot 9:
> > > Physical slot 9, ~physical slot 0x6, logical slot 9, CBus 1
> > > Microcode Status 0x4
> > > Master Enable, LED, WCS Loaded
> > > Board is analyzed
> > > Pending I/O Status: None
> > > EEPROM format version 1
> > > GEIP controller, HW rev 2.02, board revision C0
> > > Serial number: 13943816 Part number: 73-2167-05
> > > Test history: 0x00 RMA number: 00-00-00
> > > Flags: cisco 7000 board; 7500 compatible
> > >
> > > EEPROM contents (hex):
> > > 0x20: 01 21 02 02 00 D4 C4 08 49 08 77 05 00 00 00 00
> > > 0x30: 60 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> > >
> > > Slot database information:
> > > Flags: 0x4 Insertion time: 0x4380 (1w1d ago)
> > >
> > > Controller Memory Size: 32 MBytes DRAM, 4096 KBytes SRAM
> > >
> > > PA Bay 1 Information:
> > > Gigabit-Ethernet PA, 1 ports
> > > EEPROM format version 1
> > > HW rev 1.00, Board revision A1
> > > Serial number: 22598189 Part number: 73-3144-03
> > >
> > > --Boot log begin--
> > >
> > > Cisco Internetwork Operating System Software
> > > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10), RELEASE SOFTWARE
> > > (fc3)
> > > Copyright (c) 1986-2004 by cisco Systems, Inc.
> > > Compiled Mon 16-Aug-04 21:50 by kellythw
> > > Image text-base: 0x60010AE4, data-base: 0x60560000
> > >
> > >
> > > --Boot log end--
> > >
> > > Slot 10:
> > > Physical slot 10, ~physical slot 0x5, logical slot 10, CBus 1
> > > Microcode Status 0x4
> > > Master Enable, LED, WCS Loaded
> > > Board is analyzed
> > > Pending I/O Status: None
> > > EEPROM format version 1
> > > VIP2 R5K controller, HW rev 2.03, board revision A0
> > > Serial number: 17944577 Part number: 73-2167-06
> > > Test history: 0x00 RMA number: 00-00-00
> > > Flags: cisco 7000 board; 7500 compatible
> > >
> > > EEPROM contents (hex):
> > > 0x20: 01 1E 02 03 01 11 D0 01 49 08 77 06 00 00 00 00
> > > 0x30: 50 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> > >
> > > Slot database information:
> > > Flags: 0x4 Insertion time: 0x4380 (1w1d ago)
> > >
> > > Controller Memory Size: 128 MBytes DRAM, 8192 KBytes SRAM
> > >
> > > PA Bay 0 Information:
> > > Fast-Ethernet PA, 2 ports, 100BaseTX-ISL
> > > EEPROM format version 1
> > > HW rev 2.00, Board revision B0
> > > Serial number: 16413067 Part number: 73-2618-03
> > >
> > > PA Bay 1 Information:
> > > Fast-Ethernet PA, 2 ports, 100BaseTX-ISL
> > > EEPROM format version 1
> > > HW rev 1.01, Board revision A0
> > > Serial number: 14073635 Part number: 73-2618-02
> > >
> > > --Boot log begin--
> > >
> > > Cisco Internetwork Operating System Software
> > > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10), RELEASE SOFTWARE
> > > (fc3)
> > > Copyright (c) 1986-2004 by cisco Systems, Inc.
> > > Compiled Mon 16-Aug-04 21:50 by kellythw
> > > Image text-base: 0x60010AE4, data-base: 0x60560000
> > >
> > >
> > > --Boot log end--
> > >
> > > Slot 11:
> > > Physical slot 11, ~physical slot 0x4, logical slot 11, CBus 1
> > > Microcode Status 0x4
> > > Master Enable, LED, WCS Loaded
> > > Board is analyzed
> > > Pending I/O Status: None
> > > EEPROM format version 1
> > > VIP2 R5K controller, HW rev 2.03, board revision A0
> > > Serial number: 19425523 Part number: 73-2167-06
> > > Test history: 0x00 RMA number: 00-00-00
> > > Flags: cisco 7000 board; 7500 compatible
> > >
> > > EEPROM contents (hex):
> > > 0x20: 01 1E 02 03 01 28 68 F3 49 08 77 06 00 00 00 00
> > > 0x30: 50 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> > >
> > > Slot database information:
> > > Flags: 0x4 Insertion time: 0x4380 (1w1d ago)
> > >
> > > Controller Memory Size: 128 MBytes DRAM, 8192 KBytes SRAM
> > >
> > > PA Bay 0 Information:
> > > Fast-Ethernet PA, 2 ports, 100BaseTX-ISL
> > > EEPROM format version 1
> > > HW rev 2.00, Board revision B0
> > > Serial number: 16778522 Part number: 73-2618-03
> > >
> > > --Boot log begin--
> > >
> > > Cisco Internetwork Operating System Software
> > > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10), RELEASE SOFTWARE
> > > (fc3)
> > > Copyright (c) 1986-2004 by cisco Systems, Inc.
> > > Compiled Mon 16-Aug-04 21:50 by kellythw
> > > Image text-base: 0x60010AE4, data-base: 0x60560000
> > >
> > >
> > > --Boot log end--
> > >
> > > Slot 31 (virtual):
> > > EEPROM format version 1
> > > Chassis Interface, HW rev 1.01, board revision C0
> > > Serial number: 18002258 Part number: 73-1306-02
> > > Test history: 0x00 RMA number: 00-00-00
> > > Flags: cisco 7000 board
> > >
> > > EEPROM contents (hex):
> > > 0x20: 01 10 01 01 01 12 B1 52 49 05 1A 02 00 00 00 00
> > > 0x30: 60 00 00 00 FF 00 00 00 00 00 00 00 00 00 00 00
> > >
> > >
> > > On Thu, 2004-09-23 at 12:03, Rodney Dunn wrote:
> > > > How many routes do you have on the box.
> > > >
> > > > 32M isn't nearly enough to hold the full BGP
> > > > table in distributed mode.
> > > >
> > > > Can you post a 'sh diag' for the box here?
> > > >
> > > > I can't remember, do we even ship a GEIP with
> > > > 32M of memory?
> > > >
> > > > Here is the one in my lab:
> > > >
> > > > Slot 0:
> > > > Physical slot 0, ~physical slot 0xF, logical slot 0, CBus 0
> > > > Microcode Status 0x4
> > > > Master Enable, LED, WCS Loaded
> > > > Board is analyzed
> > > > Pending I/O Status: None
> > > > EEPROM format version 1
> > > > VIP4-80 RM7000 controller, FRU: VIP4-80=, HW rev 2.04, board revision A0
> > > > Serial number: 18629786 Part number: 73-3143-10
> > > > Test history: 0x00 RMA number: 00-00-00
> > > > Flags: cisco 7000 board; 7500 compatible
> > > >
> > > > EEPROM contents (hex):
> > > > 0x20: 01 22 02 04 01 1C 44 9A 49 0C 47 0A 00 00 00 00
> > > > 0x30: 50 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > > >
> > > > Slot database information:
> > > > Flags: 0x4 Insertion time: 0x32EC (1w3d ago)
> > > >
> > > > Controller Memory Size: 256 MBytes CPU SDRAM, 64 MBytes Packet SDRAM
> > > >
> > > > PA Bay 0 Information:
> > > > Gigabit-Ethernet PA(Dual-Wide), 1 ports, FRU: PA-GE=
> > > > EEPROM format version 4
> > > > HW rev 0.02, Board revision A0
> > > > Serial number: FAA045107SV Part number: 73-4520-03
> > > >
> > > >
> > > > it's 256 Meg of DRAM on the GEIP (aka: vip/pa fixed combo).
> > > >
> > > > On Thu, Sep 23, 2004 at 11:58:59AM -0400, Paul Stewart wrote:
> > > > > > Nope. We do ACL's in the distributed path.
> > > > > > Only caveat there is named ACL's in 12.0S are not supported
> > > > > > in the dCEF path.
> > > > > >
> > > > >
> > > > > Ah right.. sorry my mistake...
> > > > > >
> > > > > > > would that be causing it to punt down?
> > > > > >
> > > > > > This doesn't have anything to do with punts. Here dCEF isn't
> > > > > > even up to the VIP in slot 8 and 9.
> > > > > >
> > > > > > For some reason dCEF got disabled to those slots.
> > > > > > You would have to have the logs from when it got disabled
> > > > > > to know why.
> > > > > >
> > > > > > How much memory are on those cards?
> > > > > > 'sh diag'
> > > > > >
> > > > > 32 Meg on each card..
> > > > >
> > > > > >
> > > > > > You can try to re-enable it via: clear cef linecard
> > > > > >
> > > > > > or do:
> > > > > >
> > > > > >
> > > > > > confi t
> > > > > > ip cef <that converts everything to RSP based CEF switching
> > > > > > ip cef distributed <that will turn dCEF back on for the box.
> > > > > >
> > > > > > You have to get dCEF sync'd up to the cards first.
> > > > > > That should say "up and sync, table-up" in sh cef linecard.
> > > > > >
> > > > >
> > > > > Tried this...
> > > > >
> > > > > They still show as disabled.. weird.. must be my configuration I'm
> > > > > guessing.. but not sure why...
> > > > >
> > > > > Paul
> > > > >
More information about the cisco-nsp
mailing list