[c-nsp] Oversubscription on 6509 blade
Stephen Cobb
scobb at telecoast.com
Mon Aug 23 15:38:19 EDT 2010
If I'm not mistaken, the original 6148-GE-TX blades support up to 1Gbps
through each of two port groups, and 1.4MB buffers for every 8 ports. The
newer 6148A-GE-TX blades support 1Gbps through six port groups, and 5.5MB
buffers for every port. There's a big difference, and if you want anything
close to wire-speed, get the 6748-GE-TX cards. Check out these release notes
and search for the two different cards:
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/release/notes/hardware.html
<http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/release/notes/hardware.html>Hope
that helps a bit-
SC
___________________________________
Stephen F. Cobb • Senior Sales Engineer
CCNA/CCDA/DCNID/ATSP
Telecoast Communications, LLC • Santa Barbara, CA
aim/yahoo telecoaststephen
On Thu, Aug 19, 2010 at 7:39 AM, Keegan Holley <keegan.holley at sungard.com>wrote:
> I would agree with scrapping the 6148 for something with bigger queues. I
> can't remember any of the model numbers off hand, but you should be able to
> find one from either a cisco account rep or simple research. As far as the
> ASIC's go packets aren't buffered there it's just for high speed switching.
> Packets are only held in the RAM and ROM queues right before transmission
> and right after reception. The RAM queues are configurable and can be
> widened or have QOS policies applied to them. The ROM queue is hardware
> based and not variable. The overruns report drops in the ROM queue or ring
> buffer. The solution here is most likely new hardware.
>
> On Thu, Aug 19, 2010 at 10:14 AM, Rich Roller <rich.roller at gmail.com>
> wrote:
>
> > Most of those ports are connected to ESX servers. Flow control is on
> > desired mode and ‘on’ for some, although I’m not sure how to verify its
> > operation outside of a sniffer. It sounds like the ultimate fix is to
> dump
> > the 6148 module for a fabric enabled line card. However, I was just
> curious
> > why the 6148 would have input errors when relative volume was low.
> >
> >
> >
> > So it’s possible that the ASIC buffer (1.4MB per 8 ports) is full? The
> only
> > command I know to verify that is something like ‘show platform hardware
> > asic’ but that’s not supported on this IOS.
> >
> >
> >
> > Also, the percentage of overruns compared to total packets input is low,
> > how much of a performance impact would that have? Thanks again for your
> > assistance.
> >
> >
> >
> > *From:* Youssef Bengelloun-Zahr [mailto:youssef at 720.fr]
> > *Sent:* Thursday, August 19, 2010 4:06 AM
> > *To:* Richard Roller
> > *Cc:* Keegan Holley; cisco-nsp at puck.nether.net
> > *Subject:* Re: [c-nsp] Oversubscription on 6509 blade
> >
> >
> >
> > Hello,
> >
> > 6148 modules are known to have limited buffers. Have you checked that ?
> >
> > As Keegan said, your hosts connected on the module may send too much
> > packets causing the queues to fill too fast.
> >
> > Regards.
> >
> > Y.
> >
> >
> > 2010/8/19 Keegan Holley <keegan.holley at sungard.com>
> >
> > Overruns have nothing to do with oversubscription. They are packets
> > dropped
> > due to the input ring being full. Basically the other end is filling the
> > queue faster than the 6509 can empty them. What's on the other end?
> Flow
> > control may help here if it's supported.
> >
> > On Wed, Aug 18, 2010 at 7:28 PM, Richard Roller <rich.roller at gmail.com
> > >wrote:
> >
> >
> > > We're seeing overruns on a 6148 module in a 6509-E switch. The overruns
> > > would be considered normal if the ASIC port range was oversubscribed,
> > > however, according to interface traffic levels, it doesn't reach the
> 1Gb
> > > peak. Output from the 'show int summ' command below shows that Rxbs =
> > > ~70Mbps while Txbs = 17Mbs. (if my math is right, and assuming 17-24 is
> > one
> > > Gb ASIC)
> > >
> > >
> > > Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL
> > > ---------------------------------------------------------------------
> > > * GigabitEthernet6/17 0 0 0 0 0 0 0 0 0
> > > * GigabitEthernet6/18 0 25082261 0 45001 18475000 4897 7468000
> > > 6161 0
> > > * GigabitEthernet6/19 0 0 0 0 0 0 0 0 0
> > > * GigabitEthernet6/20 0 24751654 0 44925 19430000 1719 1927000
> > > 2407 0
> > > * GigabitEthernet6/21 0 0 0 0 0 0 0 0 0
> > > * GigabitEthernet6/22 0 68270 0 0 15823000 1680 5358000
> 2661
> > > 0
> > > * GigabitEthernet6/23 0 0 0 0 0 0 0 0 0
> > > * GigabitEthernet6/24 0 23521824 0 44281 17227000 1616 2507000
> > > 2372 0
> > >
> > >
> > > GigabitEthernet6/20 is up, line protocol is up (connected)
> > > Hardware is C6k 1000Mb 802.3, address is 0015.faca.ee7f (bia
> > > 0015.faca.ee7f)
> > > MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
> > > reliability 255/255, txload 1/255, rxload 5/255
> > > Encapsulation ARPA, loopback not set
> > > Full-duplex, 1000Mb/s
> > > input flow-control is off, output flow-control is desired
> > > Clock mode is auto
> > > ARP type: ARPA, ARP Timeout 04:00:00
> > > Last input never, output 00:00:54, output hang never
> > > Last clearing of "show interface" counters never
> > > Input queue: 0/2000/24753578/0 (size/max/drops/flushes); Total output
> > > drops: 44925
> > > Queueing strategy: fifo
> > > Output queue: 0/40 (size/max)
> > > 5 minute input rate 21955000 bits/sec, 1941 packets/sec
> > > 5 minute output rate 2132000 bits/sec, 2292 packets/sec
> > > 13954662333 packets input, 16851357461096 bytes, 0 no buffer
> > > Received 40269073 broadcasts (409685 multicast)
> > > 0 runts, 0 giants, 0 throttles
> > > 0 input errors, 0 CRC, 0 frame, 24753578 overrun, 0 ignored
> > > 0 watchdog, 0 multicast, 0 pause input
> > > 0 input packets with dribble condition detected
> > > 24094021770 packets output, 7628772920146 bytes, 0 underruns
> > > 0 output errors, 0 collisions, 2 interface resets
> > > 0 babbles, 0 late collision, 0 deferred
> > > 0 lost carrier, 0 no carrier, 0 PAUSE output
> > > 0 output buffer failures, 0 output buffers swapped out
> > >
> > > GigabitEthernet6/22 is up, line protocol is up (connected)
> > > Hardware is C6k 1000Mb 802.3, address is 0015.faca.ee81 (bia
> > > 0015.faca.ee81)
> > > MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
> > > reliability 255/255, txload 1/255, rxload 5/255
> > > Encapsulation ARPA, loopback not set
> > > Full-duplex, 1000Mb/s
> > > input flow-control is off, output flow-control is desired
> > > Clock mode is auto
> > > ARP type: ARPA, ARP Timeout 04:00:00
> > > Last input never, output 00:00:08, output hang never
> > > Last clearing of "show interface" counters 07:30:26
> > > Input queue: 0/2000/70255/0 (size/max/drops/flushes); Total output
> > drops:
> > > 0
> > > Queueing strategy: fifo
> > > Output queue: 0/40 (size/max)
> > > 5 minute input rate 21401000 bits/sec, 2163 packets/sec
> > > 5 minute output rate 5655000 bits/sec, 2671 packets/sec
> > > 75174892 packets input, 71627016585 bytes, 0 no buffer
> > > Received 26519 broadcasts (444 multicast)
> > > 0 runts, 0 giants, 0 throttles
> > > 0 input errors, 0 CRC, 0 frame, 70255 overrun, 0 ignored
> > > 0 watchdog, 0 multicast, 0 pause input
> > > 0 input packets with dribble condition detected
> > > 95130820 packets output, 46628566836 bytes, 0 underruns
> > > 0 output errors, 0 collisions, 0 interface resets
> > > 0 babbles, 0 late collision, 0 deferred
> > > 0 lost carrier, 0 no carrier, 0 PAUSE output
> > > 0 output buffer failures, 0 output buffers swapped out
> > >
> > > Note that I recently cleared the counters on gi6/22 while all the
> others
> > > have not been cleared. The overruns steadily increment throughout the
> > day.
> > >
> > > Any idea what could be causing this? Even if I were to add up the
> traffic
> > > on
> > > all the other interfaces on slot 6, it still wouldn't reach 1Gb. Output
> > > from
> > > some other commands are listed below. Thanks.
> > >
> > > show catalyst6000
> > > chassis MAC addresses: 1024 addresses from 0014.1b1d.5400 to
> > > 0014.1b1d.57ff
> > > traffic meter = 1% Never cleared
> > > peak = 20% reached at 02:43:18 EDT Mon Jun 14 2010
> > > switching-clock: clock switchover and system reset is allowed
> > >
> > > show ver
> > > Cisco Internetwork Operating System Software
> > > IOS (tm) s72033_rp Software (s72033_rp-PK9SV-M), Version 12.2(18)SXD4,
> > > RELEASE SOFTWARE (fc1)
> > > Technical Support: http://www.cisco.com/techsupport
> > > Copyright (c) 1986-2005 by cisco Systems, Inc.
> > > Compiled Tue 22-Mar-05 15:33 by yiyan
> > > Image text-base: 0x4002100C, data-base: 0x42318000
> > >
> > > ROM: System Bootstrap, Version 12.2(17r)S2, RELEASE SOFTWARE (fc1)
> > > BOOTLDR: s72033_rp Software (s72033_rp-PK9SV-M), Version 12.2(18)SXD4,
> > > RELEASE SOFTWARE (fc1)
> > >
> > > uptime is 27 weeks, 6 days, 7 hours, 45 minutes
> > > Time since switched to active is 27 weeks, 6 days, 7 hours, 45 minutes
> > > System returned to ROM by unknown reload cause - suspect
> > > boot_data[BOOT_COUNT] 0x0, BOOT_COUNT 0, BOOTDATA 19 (SP by power-on)
> > > System restarted at 10:28:29 EDT Thu Feb 4 2010
> > > System image file is "sup-bootflash:s72033-pk9sv-mz.122-18.SXD4.bin"
> > >
> > >
> > > This product contains cryptographic features and is subject to United
> > > States and local country laws governing import, export, transfer and
> > > use. Delivery of Cisco cryptographic products does not imply
> > > third-party authority to import, export, distribute or use encryption.
> > > Importers, exporters, distributors and users are responsible for
> > > compliance with U.S. and local country laws. By using this product you
> > > agree to comply with applicable laws and regulations. If you are unable
> > > to comply with U.S. and local laws, return this product immediately.
> > >
> > > A summary of U.S. laws governing Cisco cryptographic products may be
> > found
> > > at:
> > > http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
> > >
> > > If you require further assistance please contact us by sending email to
> > > export at cisco.com.
> > >
> > > cisco WS-C6509-E (R7000) processor (revision 1.1) with 458720K/65536K
> > bytes
> > > of memory.
> > > Processor board ID SMG0917N3MZ
> > > SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
> > > Last reset from power-on
> > > X.25 software, Version 3.0.0.
> > > Bridging software.
> > > 121 Virtual Ethernet/IEEE 802.3 interface(s)
> > > 214 Gigabit Ethernet/IEEE 802.3 interface(s)
> > > 1917K bytes of non-volatile configuration memory.
> > > 8192K bytes of packet buffer memory.
> > >
> > > 65536K bytes of Flash internal SIMM (Sector size 512K).
> > > Configuration register is 0x2102
> > >
> > >
> > > sho mod
> > > Mod Ports Card Type Model
> > Serial
> > > No.
> > > --- ----- -------------------------------------- ------------------
> > > -----------
> > > 1 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX
> > > SAL091594HK
> > > 2 4 SLB Application Processor Complex WS-X6066-SLB-APC
> > > SAD093109YD
> > > 3 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX
> > > SAL09211LA8
> > > 4 48 48 port 10/100/1000mb EtherModule WS-X6148-GE-TX
> > > SAL11499266
> > > 5 2 Supervisor Engine 720 (Active) WS-SUP720-3B
> > > SAD0915018N
> > > 6 48 48 port 10/100/1000mb EtherModule WS-X6148-GE-TX
> > > SAL094865J7
> > > 7 16 16 port 1000mb GBIC ethernet WS-X6416-GBIC
> > > SAD0530058M
> > >
> > > Mod MAC addresses Hw Fw Sw
> > > Status
> > > --- ---------------------------------- ------ ------------ ------------
> > > -------
> > > 1 0013.80f5.68a8 to 0013.80f5.68d7 10.1 7.2(1) 8.3(0.156)RO
> Ok
> > > 2 0014.a9bd.20f8 to 0014.a9bd.20ff 1.8 4.2(12)
> Ok
> > > 3 0013.c473.0af8 to 0013.c473.0b27 2.2 12.2(14r)S5 12.2(18)SXD4
> Ok
> > > 4 001e.4a45.4850 to 001e.4a45.487f 7.2 7.2(1) 8.3(0.156)RO
> Ok
> > > 5 0011.5cb4.afdc to 0011.5cb4.afdf 4.3 8.1(3) 12.2(18)SXD4
> Ok
> > > 6 0015.faca.ee6c to 0015.faca.ee9b 1.1 7.2(1) 8.3(0.156)RO
> Ok
> > > 7 0003.32bd.1fc2 to 0003.32bd.1fd1 1.2 5.4(2) 8.3(0.156)RO
> Ok
> > >
> > > Mod Sub-Module Model Serial Hw
> > > Status
> > > --- --------------------------- ------------------ ------------ -------
> > > -------
> > > 3 Centralized Forwarding Card WS-F6700-CFC SAL0917A6G6 2.0
> Ok
> > > 5 Policy Feature Card 3 WS-F6K-PFC3B SAD09120A3A 2.0
> Ok
> > > 5 MSFC3 Daughterboard WS-SUP720 SAD091204G5 2.3
> Ok
> > >
> > > Mod Online Diag Status
> > > --- -------------------
> > > 1 Pass
> > > 2 Pass
> > > 3 Pass
> > > 4 Pass
> > > 5 Pass
> > > 6 Pass
> > > 7 Pass
> > > _______________________________________________
> > > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > archive at http://puck.nether.net/pipermail/cisco-nsp/
> > >
> > >
> > >
> > _______________________________________________
> > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >
> >
> >
> >
> > --
> > Youssef BENGELLOUN-ZAHR ………………………………………………
> > Ingénieur Réseaux et Télécoms
> >
> >
> > Technopole de l'Aube en Champagne - BP 601 - 10901 TROYES Cedex 9
> > Agence Paris : 6, rue Charles Floquet - 92120 MONTROUGE
> > Tel +33 (0) 825 000 720
> > Tel. direct +33 (0) 1 77 35 59 14
> > Tel. portable +33 (0) 6 22 42 63 80
> > Email ybz at 720.fr
> > ……………………………………………………………………………….....www.720.fr
> >
>
>
>
> --
>
> Keegan Holley ▪ Jr. Network Architect ▪ SunGard Availability Services ▪
> 401
> North Broad St. Philadelphia, PA 19108 ▪ (215) 446-1242 ▪
> *keegan.holley at sungard.com* <keegan.holley at sungard.com> Keeping People and
> Information Connected® ▪
> *http://www.availability.sungard.com/*<http://availability.sungard.com/>
>
> P
> *Think before you print*
>
> CONFIDENTIALITY: This e-mail (including any attachments) may contain
> confidential, proprietary and privileged information, and unauthorized
> disclosure or use is prohibited. If you received this e-mail in error,
> please notify the sender and delete this e-mail from your system.
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
More information about the cisco-nsp
mailing list