[c-nsp] Oversubscription on 6509 blade
Keegan Holley
keegan.holley at sungard.com
Thu Aug 19 10:39:30 EDT 2010
I would agree with scrapping the 6148 for something with bigger queues. I
can't remember any of the model numbers off hand, but you should be able to
find one from either a cisco account rep or simple research. As far as the
ASIC's go packets aren't buffered there it's just for high speed switching.
Packets are only held in the RAM and ROM queues right before transmission
and right after reception. The RAM queues are configurable and can be
widened or have QOS policies applied to them. The ROM queue is hardware
based and not variable. The overruns report drops in the ROM queue or ring
buffer. The solution here is most likely new hardware.
On Thu, Aug 19, 2010 at 10:14 AM, Rich Roller <rich.roller at gmail.com> wrote:
> Most of those ports are connected to ESX servers. Flow control is on
> desired mode and ‘on’ for some, although I’m not sure how to verify its
> operation outside of a sniffer. It sounds like the ultimate fix is to dump
> the 6148 module for a fabric enabled line card. However, I was just curious
> why the 6148 would have input errors when relative volume was low.
>
>
>
> So it’s possible that the ASIC buffer (1.4MB per 8 ports) is full? The only
> command I know to verify that is something like ‘show platform hardware
> asic’ but that’s not supported on this IOS.
>
>
>
> Also, the percentage of overruns compared to total packets input is low,
> how much of a performance impact would that have? Thanks again for your
> assistance.
>
>
>
> *From:* Youssef Bengelloun-Zahr [mailto:youssef at 720.fr]
> *Sent:* Thursday, August 19, 2010 4:06 AM
> *To:* Richard Roller
> *Cc:* Keegan Holley; cisco-nsp at puck.nether.net
> *Subject:* Re: [c-nsp] Oversubscription on 6509 blade
>
>
>
> Hello,
>
> 6148 modules are known to have limited buffers. Have you checked that ?
>
> As Keegan said, your hosts connected on the module may send too much
> packets causing the queues to fill too fast.
>
> Regards.
>
> Y.
>
>
> 2010/8/19 Keegan Holley <keegan.holley at sungard.com>
>
> Overruns have nothing to do with oversubscription. They are packets
> dropped
> due to the input ring being full. Basically the other end is filling the
> queue faster than the 6509 can empty them. What's on the other end? Flow
> control may help here if it's supported.
>
> On Wed, Aug 18, 2010 at 7:28 PM, Richard Roller <rich.roller at gmail.com
> >wrote:
>
>
> > We're seeing overruns on a 6148 module in a 6509-E switch. The overruns
> > would be considered normal if the ASIC port range was oversubscribed,
> > however, according to interface traffic levels, it doesn't reach the 1Gb
> > peak. Output from the 'show int summ' command below shows that Rxbs =
> > ~70Mbps while Txbs = 17Mbs. (if my math is right, and assuming 17-24 is
> one
> > Gb ASIC)
> >
> >
> > Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL
> > ---------------------------------------------------------------------
> > * GigabitEthernet6/17 0 0 0 0 0 0 0 0 0
> > * GigabitEthernet6/18 0 25082261 0 45001 18475000 4897 7468000
> > 6161 0
> > * GigabitEthernet6/19 0 0 0 0 0 0 0 0 0
> > * GigabitEthernet6/20 0 24751654 0 44925 19430000 1719 1927000
> > 2407 0
> > * GigabitEthernet6/21 0 0 0 0 0 0 0 0 0
> > * GigabitEthernet6/22 0 68270 0 0 15823000 1680 5358000 2661
> > 0
> > * GigabitEthernet6/23 0 0 0 0 0 0 0 0 0
> > * GigabitEthernet6/24 0 23521824 0 44281 17227000 1616 2507000
> > 2372 0
> >
> >
> > GigabitEthernet6/20 is up, line protocol is up (connected)
> > Hardware is C6k 1000Mb 802.3, address is 0015.faca.ee7f (bia
> > 0015.faca.ee7f)
> > MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
> > reliability 255/255, txload 1/255, rxload 5/255
> > Encapsulation ARPA, loopback not set
> > Full-duplex, 1000Mb/s
> > input flow-control is off, output flow-control is desired
> > Clock mode is auto
> > ARP type: ARPA, ARP Timeout 04:00:00
> > Last input never, output 00:00:54, output hang never
> > Last clearing of "show interface" counters never
> > Input queue: 0/2000/24753578/0 (size/max/drops/flushes); Total output
> > drops: 44925
> > Queueing strategy: fifo
> > Output queue: 0/40 (size/max)
> > 5 minute input rate 21955000 bits/sec, 1941 packets/sec
> > 5 minute output rate 2132000 bits/sec, 2292 packets/sec
> > 13954662333 packets input, 16851357461096 bytes, 0 no buffer
> > Received 40269073 broadcasts (409685 multicast)
> > 0 runts, 0 giants, 0 throttles
> > 0 input errors, 0 CRC, 0 frame, 24753578 overrun, 0 ignored
> > 0 watchdog, 0 multicast, 0 pause input
> > 0 input packets with dribble condition detected
> > 24094021770 packets output, 7628772920146 bytes, 0 underruns
> > 0 output errors, 0 collisions, 2 interface resets
> > 0 babbles, 0 late collision, 0 deferred
> > 0 lost carrier, 0 no carrier, 0 PAUSE output
> > 0 output buffer failures, 0 output buffers swapped out
> >
> > GigabitEthernet6/22 is up, line protocol is up (connected)
> > Hardware is C6k 1000Mb 802.3, address is 0015.faca.ee81 (bia
> > 0015.faca.ee81)
> > MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
> > reliability 255/255, txload 1/255, rxload 5/255
> > Encapsulation ARPA, loopback not set
> > Full-duplex, 1000Mb/s
> > input flow-control is off, output flow-control is desired
> > Clock mode is auto
> > ARP type: ARPA, ARP Timeout 04:00:00
> > Last input never, output 00:00:08, output hang never
> > Last clearing of "show interface" counters 07:30:26
> > Input queue: 0/2000/70255/0 (size/max/drops/flushes); Total output
> drops:
> > 0
> > Queueing strategy: fifo
> > Output queue: 0/40 (size/max)
> > 5 minute input rate 21401000 bits/sec, 2163 packets/sec
> > 5 minute output rate 5655000 bits/sec, 2671 packets/sec
> > 75174892 packets input, 71627016585 bytes, 0 no buffer
> > Received 26519 broadcasts (444 multicast)
> > 0 runts, 0 giants, 0 throttles
> > 0 input errors, 0 CRC, 0 frame, 70255 overrun, 0 ignored
> > 0 watchdog, 0 multicast, 0 pause input
> > 0 input packets with dribble condition detected
> > 95130820 packets output, 46628566836 bytes, 0 underruns
> > 0 output errors, 0 collisions, 0 interface resets
> > 0 babbles, 0 late collision, 0 deferred
> > 0 lost carrier, 0 no carrier, 0 PAUSE output
> > 0 output buffer failures, 0 output buffers swapped out
> >
> > Note that I recently cleared the counters on gi6/22 while all the others
> > have not been cleared. The overruns steadily increment throughout the
> day.
> >
> > Any idea what could be causing this? Even if I were to add up the traffic
> > on
> > all the other interfaces on slot 6, it still wouldn't reach 1Gb. Output
> > from
> > some other commands are listed below. Thanks.
> >
> > show catalyst6000
> > chassis MAC addresses: 1024 addresses from 0014.1b1d.5400 to
> > 0014.1b1d.57ff
> > traffic meter = 1% Never cleared
> > peak = 20% reached at 02:43:18 EDT Mon Jun 14 2010
> > switching-clock: clock switchover and system reset is allowed
> >
> > show ver
> > Cisco Internetwork Operating System Software
> > IOS (tm) s72033_rp Software (s72033_rp-PK9SV-M), Version 12.2(18)SXD4,
> > RELEASE SOFTWARE (fc1)
> > Technical Support: http://www.cisco.com/techsupport
> > Copyright (c) 1986-2005 by cisco Systems, Inc.
> > Compiled Tue 22-Mar-05 15:33 by yiyan
> > Image text-base: 0x4002100C, data-base: 0x42318000
> >
> > ROM: System Bootstrap, Version 12.2(17r)S2, RELEASE SOFTWARE (fc1)
> > BOOTLDR: s72033_rp Software (s72033_rp-PK9SV-M), Version 12.2(18)SXD4,
> > RELEASE SOFTWARE (fc1)
> >
> > uptime is 27 weeks, 6 days, 7 hours, 45 minutes
> > Time since switched to active is 27 weeks, 6 days, 7 hours, 45 minutes
> > System returned to ROM by unknown reload cause - suspect
> > boot_data[BOOT_COUNT] 0x0, BOOT_COUNT 0, BOOTDATA 19 (SP by power-on)
> > System restarted at 10:28:29 EDT Thu Feb 4 2010
> > System image file is "sup-bootflash:s72033-pk9sv-mz.122-18.SXD4.bin"
> >
> >
> > This product contains cryptographic features and is subject to United
> > States and local country laws governing import, export, transfer and
> > use. Delivery of Cisco cryptographic products does not imply
> > third-party authority to import, export, distribute or use encryption.
> > Importers, exporters, distributors and users are responsible for
> > compliance with U.S. and local country laws. By using this product you
> > agree to comply with applicable laws and regulations. If you are unable
> > to comply with U.S. and local laws, return this product immediately.
> >
> > A summary of U.S. laws governing Cisco cryptographic products may be
> found
> > at:
> > http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
> >
> > If you require further assistance please contact us by sending email to
> > export at cisco.com.
> >
> > cisco WS-C6509-E (R7000) processor (revision 1.1) with 458720K/65536K
> bytes
> > of memory.
> > Processor board ID SMG0917N3MZ
> > SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
> > Last reset from power-on
> > X.25 software, Version 3.0.0.
> > Bridging software.
> > 121 Virtual Ethernet/IEEE 802.3 interface(s)
> > 214 Gigabit Ethernet/IEEE 802.3 interface(s)
> > 1917K bytes of non-volatile configuration memory.
> > 8192K bytes of packet buffer memory.
> >
> > 65536K bytes of Flash internal SIMM (Sector size 512K).
> > Configuration register is 0x2102
> >
> >
> > sho mod
> > Mod Ports Card Type Model
> Serial
> > No.
> > --- ----- -------------------------------------- ------------------
> > -----------
> > 1 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX
> > SAL091594HK
> > 2 4 SLB Application Processor Complex WS-X6066-SLB-APC
> > SAD093109YD
> > 3 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX
> > SAL09211LA8
> > 4 48 48 port 10/100/1000mb EtherModule WS-X6148-GE-TX
> > SAL11499266
> > 5 2 Supervisor Engine 720 (Active) WS-SUP720-3B
> > SAD0915018N
> > 6 48 48 port 10/100/1000mb EtherModule WS-X6148-GE-TX
> > SAL094865J7
> > 7 16 16 port 1000mb GBIC ethernet WS-X6416-GBIC
> > SAD0530058M
> >
> > Mod MAC addresses Hw Fw Sw
> > Status
> > --- ---------------------------------- ------ ------------ ------------
> > -------
> > 1 0013.80f5.68a8 to 0013.80f5.68d7 10.1 7.2(1) 8.3(0.156)RO Ok
> > 2 0014.a9bd.20f8 to 0014.a9bd.20ff 1.8 4.2(12) Ok
> > 3 0013.c473.0af8 to 0013.c473.0b27 2.2 12.2(14r)S5 12.2(18)SXD4 Ok
> > 4 001e.4a45.4850 to 001e.4a45.487f 7.2 7.2(1) 8.3(0.156)RO Ok
> > 5 0011.5cb4.afdc to 0011.5cb4.afdf 4.3 8.1(3) 12.2(18)SXD4 Ok
> > 6 0015.faca.ee6c to 0015.faca.ee9b 1.1 7.2(1) 8.3(0.156)RO Ok
> > 7 0003.32bd.1fc2 to 0003.32bd.1fd1 1.2 5.4(2) 8.3(0.156)RO Ok
> >
> > Mod Sub-Module Model Serial Hw
> > Status
> > --- --------------------------- ------------------ ------------ -------
> > -------
> > 3 Centralized Forwarding Card WS-F6700-CFC SAL0917A6G6 2.0 Ok
> > 5 Policy Feature Card 3 WS-F6K-PFC3B SAD09120A3A 2.0 Ok
> > 5 MSFC3 Daughterboard WS-SUP720 SAD091204G5 2.3 Ok
> >
> > Mod Online Diag Status
> > --- -------------------
> > 1 Pass
> > 2 Pass
> > 3 Pass
> > 4 Pass
> > 5 Pass
> > 6 Pass
> > 7 Pass
> > _______________________________________________
> > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >
> >
> >
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>
>
>
> --
> Youssef BENGELLOUN-ZAHR ………………………………………………
> Ingénieur Réseaux et Télécoms
>
>
> Technopole de l'Aube en Champagne - BP 601 - 10901 TROYES Cedex 9
> Agence Paris : 6, rue Charles Floquet - 92120 MONTROUGE
> Tel +33 (0) 825 000 720
> Tel. direct +33 (0) 1 77 35 59 14
> Tel. portable +33 (0) 6 22 42 63 80
> Email ybz at 720.fr
> ……………………………………………………………………………….....www.720.fr
>
--
Keegan Holley ▪ Jr. Network Architect ▪ SunGard Availability Services ▪ 401
North Broad St. Philadelphia, PA 19108 ▪ (215) 446-1242 ▪
*keegan.holley at sungard.com* <keegan.holley at sungard.com> Keeping People and
Information Connected® ▪
*http://www.availability.sungard.com/*<http://availability.sungard.com/>
P
*Think before you print*
CONFIDENTIALITY: This e-mail (including any attachments) may contain
confidential, proprietary and privileged information, and unauthorized
disclosure or use is prohibited. If you received this e-mail in error,
please notify the sender and delete this e-mail from your system.
More information about the cisco-nsp
mailing list