[c-nsp] RE: Unknown reason for High CPU 3620
Rodney Dunn
rodunn at cisco.com
Fri Nov 5 08:45:29 EST 2004
She included 'sh int stat' and almost all the traffic is
being switched under interrupt.
> > #sh int stat
> > ATM0/0
> > Switching path Pkts In Chars In Pkts Out Chars Out
> > Processor 60671 5098714 161118 15856770
> > Route cache 25347153 2543070267 22778769 2540265463
> > Total 25407824 2548168981 22939887 2556122233
> > Ethernet1/0
> > Switching path Pkts In Chars In Pkts Out Chars Out
> > Processor 43533 2674126 7176 430560
> > Route cache 22903843 2234233191 25355781 2188907917
> > Total 22947376 2236907317 25362957 2189338477
Rodney
On Fri, Nov 05, 2004 at 07:32:11AM -0600, Jack.W.Parks at alltel.com wrote:
> To be honest, I think the box is maxed out from a PPS perspective. You are sending 2,000 pps each direction and I bet that all the packets are process switched. The 20-40 kpps numbers are for CEF switching only. Obviously, Cisco would not sell many routers if they used non-CEF forwarding statistics in their documentation.
>
> You should be able to look at the "show interface switching" output to see if your process switching the packets. ATM SAR function should be handled by the ATM interface itself and NOT by the processor. The SAR function is not contributing to the CPU utilization. Since you are bridging, you are process switching and you are getting 2 kpps which is the maximum process switched performance you can expect.
>
> Check out this link. Scoll down to "Platform performance comparison"
>
> http://www.cisco.com/en/US/partner/products/hw/routers/ps5199/prod_bulletin09186a008032d40e.html
>
> I'm sure their are more stats about each platform, it just takes some digging around CCO.
>
> Jack
>
>
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net on behalf of Alex Foster
> Sent: Fri 11/5/2004 5:26 AM
> To: Rodney Dunn
> Cc: cisco-nsp at puck.nether.net
> Subject: RE: [c-nsp] RE: Unknown reason for High CPU 3620
>
>
>
>
> The Reasons for bridging are explained in the diagram below.
>
> Bridging Bridging
> Firewall ---- eth |7206| atm --------- atm |3620| eth ----- ISP
>
> So essentially I am using the ATM network to extend the connection from
> the ISP to our Firewall. In this situation you can see why I cant route.
> I cant see how L2TP will be of any help here - comments or suggestions
> are very welcome - Also the ATM network does not support MPLS (fairly
> old OS on our ATM switches).
>
> Thanks Again
>
> Alex
>
>
> -----Original Message-----
> From: Rodney Dunn [mailto:rodunn at cisco.com]
> Sent: 04 November 2004 15:42
> To: Alex Foster
> Cc: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] RE: Unknown reason for High CPU 3620
>
> On Wed, Nov 03, 2004 at 11:46:09PM -0000, Alex Foster wrote:
> > Hi
> >
> > The LANE configuration is used for management only - occassional snmp
> > queries etc; so the bulk of traffic is being bridged between e1/0 and
> atm
> > 0/0.2. Are you saying therefore that bridging is far more intensive
> than
> > routing/switching at interrupt level? Why is that (sorry for the
> question -
> > just curious).
>
> Yes that's correct. Old code and was never designed for high speed
> switching.
>
> >
> > The load on this connection will soon increase to a constant 40 - 60
> Mbps
> > (again bridging - cant route unfortunately). Obviously the 3620 wont
> handle
> > this - I have a 7206 (NPE-400) available, would this box suffice for
> the
> > traffic load I am estimating - I can upgrade the 7206 to a G1
> processor if
> > needs be.
>
> It would have to be tested in the lab to know for sure. It's all
> about processor speed and feature excecution on a per packet basis.
> The old transparent bridging code isn't a great performer even
> though it's still under interrupt level. It's a totally different
> code path than raw packet switching.
>
> Someone else mentioned other types of L2 bridging (ATOM/L2TPv3).
> Those are new technologies designed to carry L2 frames over
> a L3 transport.
>
> It's hard to say what your best solution would be without understanding
> the overall environement you have. ie: Why are you bridging in the
> first place?
>
>
> >
> > Thanks for your time
> >
> > Alex
> >
> > -----Original Message-----
> > From: Rodney Dunn [mailto:rodunn at cisco.com]
> > Sent: 03 November 2004 17:40
> > To: Alex Foster
> > Cc: Rodney Dunn; cisco-nsp at puck.nether.net
> > Subject: Re: [c-nsp] RE: Unknown reason for High CPU 3620
> >
> >
> > Bridging with LANE at that rate. My guess is
> > that would probably be normal on a 3620.
> >
> > Your option would be to get a faster processor
> > or stop bridging.
> >
> > To be 100% sure it would need to be set up in the
> > lab with a packet generator under a controlled
> > test.
> >
> > Your sh int stat confirms at least the most of the
> > traffic is being handled under interrupt level which
> > is good.
> >
> >
> > Rodney
> >
> > On Wed, Nov 03, 2004 at 05:19:54PM -0000, Alex Foster wrote:
> > > Thanks Rodney - find config attached - the collisions were
> historical -
> > > the interface is running full-duplex - here is the current interface
> > > stats - the input errors are being investigated - but shouldn't be
> > > responsible for the high CPU. Have also included sh vers and sh int
> > > stat.
> > >
> > > #sh int e1/0
> > > Ethernet1/0 is up, line protocol is up
> > > Hardware is AmdP2, address is 0005.3245.e070 (bia 0005.3245.e070)
> > > Description:
> > > MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
> > > reliability 255/255, txload 39/255, rxload 39/255
> > > Encapsulation ARPA, loopback not set
> > > Keepalive set (10 sec)
> > > ARP type: ARPA, ARP Timeout 04:00:00
> > > Last input 00:00:00, output 00:00:00, output hang never
> > > Last clearing of "show interface" counters 03:03:41
> > > Queueing strategy: fifo
> > > Output queue 0/40, 0 drops; input queue 1/75, 790 drops
> > > 5 minute input rate 1531000 bits/sec, 1935 packets/sec
> > > 5 minute output rate 1532000 bits/sec, 2170 packets/sec
> > > 21533985 packets input, 2108844947 bytes, 0 no buffer
> > > Received 62660 broadcasts, 0 runts, 0 giants, 0 throttles
> > > 130659 input errors, 1608 CRC, 1272 frame, 0 overrun, 129051
> > > ignored
> > > 0 input packets with dribble condition detected
> > > 23817320 packets output, 2045363543 bytes, 0 underruns(0/0/0)
> > > 0 output errors, 0 collisions, 0 interface resets
> > > 0 babbles, 0 late collision, 0 deferred
> > > 0 lost carrier, 0 no carrier
> > > 0 output buffer failures, 0 output buffers swapped out
> > >
> > > #sh vers
> > > Cisco Internetwork Operating System Software
> > > IOS (tm) 3600 Software (C3620-IS-M), Version 12.2(3), RELEASE
> SOFTWARE
> > > (fc1)
> > > Copyright (c) 1986-2001 by cisco Systems, Inc.
> > > Compiled Wed 18-Jul-01 13:45 by pwade
> > > Image text-base: 0x600089A8, data-base: 0x6116A000
> > >
> > > ROM: System Bootstrap, Version 11.1(20)AA2, EARLY DEPLOYMENT RELEASE
> > > SOFTWARE (fc1)
> > >
> > > xxxxxxxx uptime is 1 year, 29 weeks, 5 days, 1 hour, 48 minutes
> > > System returned to ROM by power-on
> > > System image file is "flash:c3620-is-mz.122-3.bin"
> > >
> > > cisco 3620 (R4700) processor (revision 0x81) with 41984K/7168K bytes
> of
> > > memory.
> > > Processor board ID 25121827
> > > R4700 CPU at 80Mhz, Implementation 33, Rev 1.0
> > > Bridging software.
> > > X.25 software, Version 3.0.0.
> > > SuperLAT software (copyright 1990 by Meridian Technology Corp).
> > > 4 Ethernet/IEEE 802.3 interface(s)
> > > 1 ATM network interface(s)
> > > DRAM configuration is 32 bits wide with parity disabled.
> > > 29K bytes of non-volatile configuration memory.
> > > 16384K bytes of processor board System flash (Read/Write)
> > >
> > > Configuration register is 0x2102
> > >
> > >
> > > #sh int stat
> > > ATM0/0
> > > Switching path Pkts In Chars In Pkts Out Chars Out
> > > Processor 60671 5098714 161118 15856770
> > > Route cache 25347153 2543070267 22778769 2540265463
> > > Total 25407824 2548168981 22939887 2556122233
> > > Ethernet1/0
> > > Switching path Pkts In Chars In Pkts Out Chars Out
> > > Processor 43533 2674126 7176 430560
> > > Route cache 22903843 2234233191 25355781 2188907917
> > > Total 22947376 2236907317 25362957 2189338477
> > > Interface Ethernet1/1 is disabled
> > >
> > > Interface Ethernet1/2 is disabled
> > >
> > > Interface Ethernet1/3 is disabled
> > >
> > > Building configuration...
> > >
> > > Current configuration : 1868 bytes
> > > !
> > > version 12.2
> > > service timestamps debug uptime
> > > service timestamps log uptime
> > > service password-encryption
> > > !
> > > hostname xxxxxxxxxx
> > > !
> > > logging buffered 4096 debugging
> > > enable secret
> > > !
> > > ip subnet-zero
> > > !
> > > !
> > > no ip domain-lookup
> > > !
> > > call rsvp-sync
> > > !
> > > bridge crb
> > > !
> > > interface ATM0/0
> > > no ip address
> > > atm uni-version 3.1
> > > no atm ilmi-keepalive
> > > pvc 0/5 qsaal
> > > !
> > > pvc 0/16 ilmi
> > > !
> > > !
> > > interface ATM0/0.1 multipoint
> > > ip address xxxxxxx
> > > lane server-atm-address C5.777777777777777777777777.777777777777.77
> > > lane client ethernet xxxxxxx
> > > no cdp enable
> > > !
> > > interface ATM0/0.2 multipoint
> > > pvc 0/103
> > > encapsulation aal5snap
> > > !
> > > bridge-group 1
> > > !
> > > interface Ethernet1/0
> > > description
> > > no ip address
> > > no ip mroute-cache
> > > full-duplex
> > > no cdp enable
> > > bridge-group 1
> > > !
> > > no ip http server
> > > !
> > > no cdp run
> > > !
> > > bridge 1 protocol ieee
> > > !
> > > dial-peer cor custom
> > > !
> > > !
> > >
> > > -----Original Message-----
> > > From: Rodney Dunn [mailto:rodunn at cisco.com]
> > > Sent: 03 November 2004 16:08
> > > To: Alex Foster
> > > Cc: cisco-nsp at puck.nether.net
> > > Subject: Re: [c-nsp] RE: Unknown reason for High CPU 3620
> > >
> > > CPU load at interrupt level is almost always packet
> > > switching/features or alignment/spurious access corrections.
> > >
> > > This CPU sounds high for that amount of traffic.
> > >
> > > Can you list the full configuration /*taking out
> > > any confidential stuff*/ so we can see the features
> > > enabled?
> > >
> > > 2nd, you really need to clean up that ethernet
> > > segement with all the collisions. Can you convert
> > > it to full-duplex? I don't recall if you can do it
> > > with the 3620 in later code.
> > >
> > > Most likely it's not an issue with the ATM side.
> > >
> > > Rodney
> > >
> > >
> > > On Wed, Nov 03, 2004 at 03:44:07PM -0000, Alex Foster wrote:
> > > > Thanks to the guys who responded - I have managed to narrow down
> the
> > > > cause to high interrupt utilization and not processor utilization
> -
> > > but
> > > > Im still a little unclear as to what is causing the high interrupt
> > > util.
> > > > - is the ATM interface causing this ? Again the traffic load is
> > > little
> > > > over 1.5 meg (in/out) - but constant - so I am wondering if the
> > > > continual SARing is maybe having a negative effect on the CPU ?
> If so
> > > -
> > > > is there a workaround or am I looking at a bigger platform to
> solve
> > > the
> > > > issue - if so any recommendations (I have a 3640 available but
> don't
> > > > know if this will provide much of an improvement).
> > > >
> > > >
> > > >
> > > > Regards
> > > >
> > > >
> > > >
> > > > Alex
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Alex Foster
> > > > Sent: 03 November 2004 13:37
> > > > To: 'cisco-nsp at puck.nether.net'
> > > > Subject: Unknown reason for High CPU 3620
> > > >
> > > >
> > > >
> > > > I have a 3620 that is currently maxing out its CPU for no apparent
> > > > reason - the traffic load on it at the moment is about 1 to 2 meg
> -
> > > and
> > > > sh proc cpu doesn't indicate any process that is causing the high
> CPU
> > > > util. Most of the traffic is SIP but I wouldn't have thought this
> > > would
> > > > have any bearing. There are only two interfaces on the box in use
> -
> > > an
> > > > ATM 155 and a 10bt. The traffic from the 10bt is bridged to a PVC
> on
> > > > the ATM interface - nothing complicated.
> > > >
> > > >
> > > >
> > > > #sh proc cpu
> > > >
> > > > CPU utilization for five seconds: 96%/93%; one minute: 90%; five
> > > > minutes: 91%
> > > >
> > > > PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY
> Process
> > > >
> > > > 1 194708 9899756 19 0.00% 0.00% 0.00%
> 0
> > > > Load Meter
> > > >
> > > > 2 2008 223 9004 1.41% 0.13%
> 0.03%
> > > > 66 Virtual Exec
> > > >
> > > >
> > > >
> > > > #sh mem sum
> > > >
> > > > Head Total(b) Used(b) Free(b)
> Lowest(b)
> > > > Largest(b)
> > > >
> > > > Processor 61D70EA0 12120416 6616992 5503424
> 5303744
> > > > 5305428
> > > >
> > > > I/O 2900000 7340032 4387068 2952964
> 2920520
> > > > 2947260
> > > >
> > > >
> > > >
> > > > #sh int e1/0
> > > >
> > > > Ethernet1/0 is up, line protocol is up
> > > >
> > > > Hardware is AmdP2, address is 0005.3245.e070 (bia
> 0005.3245.e070)
> > > >
> > > > Description:
> > > >
> > > > MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
> > > >
> > > > reliability 255/255, txload 34/255, rxload 37/255
> > > >
> > > > Encapsulation ARPA, loopback not set
> > > >
> > > > Keepalive set (10 sec)
> > > >
> > > > ARP type: ARPA, ARP Timeout 04:00:00
> > > >
> > > > Last input 00:00:00, output 00:00:00, output hang never
> > > >
> > > > Last clearing of "show interface" counters 1y4w
> > > >
> > > > Queueing strategy: fifo
> > > >
> > > > Output queue 0/40, 0 drops; input queue 0/75, 103522 drops
> > > >
> > > > 5 minute input rate 1470000 bits/sec, 1984 packets/sec
> > > >
> > > > 5 minute output rate 1355000 bits/sec, 2166 packets/sec
> > > >
> > > > 2150894411 packets input, 1646909945 bytes, 0 no buffer
> > > >
> > > > Received 88151770 broadcasts, 0 runts, 0 giants, 0 throttles
> > > >
> > > > 6397644 input errors, 42166 CRC, 32878 frame, 0 overrun,
> 6355422
> > > > ignored
> > > >
> > > > 0 input packets with dribble condition detected
> > > >
> > > > 2681876495 packets output, 1769855380 bytes, 0
> > > > underruns(556991/1663306/0)
> > > >
> > > > 165 output errors, 2220297 collisions, 50 interface resets
> > > >
> > > > 0 babbles, 47 late collision, 1067254 deferred
> > > >
> > > > 118 lost carrier, 0 no carrier
> > > >
> > > > 0 output buffer failures, 0 output buffers swapped out
> > > >
> > > >
> > > >
> > > > #sh int atm 0/0.2
> > > >
> > > > ATM0/0.2 is up, line protocol is up
> > > >
> > > > Hardware is RS8234 ATMOC3
> > > >
> > > > MTU 4470 bytes, BW 155000 Kbit, DLY 80 usec,
> > > >
> > > > reliability 255/255, txload 2/255, rxload 2/255
> > > >
> > > > Encapsulation ATM
> > > >
> > > > 3075506957 packets input, 1094442885139 bytes
> > > >
> > > > 2362546968 packets output,988686523562 bytes
> > > >
> > > > 0 OAM cells input, 0 OAM cells output
> > > >
> > > > AAL5 CRC errors : 81
> > > >
> > > > AAL5 SAR Timeouts : 0
> > > >
> > > > AAL5 Oversized SDUs : 0
> > > >
> > > > AAL5 length violation : 26
> > > >
> > > > AAL5 CPI Error : 0
> > > >
> > > >
> > > >
> > > > Thanks
> > > >
> > > >
> > > >
> > > > Alex
> > > >
> > > >
> > > >
> > > > The information in this e-mail and any attachments is confidential
> and
> > > may be subject to legal professional privilege. It is intended
> solely
> > > for the attention and use of the named addressee(s). If you are not
> the
> > > intended recipient, or person responsible for delivering this
> > > information to the intended recipient, please notify the sender
> > > immediately. Unless you are the intended recipient or his/her
> > > representative you are prohibited from, and therefore must not,
> read,
> > > copy, distribute, use or retain this message or any part of it. The
> > > views expressed in this e-mail may not represent those of Gamma
> Telecom.
> > > >
> > > > This message has been scanned for viruses by MailController
> > > > _______________________________________________
> > > > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > > archive at http://puck.nether.net/pipermail/cisco-nsp/
> > >
> > >
> > > This message has been scanned for viruses by MailController -
> > > www.MailController.altohiway.com
> >
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>
> ******************************************************************************************The information contained in this message, including attachments, may contain privileged or confidential information that is intended to be delivered only to the person identified above. If you are not the intended recipient, or the person responsible for delivering this message to the intended recipient, ALLTEL requests that you immediately notify the sender and asks that you do not read the message or its attachments, and that you delete them without copying or sending them to anyone else.
More information about the cisco-nsp
mailing list