[c-nsp] RE: Unknown reason for High CPU 3620
Rodney Dunn
rodunn at cisco.com
Thu Nov 4 10:41:54 EST 2004
On Wed, Nov 03, 2004 at 11:46:09PM -0000, Alex Foster wrote:
> Hi
>
> The LANE configuration is used for management only - occassional snmp
> queries etc; so the bulk of traffic is being bridged between e1/0 and atm
> 0/0.2. Are you saying therefore that bridging is far more intensive than
> routing/switching at interrupt level? Why is that (sorry for the question -
> just curious).
Yes that's correct. Old code and was never designed for high speed
switching.
>
> The load on this connection will soon increase to a constant 40 - 60 Mbps
> (again bridging - cant route unfortunately). Obviously the 3620 wont handle
> this - I have a 7206 (NPE-400) available, would this box suffice for the
> traffic load I am estimating - I can upgrade the 7206 to a G1 processor if
> needs be.
It would have to be tested in the lab to know for sure. It's all
about processor speed and feature excecution on a per packet basis.
The old transparent bridging code isn't a great performer even
though it's still under interrupt level. It's a totally different
code path than raw packet switching.
Someone else mentioned other types of L2 bridging (ATOM/L2TPv3).
Those are new technologies designed to carry L2 frames over
a L3 transport.
It's hard to say what your best solution would be without understanding
the overall environement you have. ie: Why are you bridging in the
first place?
>
> Thanks for your time
>
> Alex
>
> -----Original Message-----
> From: Rodney Dunn [mailto:rodunn at cisco.com]
> Sent: 03 November 2004 17:40
> To: Alex Foster
> Cc: Rodney Dunn; cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] RE: Unknown reason for High CPU 3620
>
>
> Bridging with LANE at that rate. My guess is
> that would probably be normal on a 3620.
>
> Your option would be to get a faster processor
> or stop bridging.
>
> To be 100% sure it would need to be set up in the
> lab with a packet generator under a controlled
> test.
>
> Your sh int stat confirms at least the most of the
> traffic is being handled under interrupt level which
> is good.
>
>
> Rodney
>
> On Wed, Nov 03, 2004 at 05:19:54PM -0000, Alex Foster wrote:
> > Thanks Rodney - find config attached - the collisions were historical -
> > the interface is running full-duplex - here is the current interface
> > stats - the input errors are being investigated - but shouldn't be
> > responsible for the high CPU. Have also included sh vers and sh int
> > stat.
> >
> > #sh int e1/0
> > Ethernet1/0 is up, line protocol is up
> > Hardware is AmdP2, address is 0005.3245.e070 (bia 0005.3245.e070)
> > Description:
> > MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
> > reliability 255/255, txload 39/255, rxload 39/255
> > Encapsulation ARPA, loopback not set
> > Keepalive set (10 sec)
> > ARP type: ARPA, ARP Timeout 04:00:00
> > Last input 00:00:00, output 00:00:00, output hang never
> > Last clearing of "show interface" counters 03:03:41
> > Queueing strategy: fifo
> > Output queue 0/40, 0 drops; input queue 1/75, 790 drops
> > 5 minute input rate 1531000 bits/sec, 1935 packets/sec
> > 5 minute output rate 1532000 bits/sec, 2170 packets/sec
> > 21533985 packets input, 2108844947 bytes, 0 no buffer
> > Received 62660 broadcasts, 0 runts, 0 giants, 0 throttles
> > 130659 input errors, 1608 CRC, 1272 frame, 0 overrun, 129051
> > ignored
> > 0 input packets with dribble condition detected
> > 23817320 packets output, 2045363543 bytes, 0 underruns(0/0/0)
> > 0 output errors, 0 collisions, 0 interface resets
> > 0 babbles, 0 late collision, 0 deferred
> > 0 lost carrier, 0 no carrier
> > 0 output buffer failures, 0 output buffers swapped out
> >
> > #sh vers
> > Cisco Internetwork Operating System Software
> > IOS (tm) 3600 Software (C3620-IS-M), Version 12.2(3), RELEASE SOFTWARE
> > (fc1)
> > Copyright (c) 1986-2001 by cisco Systems, Inc.
> > Compiled Wed 18-Jul-01 13:45 by pwade
> > Image text-base: 0x600089A8, data-base: 0x6116A000
> >
> > ROM: System Bootstrap, Version 11.1(20)AA2, EARLY DEPLOYMENT RELEASE
> > SOFTWARE (fc1)
> >
> > xxxxxxxx uptime is 1 year, 29 weeks, 5 days, 1 hour, 48 minutes
> > System returned to ROM by power-on
> > System image file is "flash:c3620-is-mz.122-3.bin"
> >
> > cisco 3620 (R4700) processor (revision 0x81) with 41984K/7168K bytes of
> > memory.
> > Processor board ID 25121827
> > R4700 CPU at 80Mhz, Implementation 33, Rev 1.0
> > Bridging software.
> > X.25 software, Version 3.0.0.
> > SuperLAT software (copyright 1990 by Meridian Technology Corp).
> > 4 Ethernet/IEEE 802.3 interface(s)
> > 1 ATM network interface(s)
> > DRAM configuration is 32 bits wide with parity disabled.
> > 29K bytes of non-volatile configuration memory.
> > 16384K bytes of processor board System flash (Read/Write)
> >
> > Configuration register is 0x2102
> >
> >
> > #sh int stat
> > ATM0/0
> > Switching path Pkts In Chars In Pkts Out Chars Out
> > Processor 60671 5098714 161118 15856770
> > Route cache 25347153 2543070267 22778769 2540265463
> > Total 25407824 2548168981 22939887 2556122233
> > Ethernet1/0
> > Switching path Pkts In Chars In Pkts Out Chars Out
> > Processor 43533 2674126 7176 430560
> > Route cache 22903843 2234233191 25355781 2188907917
> > Total 22947376 2236907317 25362957 2189338477
> > Interface Ethernet1/1 is disabled
> >
> > Interface Ethernet1/2 is disabled
> >
> > Interface Ethernet1/3 is disabled
> >
> > Building configuration...
> >
> > Current configuration : 1868 bytes
> > !
> > version 12.2
> > service timestamps debug uptime
> > service timestamps log uptime
> > service password-encryption
> > !
> > hostname xxxxxxxxxx
> > !
> > logging buffered 4096 debugging
> > enable secret
> > !
> > ip subnet-zero
> > !
> > !
> > no ip domain-lookup
> > !
> > call rsvp-sync
> > !
> > bridge crb
> > !
> > interface ATM0/0
> > no ip address
> > atm uni-version 3.1
> > no atm ilmi-keepalive
> > pvc 0/5 qsaal
> > !
> > pvc 0/16 ilmi
> > !
> > !
> > interface ATM0/0.1 multipoint
> > ip address xxxxxxx
> > lane server-atm-address C5.777777777777777777777777.777777777777.77
> > lane client ethernet xxxxxxx
> > no cdp enable
> > !
> > interface ATM0/0.2 multipoint
> > pvc 0/103
> > encapsulation aal5snap
> > !
> > bridge-group 1
> > !
> > interface Ethernet1/0
> > description
> > no ip address
> > no ip mroute-cache
> > full-duplex
> > no cdp enable
> > bridge-group 1
> > !
> > no ip http server
> > !
> > no cdp run
> > !
> > bridge 1 protocol ieee
> > !
> > dial-peer cor custom
> > !
> > !
> >
> > -----Original Message-----
> > From: Rodney Dunn [mailto:rodunn at cisco.com]
> > Sent: 03 November 2004 16:08
> > To: Alex Foster
> > Cc: cisco-nsp at puck.nether.net
> > Subject: Re: [c-nsp] RE: Unknown reason for High CPU 3620
> >
> > CPU load at interrupt level is almost always packet
> > switching/features or alignment/spurious access corrections.
> >
> > This CPU sounds high for that amount of traffic.
> >
> > Can you list the full configuration /*taking out
> > any confidential stuff*/ so we can see the features
> > enabled?
> >
> > 2nd, you really need to clean up that ethernet
> > segement with all the collisions. Can you convert
> > it to full-duplex? I don't recall if you can do it
> > with the 3620 in later code.
> >
> > Most likely it's not an issue with the ATM side.
> >
> > Rodney
> >
> >
> > On Wed, Nov 03, 2004 at 03:44:07PM -0000, Alex Foster wrote:
> > > Thanks to the guys who responded - I have managed to narrow down the
> > > cause to high interrupt utilization and not processor utilization -
> > but
> > > Im still a little unclear as to what is causing the high interrupt
> > util.
> > > - is the ATM interface causing this ? Again the traffic load is
> > little
> > > over 1.5 meg (in/out) - but constant - so I am wondering if the
> > > continual SARing is maybe having a negative effect on the CPU ? If so
> > -
> > > is there a workaround or am I looking at a bigger platform to solve
> > the
> > > issue - if so any recommendations (I have a 3640 available but don't
> > > know if this will provide much of an improvement).
> > >
> > >
> > >
> > > Regards
> > >
> > >
> > >
> > > Alex
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Alex Foster
> > > Sent: 03 November 2004 13:37
> > > To: 'cisco-nsp at puck.nether.net'
> > > Subject: Unknown reason for High CPU 3620
> > >
> > >
> > >
> > > I have a 3620 that is currently maxing out its CPU for no apparent
> > > reason - the traffic load on it at the moment is about 1 to 2 meg -
> > and
> > > sh proc cpu doesn't indicate any process that is causing the high CPU
> > > util. Most of the traffic is SIP but I wouldn't have thought this
> > would
> > > have any bearing. There are only two interfaces on the box in use -
> > an
> > > ATM 155 and a 10bt. The traffic from the 10bt is bridged to a PVC on
> > > the ATM interface - nothing complicated.
> > >
> > >
> > >
> > > #sh proc cpu
> > >
> > > CPU utilization for five seconds: 96%/93%; one minute: 90%; five
> > > minutes: 91%
> > >
> > > PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
> > >
> > > 1 194708 9899756 19 0.00% 0.00% 0.00% 0
> > > Load Meter
> > >
> > > 2 2008 223 9004 1.41% 0.13% 0.03%
> > > 66 Virtual Exec
> > >
> > >
> > >
> > > #sh mem sum
> > >
> > > Head Total(b) Used(b) Free(b) Lowest(b)
> > > Largest(b)
> > >
> > > Processor 61D70EA0 12120416 6616992 5503424 5303744
> > > 5305428
> > >
> > > I/O 2900000 7340032 4387068 2952964 2920520
> > > 2947260
> > >
> > >
> > >
> > > #sh int e1/0
> > >
> > > Ethernet1/0 is up, line protocol is up
> > >
> > > Hardware is AmdP2, address is 0005.3245.e070 (bia 0005.3245.e070)
> > >
> > > Description:
> > >
> > > MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
> > >
> > > reliability 255/255, txload 34/255, rxload 37/255
> > >
> > > Encapsulation ARPA, loopback not set
> > >
> > > Keepalive set (10 sec)
> > >
> > > ARP type: ARPA, ARP Timeout 04:00:00
> > >
> > > Last input 00:00:00, output 00:00:00, output hang never
> > >
> > > Last clearing of "show interface" counters 1y4w
> > >
> > > Queueing strategy: fifo
> > >
> > > Output queue 0/40, 0 drops; input queue 0/75, 103522 drops
> > >
> > > 5 minute input rate 1470000 bits/sec, 1984 packets/sec
> > >
> > > 5 minute output rate 1355000 bits/sec, 2166 packets/sec
> > >
> > > 2150894411 packets input, 1646909945 bytes, 0 no buffer
> > >
> > > Received 88151770 broadcasts, 0 runts, 0 giants, 0 throttles
> > >
> > > 6397644 input errors, 42166 CRC, 32878 frame, 0 overrun, 6355422
> > > ignored
> > >
> > > 0 input packets with dribble condition detected
> > >
> > > 2681876495 packets output, 1769855380 bytes, 0
> > > underruns(556991/1663306/0)
> > >
> > > 165 output errors, 2220297 collisions, 50 interface resets
> > >
> > > 0 babbles, 47 late collision, 1067254 deferred
> > >
> > > 118 lost carrier, 0 no carrier
> > >
> > > 0 output buffer failures, 0 output buffers swapped out
> > >
> > >
> > >
> > > #sh int atm 0/0.2
> > >
> > > ATM0/0.2 is up, line protocol is up
> > >
> > > Hardware is RS8234 ATMOC3
> > >
> > > MTU 4470 bytes, BW 155000 Kbit, DLY 80 usec,
> > >
> > > reliability 255/255, txload 2/255, rxload 2/255
> > >
> > > Encapsulation ATM
> > >
> > > 3075506957 packets input, 1094442885139 bytes
> > >
> > > 2362546968 packets output,988686523562 bytes
> > >
> > > 0 OAM cells input, 0 OAM cells output
> > >
> > > AAL5 CRC errors : 81
> > >
> > > AAL5 SAR Timeouts : 0
> > >
> > > AAL5 Oversized SDUs : 0
> > >
> > > AAL5 length violation : 26
> > >
> > > AAL5 CPI Error : 0
> > >
> > >
> > >
> > > Thanks
> > >
> > >
> > >
> > > Alex
> > >
> > >
> > >
> > > The information in this e-mail and any attachments is confidential and
> > may be subject to legal professional privilege. It is intended solely
> > for the attention and use of the named addressee(s). If you are not the
> > intended recipient, or person responsible for delivering this
> > information to the intended recipient, please notify the sender
> > immediately. Unless you are the intended recipient or his/her
> > representative you are prohibited from, and therefore must not, read,
> > copy, distribute, use or retain this message or any part of it. The
> > views expressed in this e-mail may not represent those of Gamma Telecom.
> > >
> > > This message has been scanned for viruses by MailController
> > > _______________________________________________
> > > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >
> >
> > This message has been scanned for viruses by MailController -
> > www.MailController.altohiway.com
>
More information about the cisco-nsp
mailing list