[c-nsp] RE: Unknown reason for High CPU 3620

Rodney Dunn rodunn at cisco.com
Thu Nov 4 10:41:54 EST 2004


On Wed, Nov 03, 2004 at 11:46:09PM -0000, Alex Foster wrote:
> Hi
> 
> The LANE configuration is used for management only - occassional snmp
> queries etc; so the bulk of traffic is being bridged between e1/0 and atm
> 0/0.2.  Are you saying therefore that bridging is far more intensive than
> routing/switching at interrupt level? Why is that (sorry for the question -
> just curious).

Yes that's correct.  Old code and was never designed for high speed 
switching.

> 
> The load on this connection will soon increase to a constant 40 - 60 Mbps
> (again bridging - cant route unfortunately).  Obviously the 3620 wont handle
> this - I have a 7206 (NPE-400) available, would this box suffice for the
> traffic load I am estimating - I can upgrade the 7206 to a G1 processor if
> needs be.

It would have to be tested in the lab to know for sure.  It's all
about processor speed and feature excecution on a per packet basis.
The old transparent bridging code isn't a great performer even
though it's still under interrupt level.  It's a totally different
code path than raw packet switching.

Someone else mentioned other types of L2 bridging (ATOM/L2TPv3).
Those are new technologies designed to carry L2 frames over
a L3 transport.

It's hard to say what your best solution would be without understanding
the overall environement you have.  ie: Why are you bridging in the
first place?

 
> 
> Thanks for your time
> 
> Alex
> 
> -----Original Message-----
> From: Rodney Dunn [mailto:rodunn at cisco.com]
> Sent: 03 November 2004 17:40
> To: Alex Foster
> Cc: Rodney Dunn; cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] RE: Unknown reason for High CPU 3620
> 
> 
> Bridging with LANE at that rate.  My guess is
> that would probably be normal on a 3620.
> 
> Your option would be to get a faster processor
> or stop bridging.
> 
> To be 100% sure it would need to be set up in the
> lab with a packet generator under a controlled
> test.
> 
> Your sh int stat confirms at least the most of the
> traffic is being handled under interrupt level which
> is good.
> 
> 
> Rodney
> 
> On Wed, Nov 03, 2004 at 05:19:54PM -0000, Alex Foster wrote:
> > Thanks Rodney - find config attached - the collisions were historical -
> > the interface is running full-duplex - here is the current interface
> > stats - the input errors are being investigated - but shouldn't be
> > responsible for the high CPU. Have also included sh vers and sh int
> > stat.
> >
> > #sh int e1/0
> > Ethernet1/0 is up, line protocol is up
> >   Hardware is AmdP2, address is 0005.3245.e070 (bia 0005.3245.e070)
> >   Description:
> >   MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
> >      reliability 255/255, txload 39/255, rxload 39/255
> >   Encapsulation ARPA, loopback not set
> >   Keepalive set (10 sec)
> >   ARP type: ARPA, ARP Timeout 04:00:00
> >   Last input 00:00:00, output 00:00:00, output hang never
> >   Last clearing of "show interface" counters 03:03:41
> >   Queueing strategy: fifo
> >   Output queue 0/40, 0 drops; input queue 1/75, 790 drops
> >   5 minute input rate 1531000 bits/sec, 1935 packets/sec
> >   5 minute output rate 1532000 bits/sec, 2170 packets/sec
> >      21533985 packets input, 2108844947 bytes, 0 no buffer
> >      Received 62660 broadcasts, 0 runts, 0 giants, 0 throttles
> >      130659 input errors, 1608 CRC, 1272 frame, 0 overrun, 129051
> > ignored
> >      0 input packets with dribble condition detected
> >      23817320 packets output, 2045363543 bytes, 0 underruns(0/0/0)
> >      0 output errors, 0 collisions, 0 interface resets
> >      0 babbles, 0 late collision, 0 deferred
> >      0 lost carrier, 0 no carrier
> >      0 output buffer failures, 0 output buffers swapped out
> >
> > #sh vers
> > Cisco Internetwork Operating System Software
> > IOS (tm) 3600 Software (C3620-IS-M), Version 12.2(3), RELEASE SOFTWARE
> > (fc1)
> > Copyright (c) 1986-2001 by cisco Systems, Inc.
> > Compiled Wed 18-Jul-01 13:45 by pwade
> > Image text-base: 0x600089A8, data-base: 0x6116A000
> >
> > ROM: System Bootstrap, Version 11.1(20)AA2, EARLY DEPLOYMENT RELEASE
> > SOFTWARE (fc1)
> >
> > xxxxxxxx uptime is 1 year, 29 weeks, 5 days, 1 hour, 48 minutes
> > System returned to ROM by power-on
> > System image file is "flash:c3620-is-mz.122-3.bin"
> >
> > cisco 3620 (R4700) processor (revision 0x81) with 41984K/7168K bytes of
> > memory.
> > Processor board ID 25121827
> > R4700 CPU at 80Mhz, Implementation 33, Rev 1.0
> > Bridging software.
> > X.25 software, Version 3.0.0.
> > SuperLAT software (copyright 1990 by Meridian Technology Corp).
> > 4 Ethernet/IEEE 802.3 interface(s)
> > 1 ATM network interface(s)
> > DRAM configuration is 32 bits wide with parity disabled.
> > 29K bytes of non-volatile configuration memory.
> > 16384K bytes of processor board System flash (Read/Write)
> >
> > Configuration register is 0x2102
> >
> >
> > #sh int stat
> > ATM0/0
> >           Switching path    Pkts In   Chars In   Pkts Out  Chars Out
> >                Processor      60671    5098714     161118   15856770
> >              Route cache   25347153 2543070267   22778769 2540265463
> >                    Total   25407824 2548168981   22939887 2556122233
> > Ethernet1/0
> >           Switching path    Pkts In   Chars In   Pkts Out  Chars Out
> >                Processor      43533    2674126       7176     430560
> >              Route cache   22903843 2234233191   25355781 2188907917
> >                    Total   22947376 2236907317   25362957 2189338477
> > Interface Ethernet1/1 is disabled
> >
> > Interface Ethernet1/2 is disabled
> >
> > Interface Ethernet1/3 is disabled
> >
> > Building configuration...
> >
> > Current configuration : 1868 bytes
> > !
> > version 12.2
> > service timestamps debug uptime
> > service timestamps log uptime
> > service password-encryption
> > !
> > hostname xxxxxxxxxx
> > !
> > logging buffered 4096 debugging
> > enable secret
> > !
> > ip subnet-zero
> > !
> > !
> > no ip domain-lookup
> > !
> > call rsvp-sync
> > !
> > bridge crb
> > !
> > interface ATM0/0
> >  no ip address
> >  atm uni-version 3.1
> >  no atm ilmi-keepalive
> >  pvc 0/5 qsaal
> >  !
> >  pvc 0/16 ilmi
> >  !
> > !
> > interface ATM0/0.1 multipoint
> >  ip address xxxxxxx
> >  lane server-atm-address C5.777777777777777777777777.777777777777.77
> >  lane client ethernet xxxxxxx
> >  no cdp enable
> > !
> > interface ATM0/0.2 multipoint
> >  pvc 0/103
> >   encapsulation aal5snap
> >  !
> >  bridge-group 1
> > !
> > interface Ethernet1/0
> >  description
> >  no ip address
> >  no ip mroute-cache
> >  full-duplex
> >  no cdp enable
> >  bridge-group 1
> > !
> > no ip http server
> > !
> > no cdp run
> > !
> > bridge 1 protocol ieee
> > !
> > dial-peer cor custom
> > !
> > !
> >
> > -----Original Message-----
> > From: Rodney Dunn [mailto:rodunn at cisco.com]
> > Sent: 03 November 2004 16:08
> > To: Alex Foster
> > Cc: cisco-nsp at puck.nether.net
> > Subject: Re: [c-nsp] RE: Unknown reason for High CPU 3620
> >
> > CPU load at interrupt level is almost always packet
> > switching/features or alignment/spurious access corrections.
> >
> > This CPU sounds high for that amount of traffic.
> >
> > Can you list the full configuration /*taking out
> > any confidential stuff*/ so we can see the features
> > enabled?
> >
> > 2nd, you really need to clean up that ethernet
> > segement with all the collisions.  Can you convert
> > it to full-duplex?  I don't recall if you can do it
> > with the 3620 in later code.
> >
> > Most likely it's not an issue with the ATM side.
> >
> > Rodney
> >
> >
> > On Wed, Nov 03, 2004 at 03:44:07PM -0000, Alex Foster wrote:
> > > Thanks to the guys who responded - I have managed to narrow down the
> > > cause to high interrupt utilization and not processor utilization -
> > but
> > > Im still a little unclear as to what is causing the high interrupt
> > util.
> > > - is the ATM interface causing this ?  Again the traffic load is
> > little
> > > over 1.5 meg (in/out) - but constant - so I am wondering if the
> > > continual SARing is maybe having a negative effect on the CPU ?  If so
> > -
> > > is there a workaround or am I looking at a bigger platform to solve
> > the
> > > issue - if so any recommendations (I have a 3640 available but don't
> > > know if this will provide much of an improvement).
> > >
> > >
> > >
> > > Regards
> > >
> > >
> > >
> > > Alex
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Alex Foster
> > > Sent: 03 November 2004 13:37
> > > To: 'cisco-nsp at puck.nether.net'
> > > Subject: Unknown reason for High CPU 3620
> > >
> > >
> > >
> > > I have a 3620 that is currently maxing out its CPU for no apparent
> > > reason - the traffic load on it at the moment is about 1 to 2 meg -
> > and
> > > sh proc cpu doesn't indicate any process that is causing the high CPU
> > > util.  Most of the traffic is SIP but I wouldn't have thought this
> > would
> > > have any bearing.  There are only two interfaces on the box in use -
> > an
> > > ATM 155 and a 10bt.  The traffic from the 10bt is bridged to a PVC on
> > > the ATM interface - nothing complicated.
> > >
> > >
> > >
> > > #sh proc cpu
> > >
> > > CPU utilization for five seconds: 96%/93%; one minute: 90%; five
> > > minutes: 91%
> > >
> > >  PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
> > >
> > >    1      194708       9899756         19    0.00%  0.00%  0.00%   0
> > > Load Meter
> > >
> > >    2        2008          223                9004  1.41%  0.13%  0.03%
> > > 66 Virtual Exec
> > >
> > >
> > >
> > > #sh mem sum
> > >
> > >                 Head    Total(b)     Used(b)     Free(b)   Lowest(b)
> > > Largest(b)
> > >
> > > Processor   61D70EA0    12120416     6616992     5503424     5303744
> > > 5305428
> > >
> > >       I/O    2900000     7340032     4387068     2952964     2920520
> > > 2947260
> > >
> > >
> > >
> > > #sh int e1/0
> > >
> > > Ethernet1/0 is up, line protocol is up
> > >
> > >   Hardware is AmdP2, address is 0005.3245.e070 (bia 0005.3245.e070)
> > >
> > >   Description:
> > >
> > >   MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
> > >
> > >      reliability 255/255, txload 34/255, rxload 37/255
> > >
> > >   Encapsulation ARPA, loopback not set
> > >
> > >   Keepalive set (10 sec)
> > >
> > >   ARP type: ARPA, ARP Timeout 04:00:00
> > >
> > >   Last input 00:00:00, output 00:00:00, output hang never
> > >
> > >   Last clearing of "show interface" counters 1y4w
> > >
> > >   Queueing strategy: fifo
> > >
> > >   Output queue 0/40, 0 drops; input queue 0/75, 103522 drops
> > >
> > >   5 minute input rate 1470000 bits/sec, 1984 packets/sec
> > >
> > >   5 minute output rate 1355000 bits/sec, 2166 packets/sec
> > >
> > >      2150894411 packets input, 1646909945 bytes, 0 no buffer
> > >
> > >      Received 88151770 broadcasts, 0 runts, 0 giants, 0 throttles
> > >
> > >      6397644 input errors, 42166 CRC, 32878 frame, 0 overrun, 6355422
> > > ignored
> > >
> > >      0 input packets with dribble condition detected
> > >
> > >      2681876495 packets output, 1769855380 bytes, 0
> > > underruns(556991/1663306/0)
> > >
> > >      165 output errors, 2220297 collisions, 50 interface resets
> > >
> > >      0 babbles, 47 late collision, 1067254 deferred
> > >
> > >      118 lost carrier, 0 no carrier
> > >
> > >      0 output buffer failures, 0 output buffers swapped out
> > >
> > >
> > >
> > > #sh int atm 0/0.2
> > >
> > > ATM0/0.2 is up, line protocol is up
> > >
> > >   Hardware is RS8234 ATMOC3
> > >
> > >   MTU 4470 bytes, BW 155000 Kbit, DLY 80 usec,
> > >
> > >      reliability 255/255, txload 2/255, rxload 2/255
> > >
> > >   Encapsulation ATM
> > >
> > >   3075506957 packets input, 1094442885139 bytes
> > >
> > >   2362546968 packets output,988686523562 bytes
> > >
> > >   0 OAM cells input, 0 OAM cells output
> > >
> > >   AAL5 CRC errors : 81
> > >
> > >   AAL5 SAR Timeouts : 0
> > >
> > >   AAL5 Oversized SDUs : 0
> > >
> > >   AAL5 length violation : 26
> > >
> > >   AAL5 CPI Error : 0
> > >
> > >
> > >
> > > Thanks
> > >
> > >
> > >
> > > Alex
> > >
> > >
> > >
> > > The information in this e-mail and any attachments is confidential and
> > may be subject to legal professional privilege. It is intended solely
> > for the attention and use of the named addressee(s). If you are not the
> > intended recipient, or person responsible for delivering this
> > information to the intended recipient, please notify the sender
> > immediately. Unless you are the intended recipient or his/her
> > representative you are prohibited from, and therefore must not, read,
> > copy, distribute, use or retain this message or any part of it. The
> > views expressed in this e-mail may not represent those of Gamma Telecom.
> > >
> > > This message has been scanned for viruses by MailController
> > > _______________________________________________
> > > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >
> >
> > This message has been scanned for viruses by MailController -
> > www.MailController.altohiway.com
> 


More information about the cisco-nsp mailing list