[c-nsp] RE: Unknown reason for High CPU 3620
Jack.W.Parks at alltel.com
Jack.W.Parks at alltel.com
Fri Nov 5 10:06:40 EST 2004
Try this link for an explaination of the packet switching types.
Cisco 3600 Series Router Architecture - Document ID: 7442
http://www.cisco.com/en/US/partner/products/hw/routers/ps274/products_tech_note09186a00801e1155.shtml
Check out the "Packet Switching" section.
Jack
-----Original Message-----
From: Alex Foster [mailto:afoster at gammatelecom.com]
Sent: Fri 11/5/2004 8:43 AM
To: Parks, Jack W
Cc: Rodney Dunn; Temkin, David; cisco-nsp at puck.nether.net
Subject: RE: [c-nsp] RE: Unknown reason for High CPU 3620
Confusion has just set in - please enlighten me ???
Are these statements correct ??
Bridging is handled at interrupt level and is then process-switched
Routing (without CEF) is handled at process level and is then
fast-switched
Routing (with CEF) is handled at interrupt level and is then
fast-switched
These may be generically incorrect - I understand some types of traffic
are always handled at process-level ie: telnet, ping, protocol updates
etc;
Regards
Alex
-----Original Message-----
From: Jack.W.Parks at alltel.com [mailto:Jack.W.Parks at alltel.com]
Sent: 05 November 2004 13:32
To: Alex Foster; rodunn at cisco.com
Cc: cisco-nsp at puck.nether.net
Subject: RE: [c-nsp] RE: Unknown reason for High CPU 3620
To be honest, I think the box is maxed out from a PPS perspective. You
are sending 2,000 pps each direction and I bet that all the packets are
process switched. The 20-40 kpps numbers are for CEF switching only.
Obviously, Cisco would not sell many routers if they used non-CEF
forwarding statistics in their documentation.
You should be able to look at the "show interface switching" output to
see if your process switching the packets. ATM SAR function should be
handled by the ATM interface itself and NOT by the processor. The SAR
function is not contributing to the CPU utilization. Since you are
bridging, you are process switching and you are getting 2 kpps which is
the maximum process switched performance you can expect.
Check out this link. Scoll down to "Platform performance comparison"
http://www.cisco.com/en/US/partner/products/hw/routers/ps5199/prod_bulle
tin09186a008032d40e.html
I'm sure their are more stats about each platform, it just takes some
digging around CCO.
Jack
-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net on behalf of Alex Foster
Sent: Fri 11/5/2004 5:26 AM
To: Rodney Dunn
Cc: cisco-nsp at puck.nether.net
Subject: RE: [c-nsp] RE: Unknown reason for High CPU 3620
The Reasons for bridging are explained in the diagram below.
Bridging Bridging
Firewall ---- eth |7206| atm --------- atm |3620| eth ----- ISP
So essentially I am using the ATM network to extend the
connection from
the ISP to our Firewall. In this situation you can see why I
cant route.
I cant see how L2TP will be of any help here - comments or
suggestions
are very welcome - Also the ATM network does not support MPLS
(fairly
old OS on our ATM switches).
Thanks Again
Alex
-----Original Message-----
From: Rodney Dunn [mailto:rodunn at cisco.com]
Sent: 04 November 2004 15:42
To: Alex Foster
Cc: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] RE: Unknown reason for High CPU 3620
On Wed, Nov 03, 2004 at 11:46:09PM -0000, Alex Foster wrote:
> Hi
>
> The LANE configuration is used for management only -
occassional snmp
> queries etc; so the bulk of traffic is being bridged between
e1/0 and
atm
> 0/0.2. Are you saying therefore that bridging is far more
intensive
than
> routing/switching at interrupt level? Why is that (sorry for
the
question -
> just curious).
Yes that's correct. Old code and was never designed for high
speed
switching.
>
> The load on this connection will soon increase to a constant
40 - 60
Mbps
> (again bridging - cant route unfortunately). Obviously the
3620 wont
handle
> this - I have a 7206 (NPE-400) available, would this box
suffice for
the
> traffic load I am estimating - I can upgrade the 7206 to a G1
processor if
> needs be.
It would have to be tested in the lab to know for sure. It's
all
about processor speed and feature excecution on a per packet
basis.
The old transparent bridging code isn't a great performer even
though it's still under interrupt level. It's a totally
different
code path than raw packet switching.
Someone else mentioned other types of L2 bridging (ATOM/L2TPv3).
Those are new technologies designed to carry L2 frames over
a L3 transport.
It's hard to say what your best solution would be without
understanding
the overall environement you have. ie: Why are you bridging in
the
first place?
>
> Thanks for your time
>
> Alex
>
> -----Original Message-----
> From: Rodney Dunn [mailto:rodunn at cisco.com]
> Sent: 03 November 2004 17:40
> To: Alex Foster
> Cc: Rodney Dunn; cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] RE: Unknown reason for High CPU 3620
>
>
> Bridging with LANE at that rate. My guess is
> that would probably be normal on a 3620.
>
> Your option would be to get a faster processor
> or stop bridging.
>
> To be 100% sure it would need to be set up in the
> lab with a packet generator under a controlled
> test.
>
> Your sh int stat confirms at least the most of the
> traffic is being handled under interrupt level which
> is good.
>
>
> Rodney
>
> On Wed, Nov 03, 2004 at 05:19:54PM -0000, Alex Foster wrote:
> > Thanks Rodney - find config attached - the collisions were
historical -
> > the interface is running full-duplex - here is the current
interface
> > stats - the input errors are being investigated - but
shouldn't be
> > responsible for the high CPU. Have also included sh vers and
sh int
> > stat.
> >
> > #sh int e1/0
> > Ethernet1/0 is up, line protocol is up
> > Hardware is AmdP2, address is 0005.3245.e070 (bia
0005.3245.e070)
> > Description:
> > MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
> > reliability 255/255, txload 39/255, rxload 39/255
> > Encapsulation ARPA, loopback not set
> > Keepalive set (10 sec)
> > ARP type: ARPA, ARP Timeout 04:00:00
> > Last input 00:00:00, output 00:00:00, output hang never
> > Last clearing of "show interface" counters 03:03:41
> > Queueing strategy: fifo
> > Output queue 0/40, 0 drops; input queue 1/75, 790 drops
> > 5 minute input rate 1531000 bits/sec, 1935 packets/sec
> > 5 minute output rate 1532000 bits/sec, 2170 packets/sec
> > 21533985 packets input, 2108844947 bytes, 0 no buffer
> > Received 62660 broadcasts, 0 runts, 0 giants, 0
throttles
> > 130659 input errors, 1608 CRC, 1272 frame, 0 overrun,
129051
> > ignored
> > 0 input packets with dribble condition detected
> > 23817320 packets output, 2045363543 bytes, 0
underruns(0/0/0)
> > 0 output errors, 0 collisions, 0 interface resets
> > 0 babbles, 0 late collision, 0 deferred
> > 0 lost carrier, 0 no carrier
> > 0 output buffer failures, 0 output buffers swapped out
> >
> > #sh vers
> > Cisco Internetwork Operating System Software
> > IOS (tm) 3600 Software (C3620-IS-M), Version 12.2(3),
RELEASE
SOFTWARE
> > (fc1)
> > Copyright (c) 1986-2001 by cisco Systems, Inc.
> > Compiled Wed 18-Jul-01 13:45 by pwade
> > Image text-base: 0x600089A8, data-base: 0x6116A000
> >
> > ROM: System Bootstrap, Version 11.1(20)AA2, EARLY DEPLOYMENT
RELEASE
> > SOFTWARE (fc1)
> >
> > xxxxxxxx uptime is 1 year, 29 weeks, 5 days, 1 hour, 48
minutes
> > System returned to ROM by power-on
> > System image file is "flash:c3620-is-mz.122-3.bin"
> >
> > cisco 3620 (R4700) processor (revision 0x81) with
41984K/7168K bytes
of
> > memory.
> > Processor board ID 25121827
> > R4700 CPU at 80Mhz, Implementation 33, Rev 1.0
> > Bridging software.
> > X.25 software, Version 3.0.0.
> > SuperLAT software (copyright 1990 by Meridian Technology
Corp).
> > 4 Ethernet/IEEE 802.3 interface(s)
> > 1 ATM network interface(s)
> > DRAM configuration is 32 bits wide with parity disabled.
> > 29K bytes of non-volatile configuration memory.
> > 16384K bytes of processor board System flash (Read/Write)
> >
> > Configuration register is 0x2102
> >
> >
> > #sh int stat
> > ATM0/0
> > Switching path Pkts In Chars In Pkts Out
Chars Out
> > Processor 60671 5098714 161118
15856770
> > Route cache 25347153 2543070267 22778769
2540265463
> > Total 25407824 2548168981 22939887
2556122233
> > Ethernet1/0
> > Switching path Pkts In Chars In Pkts Out
Chars Out
> > Processor 43533 2674126 7176
430560
> > Route cache 22903843 2234233191 25355781
2188907917
> > Total 22947376 2236907317 25362957
2189338477
> > Interface Ethernet1/1 is disabled
> >
> > Interface Ethernet1/2 is disabled
> >
> > Interface Ethernet1/3 is disabled
> >
> > Building configuration...
> >
> > Current configuration : 1868 bytes
> > !
> > version 12.2
> > service timestamps debug uptime
> > service timestamps log uptime
> > service password-encryption
> > !
> > hostname xxxxxxxxxx
> > !
> > logging buffered 4096 debugging
> > enable secret
> > !
> > ip subnet-zero
> > !
> > !
> > no ip domain-lookup
> > !
> > call rsvp-sync
> > !
> > bridge crb
> > !
> > interface ATM0/0
> > no ip address
> > atm uni-version 3.1
> > no atm ilmi-keepalive
> > pvc 0/5 qsaal
> > !
> > pvc 0/16 ilmi
> > !
> > !
> > interface ATM0/0.1 multipoint
> > ip address xxxxxxx
> > lane server-atm-address
C5.777777777777777777777777.777777777777.77
> > lane client ethernet xxxxxxx
> > no cdp enable
> > !
> > interface ATM0/0.2 multipoint
> > pvc 0/103
> > encapsulation aal5snap
> > !
> > bridge-group 1
> > !
> > interface Ethernet1/0
> > description
> > no ip address
> > no ip mroute-cache
> > full-duplex
> > no cdp enable
> > bridge-group 1
> > !
> > no ip http server
> > !
> > no cdp run
> > !
> > bridge 1 protocol ieee
> > !
> > dial-peer cor custom
> > !
> > !
> >
> > -----Original Message-----
> > From: Rodney Dunn [mailto:rodunn at cisco.com]
> > Sent: 03 November 2004 16:08
> > To: Alex Foster
> > Cc: cisco-nsp at puck.nether.net
> > Subject: Re: [c-nsp] RE: Unknown reason for High CPU 3620
> >
> > CPU load at interrupt level is almost always packet
> > switching/features or alignment/spurious access corrections.
> >
> > This CPU sounds high for that amount of traffic.
> >
> > Can you list the full configuration /*taking out
> > any confidential stuff*/ so we can see the features
> > enabled?
> >
> > 2nd, you really need to clean up that ethernet
> > segement with all the collisions. Can you convert
> > it to full-duplex? I don't recall if you can do it
> > with the 3620 in later code.
> >
> > Most likely it's not an issue with the ATM side.
> >
> > Rodney
> >
> >
> > On Wed, Nov 03, 2004 at 03:44:07PM -0000, Alex Foster wrote:
> > > Thanks to the guys who responded - I have managed to
narrow down
the
> > > cause to high interrupt utilization and not processor
utilization
-
> > but
> > > Im still a little unclear as to what is causing the high
interrupt
> > util.
> > > - is the ATM interface causing this ? Again the traffic
load is
> > little
> > > over 1.5 meg (in/out) - but constant - so I am wondering
if the
> > > continual SARing is maybe having a negative effect on the
CPU ?
If so
> > -
> > > is there a workaround or am I looking at a bigger platform
to
solve
> > the
> > > issue - if so any recommendations (I have a 3640 available
but
don't
> > > know if this will provide much of an improvement).
> > >
> > >
> > >
> > > Regards
> > >
> > >
> > >
> > > Alex
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Alex Foster
> > > Sent: 03 November 2004 13:37
> > > To: 'cisco-nsp at puck.nether.net'
> > > Subject: Unknown reason for High CPU 3620
> > >
> > >
> > >
> > > I have a 3620 that is currently maxing out its CPU for no
apparent
> > > reason - the traffic load on it at the moment is about 1
to 2 meg
-
> > and
> > > sh proc cpu doesn't indicate any process that is causing
the high
CPU
> > > util. Most of the traffic is SIP but I wouldn't have
thought this
> > would
> > > have any bearing. There are only two interfaces on the
box in use
-
> > an
> > > ATM 155 and a 10bt. The traffic from the 10bt is bridged
to a PVC
on
> > > the ATM interface - nothing complicated.
> > >
> > >
> > >
> > > #sh proc cpu
> > >
> > > CPU utilization for five seconds: 96%/93%; one minute:
90%; five
> > > minutes: 91%
> > >
> > > PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min
TTY
Process
> > >
> > > 1 194708 9899756 19 0.00% 0.00%
0.00%
0
> > > Load Meter
> > >
> > > 2 2008 223 9004 1.41%
0.13%
0.03%
> > > 66 Virtual Exec
> > >
> > >
> > >
> > > #sh mem sum
> > >
> > > Head Total(b) Used(b) Free(b)
Lowest(b)
> > > Largest(b)
> > >
> > > Processor 61D70EA0 12120416 6616992 5503424
5303744
> > > 5305428
> > >
> > > I/O 2900000 7340032 4387068 2952964
2920520
> > > 2947260
> > >
> > >
> > >
> > > #sh int e1/0
> > >
> > > Ethernet1/0 is up, line protocol is up
> > >
> > > Hardware is AmdP2, address is 0005.3245.e070 (bia
0005.3245.e070)
> > >
> > > Description:
> > >
> > > MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
> > >
> > > reliability 255/255, txload 34/255, rxload 37/255
> > >
> > > Encapsulation ARPA, loopback not set
> > >
> > > Keepalive set (10 sec)
> > >
> > > ARP type: ARPA, ARP Timeout 04:00:00
> > >
> > > Last input 00:00:00, output 00:00:00, output hang never
> > >
> > > Last clearing of "show interface" counters 1y4w
> > >
> > > Queueing strategy: fifo
> > >
> > > Output queue 0/40, 0 drops; input queue 0/75, 103522
drops
> > >
> > > 5 minute input rate 1470000 bits/sec, 1984 packets/sec
> > >
> > > 5 minute output rate 1355000 bits/sec, 2166 packets/sec
> > >
> > > 2150894411 packets input, 1646909945 bytes, 0 no
buffer
> > >
> > > Received 88151770 broadcasts, 0 runts, 0 giants, 0
throttles
> > >
> > > 6397644 input errors, 42166 CRC, 32878 frame, 0
overrun,
6355422
> > > ignored
> > >
> > > 0 input packets with dribble condition detected
> > >
> > > 2681876495 packets output, 1769855380 bytes, 0
> > > underruns(556991/1663306/0)
> > >
> > > 165 output errors, 2220297 collisions, 50 interface
resets
> > >
> > > 0 babbles, 47 late collision, 1067254 deferred
> > >
> > > 118 lost carrier, 0 no carrier
> > >
> > > 0 output buffer failures, 0 output buffers swapped
out
> > >
> > >
> > >
> > > #sh int atm 0/0.2
> > >
> > > ATM0/0.2 is up, line protocol is up
> > >
> > > Hardware is RS8234 ATMOC3
> > >
> > > MTU 4470 bytes, BW 155000 Kbit, DLY 80 usec,
> > >
> > > reliability 255/255, txload 2/255, rxload 2/255
> > >
> > > Encapsulation ATM
> > >
> > > 3075506957 packets input, 1094442885139 bytes
> > >
> > > 2362546968 packets output,988686523562 bytes
> > >
> > > 0 OAM cells input, 0 OAM cells output
> > >
> > > AAL5 CRC errors : 81
> > >
> > > AAL5 SAR Timeouts : 0
> > >
> > > AAL5 Oversized SDUs : 0
> > >
> > > AAL5 length violation : 26
> > >
> > > AAL5 CPI Error : 0
> > >
> > >
> > >
> > > Thanks
> > >
> > >
> > >
> > > Alex
> > >
> > >
> > >
> > > The information in this e-mail and any attachments is
confidential
and
> > may be subject to legal professional privilege. It is
intended
solely
> > for the attention and use of the named addressee(s). If you
are not
the
> > intended recipient, or person responsible for delivering
this
> > information to the intended recipient, please notify the
sender
> > immediately. Unless you are the intended recipient or
his/her
> > representative you are prohibited from, and therefore must
not,
read,
> > copy, distribute, use or retain this message or any part of
it. The
> > views expressed in this e-mail may not represent those of
Gamma
Telecom.
> > >
> > > This message has been scanned for viruses by
MailController
> > > _______________________________________________
> > > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >
> >
> > This message has been scanned for viruses by MailController
-
> > www.MailController.altohiway.com
>
_______________________________________________
cisco-nsp mailing list cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
************************************************************************
******************
The information contained in this message, including attachments, may
contain
privileged or confidential information that is intended to be delivered
only to the
person identified above. If you are not the intended recipient, or the
person
responsible for delivering this message to the intended recipient,
ALLTEL requests
that you immediately notify the sender and asks that you do not read the
message or its
attachments, and that you delete them without copying or sending them to
anyone else.
******************************************************************************************
The information contained in this message, including attachments, may contain
privileged or confidential information that is intended to be delivered only to the
person identified above. If you are not the intended recipient, or the person
responsible for delivering this message to the intended recipient, ALLTEL requests
that you immediately notify the sender and asks that you do not read the message or its
attachments, and that you delete them without copying or sending them to anyone else.
More information about the cisco-nsp
mailing list