[cisco-bba] need help on troubleshooting high cpu on 7206NPE300 LNS

Paul Horrocks (phorrock) phorrock at cisco.com
Thu Jan 25 12:02:46 EST 2007


Anthony, 
 
Whilst you wait for the peak period have a look at the below URL's, they
may assist if it does point to re-assembly 
 
http://www.cisco.com/en/US/tech/tk801/tk703/technologies_tech_note09186a
0080094c4f.shtml
http://www.cisco.com/warp/public/105/pmtud_ipfrag.html
 
regards
paul..
 


________________________________

	From: cisco-bba-bounces at puck.nether.net
[mailto:cisco-bba-bounces at puck.nether.net] On Behalf Of Anthony Law
	Sent: Thursday, January 25, 2007 2:25 PM
	To: cisco-bba at puck.nether.net
	Subject: Re: [cisco-bba] need help on troubleshooting high cpu
on 7206NPE300 LNS
	
	
	Hi,
	 
	Thanks for all of your input again. Since this is just the start
of the day, our traffic is low at this time &
	 
	sh proc cpu is showing
	 
	CPU utilization for five seconds: 55%/37%; one minute: 55%; five
minutes: 56%
	   5   484808196 103563445       4681  0.49%  0.64%  0.86%   0
Pool Manager  
	  37  11481426841072956389       1070 17.50% 17.17% 18.04%   0
IP Input 
	
	Below is how >sh ip traffic looks like
	 
	sh ip traffic
	IP statistics:
	  Rcvd:  674456349 total, 3035990691 local destination
	         9258 format errors, 3285179 checksum errors, 6694426
bad hop count
	         2 unknown protocol, 159176 not a gateway
	         0 security failures, 57 bad options, 293393 with
options
	  Opts:  0 end, 148 nop, 615 basic security, 0 loose source
route
	         0 timestamp, 0 extended security, 148 record route
	         0 stream ID, 0 strict source route, 292573 alert, 0
cipso, 0 ump
	         0 other
	  Frags: 3012940604 reassembled, 3424934 timeouts, 118523
couldn't reassemble
	         2998380890 fragmented, 3205560 couldn't fragment
	  Bcast: 5550941 received, 3022 sent
	  Mcast: 0 received, 0 sent
	  Sent:  302118429 generated, 3616922117 forwarded
	  Drop:  6396472 encapsulation failed, 163 unresolved, 0 no
adjacency
	         4485 no route, 0 unicast RPF, 4426667 forced drop
	  Drop:  0 packets with source IP address zero
	ICMP statistics:
	  Rcvd: 10 format errors, 120 checksum errors, 469 redirects,
11499 unreachable
	        3762935 echo, 2838 echo reply, 0 mask requests, 0 mask
replies, 5 quench
	        0 parameter, 65 timestamp, 1 info request, 225 other
	        1 irdp solicitations, 5 irdp advertisements
	  Sent: 246725 redirects, 3280755 unreachable, 3853 echo,
3762867 echo reply
	        0 mask requests, 0 mask replies, 0 quench, 65 timestamp
	        1 info reply, 5222083 time exceeded, 3 parameter problem
	        0 irdp solicitations, 0 irdp advertisements
	UDP statistics:
	  Rcvd: 3031423679 total, 53 checksum errors, 5498341 no port
	  Sent: 289151419 total, 0 forwarded broadcasts
	TCP statistics:
	  Rcvd: 785273 total, 1727 checksum errors, 2886 no port
	  Sent: 450601 total
	Probe statistics:
	  Rcvd: 0 address requests, 0 address replies
	        0 proxy name requests, 0 where-is requests, 0 other
	  Sent: 0 address requests, 0 address replies (0 proxy)
	        0 proxy name replies, 0 where-is replies
	BGP statistics:
	  Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
	        0 keepalives, 0 route-refresh, 0 unrecognized
	  Sent: 0 total, 0 opens, 0 notifications, 0 updates
	        0 keepalives, 0 route-refresh
	EGP statistics:
	  Rcvd: 0 total, 0 format errors, 0 checksum errors, 0 no
listener
	  Sent: 0 total
	IGRP statistics:
	  Rcvd: 0 total, 0 checksum errors
	  Sent: 0 total
	OSPF statistics:
	  Rcvd: 0 total, 0 checksum errors
	        0 hello, 0 database desc, 0 link state req
	        0 link state updates, 0 link state acks
	  Sent: 0 total
	IP-IGRP2 statistics:
	  Rcvd: 0 total
	  Sent: 0 total
	PIMv2 statistics: Sent/Received
	  Total: 0/0, 0 checksum errors, 0 format errors
	  Registers: 0/0, Register Stops: 0/0,  Hellos: 0/0
	  Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
	  Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
	  State-Refresh: 0/0
	IGMP statistics: Sent/Received
	  Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
	  Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0 
	  DVMRP: 0/0, PIM: 0/0
	ARP statistics:
	  Rcvd: 15597477 requests, 294820 replies, 0 reverse, 0 other
	  Sent: 4637290 requests, 27974487 replies (1776972 proxy), 0
reverse
	
	>Are still users connected which received a framed-compression
attribute before you made the change?
	
	 
	After making changes to our radius. I have reset all tunnels
therefore bumped off everyone from their vpdn sess & I have verified
	 that they are not receiving "compression" anymore
	 
	I'll post some more stats during the peak period.
	 
	Thanks.
	 
	Anthony
	
	

________________________________

		Subject: RE: [cisco-bba] need help on troubleshooting
high cpu on 7206NPE300 LNS
		Date: Thu, 25 Jan 2007 10:13:20 +0100
		From: oboehmer at cisco.com
		To: ariev at vayner.net; antnada at hotmail.com;
cisco-bba at puck.nether.net
		
		
		Arie,
		 
		encapsulating/decapsulating L2TP packets should not
happen in IP Input process, this is done in the interrupt path
		 
		Anthony: Something is preventing your interfaces from
interrupt-switching the traffic. Another possibility is packet
re-assembly (which would be shown in "show ip traffic", as Paul just
suggested). Do a "clear counter" and then check "show int stat" which
interface(s) send the majority of pkts in the process path. Are still
users connected which received a framed-compression attribute before you
made the change?
		 
		    oli


________________________________

			From: cisco-bba-bounces at puck.nether.net
[mailto:cisco-bba-bounces at puck.nether.net] On Behalf Of Arie Vayner
			Sent: Thursday, January 25, 2007 8:38 AM
			To: Anthony Law; cisco-bba at puck.nether.net
			Subject: Re: [cisco-bba] need help on
troubleshooting high cpu on 7206NPE300 LNS
			
			


			On 1/25/07, Arie Vayner <ariev at vayner.net>
wrote: 

				Anthony,
				
				The high CPU on IP Input is normal, as
this is where the L2TP work is being done.
				Also note that you have a high rate of
CPU being used in Interrupts (91%/44% means that 44% is used for
Interrupts). Interrupts on Cisco routers are usually linked directly to
a high rate of traffic (on centralized CPU devices). 
				I would assume you box is very close to
its limit of how much traffic it can handle. Could you please send some
of the "show interface" outputs (for the FastEthernet/GigE/ATM ports you
might have). This would allow us to get a better estimation. 
				
				You need to take into account that this
is a centralized CPU platform, and all traffic is handled by the CPU.
This means that the scale factor is not only a question of how many
sessions you have concurrently, but also how much traffic (mostly in PPS
and not BPS) they transmit. 
				
				Thanks
				Arie 
				
				
				
				On 1/25/07, Anthony Law <
antnada at hotmail.com <mailto:antnada at hotmail.com> > wrote: 


				Dear all
				 
				Thank you for all of your input. I
configured vpdn ip udp ignore checksum
				& I have corrected a mis-config on our
radius server (passing compression attribute to cisco) now that the L2TP
data daemon is running normal, but I am still facing high cpu on Pool
Manager & IP Input
				anymore suggestions?
				 
				 
				CPU utilization for five seconds:
91%/44%; one minute: 91%; five minutes: 86%
				 PID Runtime(ms)   Invoked      uSecs
5Sec   1Min   5Min TTY Process 
				   1           4       175         22
0.00%  0.00%  0.00%   0 Chunk Manager    
				   2      487964   5014024         97
0.00%  0.00%  0.00%   0 Load Meter       
				   3     1606476    870141       1846
0.00%  0.00%  0.00%   0 CEF Scanner      
				   4    22428792   3318958       6757
0.00%  0.06%   0.05%   0 Check heaps      
				   5   481842360 102963163       4679
9.05%  9.70%  7.90%   0 Pool Manager     
				  37  11275060121049358292       1074
36.02% 35.07% 32.40%   0 IP Input 
				
				Thank You
				 
				Anthony
				
				
				
				
________________________________


				> Date: Wed, 24 Jan 2007 02:37:10 +0200
				> From: nitzan.tzelniker at gmail.com
				> To: antnada at hotmail.com
				> Subject: Re: [cisco-bba] need help on
troubleshooting high cpu on 7206 NPE300 LNS
				> CC: cisco-bba at puck.nether.net
				> 
				> You can try
				> 
				> vpdn ip udp ignore checksum
				> 
				> Nitzan
				> 
				> On 1/24/07, Anthony Law <
antnada at hotmail.com <mailto:antnada at hotmail.com> > wrote:
				> > Dear all,
				> >
				> > We have a 7206 w/NPE300 running as a
LNS terminating pppoe sessions from our
				> > telco. We are concurrently running
around 360 pppoe sessions. 
				> >
				> > Recently. I noticed that our 7206 is
having extremely high cpu, at times
				> > going to 100%, please see below
				> >
				> > CPU utilization for five seconds:
99%/42%; one minute: 99%; five minutes: 
				> > 99%
				> > PID Runtime(ms) Invoked uSecs 5Sec
1Min 5Min TTY Process
				> > 1 0 75 0 0.00% 0.00% 0.00% 0 Chunk
Manager
				> >
				> >
				> > 5 472509060 101324023 4663 7.65%
8.80% 8.84% 0 Pool Manager
				> >
				> > 37 10810547881019294234 1060 22.79%
25.16% 25.51% 0 IP Input
				> >
				> >
				> > 101 705044020 800103660 881 18.89%
21.35% 19.34% 0 L2TP data
				> > daemon 
				> > 102 53153196 10197928 5212 2.19%
0.46% 0.45% 0 L2TP mgmt
				> > daemon
				> >
				> >
				> > It seemed that Pool Manager + IP
Input + L2TP data daemon together is
				> > causing this issue. I was searching
for documents regarding this on google 
				> > and came to this mailing list. I am
wondering if you guys can help me out by
				> > identifying the mis-configuration
that I have on my end as it is my
				> > understanding that a 7206 should at
least take close 1000 pppoe sessions. 
				> > Thank You in advance for your input.
				> >
				> >
				> > hostname LNS
				> > !
				> > boot system
slot1:c7200-is-mz.122-32.bin
				> > boot system
slot1:c7200-is-mz.120-3.T3 
				> > aaa new-model
				> > aaa authentication login default
local
				> > aaa authentication login no_rad line
				> > aaa authentication ppp default group
radius local
				> > aaa authentication ppp vpdn group
radius 
				> > aaa authorization network default
group radius
				> > aaa authorization configuration
default group radius
				> > aaa accounting delay-start
				> > aaa accounting exec default
start-stop group radius 
				> > aaa accounting network default
start-stop group radius
				> > enable secret 5
XXXXXXXXXXXXXXXXXXXXXXXXXXX
				> > !
				> > clock timezone EST -5
				> > clock summer-time EDT recurring 
				> > ip subnet-zero
				> > no ip source-route
				> > ip cef
				> > !
				> > !
				> > ip name-server XXXXXX
				> > ip name-server XXXXXX
				> > ip name-server XXXXXX
				> > ! 
				> > vpdn enable
				> > !
				> > vpdn-group XXXXXXXX
				> > accept-dialin
				> > protocol l2tp
				> > virtual-template 1
				> > terminate-from hostname XXXXXX
				> > local name XXXXXXX 
				> > lcp renegotiation always
				> > !
				> > interface FastEthernet0/0
				> > ip address X.X.X.X 255.255.255.192
<http://255.255.255.192/> 
				> > no ip mroute-cache
				> > duplex full
				> > !
				> > interface FastEthernet1/0
				> > no ip address
				> > no ip mroute-cache
				> > duplex full
				> > ! 
				> > interface FastEthernet1/0.401
				> > description
!!XXXXXXXXXXXXXXXXXXXXXXXX!!
				> > encapsulation dot1Q 401
				> > ip address 10.70.X.X 255.255.255.252
<http://255.255.255.252/> 
				> > no ip mroute-cache
				> > !
				> > interface FastEthernet2/0
				> > description !!Internet Feed!!
				> > ip address Y.Y.Y.Y 255.255.255.252
<http://255.255.255.252/> 
				> > no ip mroute-cache
				> > duplex full
				> > !
				> > interface Virtual-Template1
				> > mtu 1492
				> > ip unnumbered FastEthernet2/0
				> > peer default ip address pool
internet1 internet2 
				> > ppp authentication pap vpdn
				> > !
				> > ip local pool internet1 A.A.A.A
B.B.B.B
				> > ip local pool internet2 C.C.C.C
D.D.D.D
				> > ip classless
				> > ip route 0.0.0.0 <http://0.0.0.0/>
0.0.0.0 <http://0.0.0.0/>  Y.Y.Y.Y
				> > no ip http server
				> > !
				> > ip radius source-interface
FastEthernet0/0 
				> > radius-server host X.X.X.X auth-port
1645 acct-port 1646
				> > radius-server host X.X.X.X auth-port
1645 acct-port 1646
				> > radius-server key 7 ZZZZZZZZZZZZZZZ
				> >
				> > Anthony 
				> >
				> > ________________________________
				> > Be one of the first to try Windows
Live Mail.
				> >
_______________________________________________
				> > cisco-bba mailing list
				> > cisco-bba at puck.nether.net
				> >
https://puck.nether.net/mailman/listinfo/cisco-bba
				> >
				> >
				> >
				
				
				
________________________________

				Be one of the first to try Windows Live
Mail.
<http://ideas.live.com/programpage.aspx?versionId=5d21c51a-b161-4314-9b0
e-4911fb2b2e6d> 

	
_______________________________________________
				cisco-bba mailing list
				cisco-bba at puck.nether.net 
	
https://puck.nether.net/mailman/listinfo/cisco-bba
				
				
				




________________________________

	Be one of the first to try Windows Live Mail.
<http://ideas.live.com/programpage.aspx?versionId=5d21c51a-b161-4314-9b0
e-4911fb2b2e6d>  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://puck.nether.net/pipermail/cisco-bba/attachments/20070125/09e85507/attachment-0001.html 


More information about the cisco-bba mailing list