<html>
<head>
<style>
P
{
margin:0px;
padding:0px
}
body
{
FONT-SIZE: 10pt;
FONT-FAMILY:Tahoma
}
</style>
</head>
<body><BR>Thank You Aire & Paul for your suggestion.<BR>
<BR>
I have changed mtu size to lower at 1460 & I shall read upon on the docs again. <BR>
<BR>
Anthony<BR><BR><BR><BR><BR>
<BLOCKQUOTE style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #008080 2px solid; MARGIN-RIGHT: 0px">
<HR id=EC_stopSpelling>
Subject: RE: [cisco-bba] need help on troubleshooting high cpu on 7206NPE300 LNS<BR>Date: Thu, 25 Jan 2007 18:02:46 +0100<BR>From: phorrock@cisco.com<BR>To: antnada@hotmail.com; cisco-bba@puck.nether.net<BR><BR>
<META content="Microsoft SafeHTML" name=Generator>
<STYLE>
.ExternalClass P
{padding:0px;}
.ExternalClass
{font-size:10pt;font-family:Tahoma;}
</STYLE>
<DIV dir=ltr align=left><SPAN class=EC_068225216-25012007>Anthony, </SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=EC_068225216-25012007></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=EC_068225216-25012007>Whilst you wait for the peak period have a look at the below URL's, they may assist if it does point to re-assembly </SPAN></DIV>
<DIV dir=ltr align=left> </DIV>
<DIV dir=ltr align=left><A href="http://www.cisco.com/en/US/tech/tk801/tk703/technologies_tech_note09186a0080094c4f.shtml" target=_blank>http://www.cisco.com/en/US/tech/tk801/tk703/technologies_tech_note09186a0080094c4f.shtml</A></DIV>
<DIV dir=ltr align=left><A href="http://www.cisco.com/warp/public/105/pmtud_ipfrag.html" target=_blank>http://www.cisco.com/warp/public/105/pmtud_ipfrag.html</A></DIV>
<DIV dir=ltr align=left> </DIV>
<DIV dir=ltr align=left><SPAN class=EC_068225216-25012007>regards</SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=EC_068225216-25012007>paul..</SPAN></DIV>
<DIV dir=ltr align=left> </DIV><BR>
<BLOCKQUOTE style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
<DIV class=EC_OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR>
<B>From:</B> cisco-bba-bounces@puck.nether.net [mailto:cisco-bba-bounces@puck.nether.net] <B>On Behalf Of </B>Anthony Law<BR><B>Sent:</B> Thursday, January 25, 2007 2:25 PM<BR><B>To:</B> cisco-bba@puck.nether.net<BR><B>Subject:</B> Re: [cisco-bba] need help on troubleshooting high cpu on 7206NPE300 LNS<BR><BR></DIV>
<DIV></DIV>Hi,<BR> <BR>Thanks for all of your input again. Since this is just the start of the day, our traffic is low at this time &<BR> <BR>sh proc cpu is showing<BR> <BR>CPU utilization for five seconds: 55%/37%; one minute: 55%; five minutes: 56%<BR> 5 484808196 103563445 4681 0.49% 0.64% 0.86% 0 Pool Manager <BR> 37 11481426841072956389 1070 17.50% 17.17% 18.04% 0 IP Input <BR><BR>Below is how >sh ip traffic looks like<BR> <BR>sh ip traffic<BR>IP statistics:<BR> Rcvd: 674456349 total, 3035990691 local destination<BR> 9258 format errors, 3285179 checksum errors, 6694426 bad hop count<BR> 2 unknown protocol, 159176 not a gateway<BR> 0 security failures, 57 bad options, 293393 with options<BR> Opts: 0 end, 148 nop, 615 basic security, 0 loose source route<BR> 0 timestamp, 0 extended security, 148 record route<BR> 0 stream ID, 0 strict source route, 292573 alert, 0 cipso, 0 ump<BR> 0 other<BR> Frags: 3012940604 reassembled, 3424934 timeouts, 118523 couldn't reassemble<BR> 2998380890 fragmented, 3205560 couldn't fragment<BR> Bcast: 5550941 received, 3022 sent<BR> Mcast: 0 received, 0 sent<BR> Sent: 302118429 generated, 3616922117 forwarded<BR> Drop: 6396472 encapsulation failed, 163 unresolved, 0 no adjacency<BR> 4485 no route, 0 unicast RPF, 4426667 forced drop<BR> Drop: 0 packets with source IP address zero<BR>ICMP statistics:<BR> Rcvd: 10 format errors, 120 checksum errors, 469 redirects, 11499 unreachable<BR> 3762935 echo, 2838 echo reply, 0 mask requests, 0 mask replies, 5 quench<BR> 0 parameter, 65 timestamp, 1 info request, 225 other<BR> 1 irdp solicitations, 5 irdp advertisements<BR> Sent: 246725 redirects, 3280755 unreachable, 3853 echo, 3762867 echo reply<BR> 0 mask requests, 0 mask replies, 0 quench, 65 timestamp<BR> 1 info reply, 5222083 time exceeded, 3 parameter problem<BR> 0 irdp solicitations, 0 irdp advertisements<BR>UDP statistics:<BR> Rcvd: 3031423679 total, 53 checksum errors, 5498341 no port<BR> Sent: 289151419 total, 0 forwarded broadcasts<BR>TCP statistics:<BR> Rcvd: 785273 total, 1727 checksum errors, 2886 no port<BR> Sent: 450601 total<BR>Probe statistics:<BR> Rcvd: 0 address requests, 0 address replies<BR> 0 proxy name requests, 0 where-is requests, 0 other<BR> Sent: 0 address requests, 0 address replies (0 proxy)<BR> 0 proxy name replies, 0 where-is replies<BR>BGP statistics:<BR> Rcvd: 0 total, 0 opens, 0 notifications, 0 updates<BR> 0 keepalives, 0 route-refresh, 0 unrecognized<BR> Sent: 0 total, 0 opens, 0 notifications, 0 updates<BR> 0 keepalives, 0 route-refresh<BR>EGP statistics:<BR> Rcvd: 0 total, 0 format errors, 0 checksum errors, 0 no listener<BR> Sent: 0 total<BR>IGRP statistics:<BR> Rcvd: 0 total, 0 checksum errors<BR> Sent: 0 total<BR>OSPF statistics:<BR> Rcvd: 0 total, 0 checksum errors<BR> 0 hello, 0 database desc, 0 link state req<BR> 0 link state updates, 0 link state acks<BR> Sent: 0 total<BR>IP-IGRP2 statistics:<BR> Rcvd: 0 total<BR> Sent: 0 total<BR>PIMv2 statistics: Sent/Received<BR> Total: 0/0, 0 checksum errors, 0 format errors<BR> Registers: 0/0, Register Stops: 0/0, Hellos: 0/0<BR> Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0<BR> Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0<BR> State-Refresh: 0/0<BR>IGMP statistics: Sent/Received<BR> Total: 0/0, Format errors: 0/0, Checksum errors: 0/0<BR> Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0 <BR> DVMRP: 0/0, PIM: 0/0<BR>ARP statistics:<BR> Rcvd: 15597477 requests, 294820 replies, 0 reverse, 0 other<BR> Sent: 4637290 requests, 27974487 replies (1776972 proxy), 0 reverse<BR><BR>><FONT face=Arial>Are still users connected which received a framed-compression attribute before you made the change?</FONT><BR>
<DIV dir=ltr align=left><SPAN class=EC_EC_600440108-25012007></SPAN> </DIV>After making changes to our radius. I have reset all tunnels therefore bumped off everyone from their vpdn sess & I have verified<BR> that they are not receiving "compression" anymore<BR> <BR>I'll post some more stats during the peak period.<BR> <BR>Thanks.<BR> <BR>Anthony<BR><BR>
<BLOCKQUOTE style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #008080 2px solid; MARGIN-RIGHT: 0px">
<HR id=EC_EC_stopSpelling>
Subject: RE: [cisco-bba] need help on troubleshooting high cpu on 7206NPE300 LNS<BR>Date: Thu, 25 Jan 2007 10:13:20 +0100<BR>From: oboehmer@cisco.com<BR>To: ariev@vayner.net; antnada@hotmail.com; cisco-bba@puck.nether.net<BR><BR>
<DIV dir=ltr align=left><SPAN class=EC_EC_600440108-25012007><FONT face=Arial>Arie,</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=EC_EC_600440108-25012007><FONT face=Arial></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=EC_EC_600440108-25012007><FONT face=Arial>encapsulating/decapsulating L2TP packets should not happen in IP Input process, this is done in the interrupt path</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=EC_EC_600440108-25012007><FONT face=Arial></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=EC_EC_600440108-25012007><FONT face=Arial>Anthony: S</FONT></SPAN><SPAN class=EC_EC_600440108-25012007><FONT face=Arial>omething is preventing your interfaces from interrupt-switching the traffic. Another possibility is packet re-assembly (which would be shown in "show ip traffic", as Paul just suggested). Do a "clear counter" and then check "show int stat" which interface(s) send the majority of pkts in the process path. Are still users connected which received a framed-compression attribute before you made the change?</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=EC_EC_600440108-25012007><FONT face=Arial></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=EC_EC_600440108-25012007> <FONT face=Arial>oli</FONT></SPAN></DIV><BR>
<BLOCKQUOTE style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
<DIV class=EC_EC_OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR>
<FONT face=Tahoma><B>From:</B> cisco-bba-bounces@puck.nether.net [mailto:cisco-bba-bounces@puck.nether.net] <B>On Behalf Of </B>Arie Vayner<BR><B>Sent:</B> Thursday, January 25, 2007 8:38 AM<BR><B>To:</B> Anthony Law; cisco-bba@puck.nether.net<BR><B>Subject:</B> Re: [cisco-bba] need help on troubleshooting high cpu on 7206NPE300 LNS<BR></FONT><BR></DIV>
<DIV></DIV><BR><BR>
<DIV><SPAN class=EC_EC_gmail_quote>On 1/25/07, <B class=EC_EC_gmail_sendername>Arie Vayner</B> <<A href="mailto:ariev@vayner.net">ariev@vayner.net</A>> wrote:</SPAN>
<BLOCKQUOTE class=EC_EC_gmail_quote style="PADDING-LEFT: 1ex; BORDER-LEFT: rgb(204,204,204) 1px solid">Anthony,<BR><BR>The high CPU on IP Input is normal, as this is where the L2TP work is being done.<BR>Also note that you have a high rate of CPU being used in Interrupts (91%/44% means that 44% is used for Interrupts). Interrupts on Cisco routers are usually linked directly to a high rate of traffic (on centralized CPU devices). <BR>I would assume you box is very close to its limit of how much traffic it can handle. Could you please send some of the "show interface" outputs (for the FastEthernet/GigE/ATM ports you might have). This would allow us to get a better estimation. <BR><BR>You need to take into account that this is a centralized CPU platform, and all traffic is handled by the CPU. This means that the scale factor is not only a question of how many sessions you have concurrently, but also how much traffic (mostly in PPS and not BPS) they transmit. <BR><BR>Thanks<BR><SPAN class=EC_EC_sg>Arie</SPAN>
<DIV><SPAN class=EC_EC_e id=EC_EC_q_110583013859d0e2_2><BR><BR>
<DIV><SPAN class=EC_EC_gmail_quote>On 1/25/07, <B class=EC_EC_gmail_sendername>Anthony Law</B> <<A href="mailto:antnada@hotmail.com"> antnada@hotmail.com</A>> wrote:</SPAN>
<BLOCKQUOTE class=EC_EC_gmail_quote style="PADDING-LEFT: 1ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV><BR>Dear all<BR> <BR>Thank you for all of your input. I configured vpdn ip udp ignore checksum<BR>& I have corrected a mis-config on our radius server (passing compression attribute to cisco) now that the L2TP data daemon is running normal, but I am still facing high cpu on Pool Manager & IP Input<BR>anymore suggestions?<BR> <BR> <BR>CPU utilization for five seconds: 91%/44%; one minute: 91%; five minutes: 86%<BR> PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process <BR> 1 4 175 22 0.00% 0.00% 0.00% 0 Chunk Manager <BR> 2 487964 5014024 97 0.00% 0.00% 0.00% 0 Load Meter <BR> 3 1606476 870141 1846 0.00% 0.00% 0.00% 0 CEF Scanner <BR> 4 22428792 3318958 6757 0.00% 0.06% 0.05% 0 Check heaps <BR> 5 481842360 102963163 4679 9.05% 9.70% 7.90% 0 Pool Manager <BR> 37 11275060121049358292 1074 36.02% 35.07% 32.40% 0 IP Input <BR><BR>Thank You<BR> <BR>Anthony<BR><BR><BR><BR>
<HR>
<BR>> Date: Wed, 24 Jan 2007 02:37:10 +0200<BR>> From: <A href="mailto:nitzan.tzelniker@gmail.com">nitzan.tzelniker@gmail.com</A><BR>> To: <A href="mailto:antnada@hotmail.com">antnada@hotmail.com</A><BR>> Subject: Re: [cisco-bba] need help on troubleshooting high cpu on 7206 NPE300 LNS<BR>> CC: <A href="mailto:cisco-bba@puck.nether.net">cisco-bba@puck.nether.net</A><BR>> <BR>> You can try<BR>> <BR>> vpdn ip udp ignore checksum<BR>> <BR>> Nitzan<BR>> <BR>> On 1/24/07, Anthony Law <<A href="mailto:antnada@hotmail.com"> antnada@hotmail.com</A>> wrote:<BR>> > Dear all,<BR>> ><BR>> > We have a 7206 w/NPE300 running as a LNS terminating pppoe sessions from our<BR>> > telco. We are concurrently running around 360 pppoe sessions. <BR>> ><BR>> > Recently. I noticed that our 7206 is having extremely high cpu, at times<BR>> > going to 100%, please see below<BR>> ><BR>> > CPU utilization for five seconds: 99%/42%; one minute: 99%; five minutes: <BR>> > 99%<BR>> > PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process<BR>> > 1 0 75 0 0.00% 0.00% 0.00% 0 Chunk Manager<BR>> ><BR>> ><BR>> > 5 472509060 101324023 4663 7.65% 8.80% 8.84% 0 Pool Manager<BR>> ><BR>> > 37 10810547881019294234 1060 22.79% 25.16% 25.51% 0 IP Input<BR>> ><BR>> ><BR>> > 101 705044020 800103660 881 18.89% 21.35% 19.34% 0 L2TP data<BR>> > daemon <BR>> > 102 53153196 10197928 5212 2.19% 0.46% 0.45% 0 L2TP mgmt<BR>> > daemon<BR>> ><BR>> ><BR>> > It seemed that Pool Manager + IP Input + L2TP data daemon together is<BR>> > causing this issue. I was searching for documents regarding this on google <BR>> > and came to this mailing list. I am wondering if you guys can help me out by<BR>> > identifying the mis-configuration that I have on my end as it is my<BR>> > understanding that a 7206 should at least take close 1000 pppoe sessions. <BR>> > Thank You in advance for your input.<BR>> ><BR>> ><BR>> > hostname LNS<BR>> > !<BR>> > boot system slot1:c7200-is-mz.122-32.bin<BR>> > boot system slot1:c7200-is-mz.120-3.T3 <BR>> > aaa new-model<BR>> > aaa authentication login default local<BR>> > aaa authentication login no_rad line<BR>> > aaa authentication ppp default group radius local<BR>> > aaa authentication ppp vpdn group radius <BR>> > aaa authorization network default group radius<BR>> > aaa authorization configuration default group radius<BR>> > aaa accounting delay-start<BR>> > aaa accounting exec default start-stop group radius <BR>> > aaa accounting network default start-stop group radius<BR>> > enable secret 5 XXXXXXXXXXXXXXXXXXXXXXXXXXX<BR>> > !<BR>> > clock timezone EST -5<BR>> > clock summer-time EDT recurring <BR>> > ip subnet-zero<BR>> > no ip source-route<BR>> > ip cef<BR>> > !<BR>> > !<BR>> > ip name-server XXXXXX<BR>> > ip name-server XXXXXX<BR>> > ip name-server XXXXXX<BR>> > ! <BR>> > vpdn enable<BR>> > !<BR>> > vpdn-group XXXXXXXX<BR>> > accept-dialin<BR>> > protocol l2tp<BR>> > virtual-template 1<BR>> > terminate-from hostname XXXXXX<BR>> > local name XXXXXXX <BR>> > lcp renegotiation always<BR>> > !<BR>> > interface FastEthernet0/0<BR>> > ip address X.X.X.X <A href="http://255.255.255.192/" target=_blank>255.255.255.192</A><BR>> > no ip mroute-cache<BR>> > duplex full<BR>> > !<BR>> > interface FastEthernet1/0<BR>> > no ip address<BR>> > no ip mroute-cache<BR>> > duplex full<BR>> > ! <BR>> > interface FastEthernet1/0.401<BR>> > description !!XXXXXXXXXXXXXXXXXXXXXXXX!!<BR>> > encapsulation dot1Q 401<BR>> > ip address 10.70.X.X <A href="http://255.255.255.252/" target=_blank>255.255.255.252</A><BR>> > no ip mroute-cache<BR>> > !<BR>> > interface FastEthernet2/0<BR>> > description !!Internet Feed!!<BR>> > ip address Y.Y.Y.Y <A href="http://255.255.255.252/" target=_blank>255.255.255.252</A><BR>> > no ip mroute-cache<BR>> > duplex full<BR>> > !<BR>> > interface Virtual-Template1<BR>> > mtu 1492<BR>> > ip unnumbered FastEthernet2/0<BR>> > peer default ip address pool internet1 internet2 <BR>> > ppp authentication pap vpdn<BR>> > !<BR>> > ip local pool internet1 A.A.A.A B.B.B.B<BR>> > ip local pool internet2 C.C.C.C D.D.D.D<BR>> > ip classless<BR>> > ip route <A href="http://0.0.0.0/" target=_blank>0.0.0.0</A> <A href="http://0.0.0.0/" target=_blank>0.0.0.0</A> Y.Y.Y.Y<BR>> > no ip http server<BR>> > !<BR>> > ip radius source-interface FastEthernet0/0 <BR>> > radius-server host X.X.X.X auth-port 1645 acct-port 1646<BR>> > radius-server host X.X.X.X auth-port 1645 acct-port 1646<BR>> > radius-server key 7 ZZZZZZZZZZZZZZZ<BR>> ><BR>> > Anthony <BR>> ><BR>> > ________________________________<BR>> > Be one of the first to try Windows Live Mail.<BR>> > _______________________________________________<BR>> > cisco-bba mailing list<BR>> > <A href="mailto:cisco-bba@puck.nether.net">cisco-bba@puck.nether.net</A><BR>> > <A href="https://puck.nether.net/mailman/listinfo/cisco-bba" target=_blank>https://puck.nether.net/mailman/listinfo/cisco-bba</A><BR>> ><BR>> ><BR>> ><BR><BR><BR>
<HR>
Be one of the first to try <A href="http://ideas.live.com/programpage.aspx?versionId=5d21c51a-b161-4314-9b0e-4911fb2b2e6d" target=_blank>Windows Live Mail.</A></DIV><BR>_______________________________________________<BR>cisco-bba mailing list<BR><A href="mailto:cisco-bba@puck.nether.net">cisco-bba@puck.nether.net </A><BR><A href="https://puck.nether.net/mailman/listinfo/cisco-bba" target=_blank>https://puck.nether.net/mailman/listinfo/cisco-bba</A><BR><BR><BR></BLOCKQUOTE></DIV><BR></SPAN></DIV></BLOCKQUOTE></DIV><BR></BLOCKQUOTE></BLOCKQUOTE><BR>
<HR>
Be one of the first to try <A href="http://ideas.live.com/programpage.aspx?versionId=5d21c51a-b161-4314-9b0e-4911fb2b2e6d" target=_blank>Windows Live Mail.</A> </BLOCKQUOTE></BLOCKQUOTE><br /><hr />Be one of the first to try Windows Live Mail. <a href='http://ideas.live.com/programpage.aspx?versionId=5d21c51a-b161-4314-9b0e-4911fb2b2e6d' target='_new'>Windows Live Mail.</a></body>
</html>