Anthony,<br><br>Any reason why would you get so much Fragments?<br>Can you describe the path that the L2TP traffic takes from the LNS to the LAC?<br>It might be interesting for you to implement MSS Adjust on the Virtual-Template...
<br><br>Arie<br><br><div><span class="gmail_quote">On 1/25/07, <b class="gmail_sendername">Anthony Law</b> <<a href="mailto:antnada@hotmail.com">antnada@hotmail.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>Hi,<br>
<br>
Thanks for all of your input again. Since this is just the start of the day, our traffic is low at this time &<br>
<br>
sh proc cpu is showing<br>
<br>
CPU utilization for five seconds: 55%/37%; one minute: 55%; five minutes: 56%<br>
5 484808196 103563445 4681 0.49% 0.64% 0.86% 0 Pool Manager <br> 37 11481426841072956389 1070 17.50% 17.17% 18.04% 0 IP Input <br><br>Below is how >sh ip traffic looks like<br>
<br>
sh ip traffic<br>IP statistics:<br> Rcvd: 674456349 total, 3035990691 local destination<br> 9258 format errors, 3285179 checksum errors, 6694426 bad hop count<br> 2 unknown protocol, 159176 not a gateway
<br> 0 security failures, 57 bad options, 293393 with options<br> Opts: 0 end, 148 nop, 615 basic security, 0 loose source route<br> 0 timestamp, 0 extended security, 148 record route<br> 0 stream ID, 0 strict source route, 292573 alert, 0 cipso, 0 ump
<br> 0 other<br> Frags: 3012940604 reassembled, 3424934 timeouts, 118523 couldn't reassemble<br> 2998380890 fragmented, 3205560 couldn't fragment<br> Bcast: 5550941 received, 3022 sent<br> Mcast: 0 received, 0 sent
<br> Sent: 302118429 generated, 3616922117 forwarded<br> Drop: 6396472 encapsulation failed, 163 unresolved, 0 no adjacency<br> 4485 no route, 0 unicast RPF, 4426667 forced drop<br> Drop: 0 packets with source IP address zero
<br>
ICMP statistics:<br> Rcvd: 10 format errors, 120 checksum errors, 469 redirects, 11499 unreachable<br> 3762935 echo, 2838 echo reply, 0 mask requests, 0 mask replies, 5 quench<br> 0 parameter, 65 timestamp, 1 info request, 225 other
<br> 1 irdp solicitations, 5 irdp advertisements<br> Sent: 246725 redirects, 3280755 unreachable, 3853 echo, 3762867 echo reply<br> 0 mask requests, 0 mask replies, 0 quench, 65 timestamp<br> 1 info reply, 5222083 time exceeded, 3 parameter problem
<br> 0 irdp solicitations, 0 irdp advertisements<br>
UDP statistics:<br> Rcvd: 3031423679 total, 53 checksum errors, 5498341 no port<br> Sent: 289151419 total, 0 forwarded broadcasts<br>
TCP statistics:<br> Rcvd: 785273 total, 1727 checksum errors, 2886 no port<br> Sent: 450601 total<br>
Probe statistics:<br> Rcvd: 0 address requests, 0 address replies<br> 0 proxy name requests, 0 where-is requests, 0 other<br> Sent: 0 address requests, 0 address replies (0 proxy)<br> 0 proxy name replies, 0 where-is replies
<br>
BGP statistics:<br> Rcvd: 0 total, 0 opens, 0 notifications, 0 updates<br> 0 keepalives, 0 route-refresh, 0 unrecognized<br> Sent: 0 total, 0 opens, 0 notifications, 0 updates<br> 0 keepalives, 0 route-refresh
<br>
EGP statistics:<br> Rcvd: 0 total, 0 format errors, 0 checksum errors, 0 no listener<br> Sent: 0 total<br>
IGRP statistics:<br> Rcvd: 0 total, 0 checksum errors<br> Sent: 0 total<br>
OSPF statistics:<br> Rcvd: 0 total, 0 checksum errors<br> 0 hello, 0 database desc, 0 link state req<br> 0 link state updates, 0 link state acks<br>
Sent: 0 total<br>
IP-IGRP2 statistics:<br> Rcvd: 0 total<br> Sent: 0 total<br>
PIMv2 statistics: Sent/Received<br> Total: 0/0, 0 checksum errors, 0 format errors<br> Registers: 0/0, Register Stops: 0/0, Hellos: 0/0<br> Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0<br> Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
<br> State-Refresh: 0/0<br>
IGMP statistics: Sent/Received<br> Total: 0/0, Format errors: 0/0, Checksum errors: 0/0<br> Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0 <br> DVMRP: 0/0, PIM: 0/0<br>
ARP statistics:<br> Rcvd: 15597477 requests, 294820 replies, 0 reverse, 0 other<br> Sent: 4637290 requests, 27974487 replies (1776972 proxy), 0 reverse<span class="q"><br><br>><font face="Arial">Are still users connected which received a framed-compression attribute before you made the change?
</font><br>
<div dir="ltr" align="left"><span><font size="2"></font></span> </div></span>
After making changes to our radius. I have reset all tunnels therefore bumped off everyone from their vpdn sess & I have verified<br>
that they are not receiving "compression" anymore<br>
<br>
I'll post some more stats during the peak period.<br>
<br>
Thanks.<br><span class="sg">
<br>
Anthony<br></span><span class="ad"><br>
</span><blockquote style="border-left: 2px solid rgb(0, 128, 128); padding-left: 5px; margin-left: 5px; margin-right: 0px;"><span class="ad">
<hr>
Subject: RE: [cisco-bba] need help on troubleshooting high cpu on 7206NPE300 LNS<br>Date: Thu, 25 Jan 2007 10:13:20 +0100<br>From: <a href="mailto:oboehmer@cisco.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
oboehmer@cisco.com</a><br>To: <a href="mailto:ariev@vayner.net" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">ariev@vayner.net</a>; <a href="mailto:antnada@hotmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
antnada@hotmail.com</a>; <a href="mailto:cisco-bba@puck.nether.net" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">cisco-bba@puck.nether.net</a></span><div><span class="e" id="q_11059a7afd42a332_7">
<br><br>
<div dir="ltr" align="left"><span><font face="Arial" size="2">Arie,</font></span></div>
<div dir="ltr" align="left"><span><font face="Arial" size="2"></font></span> </div>
<div dir="ltr" align="left"><span><font face="Arial" size="2">encapsulating/decapsulating L2TP packets should not happen in IP Input process, this is done in the interrupt path</font></span></div>
<div dir="ltr" align="left"><span><font face="Arial" size="2"></font></span> </div>
<div dir="ltr" align="left"><span><font face="Arial" size="2">Anthony: S</font></span><span><font face="Arial" size="2">omething is preventing your interfaces from interrupt-switching the traffic. Another possibility is packet re-assembly (which would be shown in "show ip traffic", as Paul just suggested). Do a "clear counter" and then check "show int stat" which interface(s) send the majority of pkts in the process path. Are still users connected which received a framed-compression attribute before you made the change?
</font></span></div>
<div dir="ltr" align="left"><span><font face="Arial" size="2"></font></span> </div>
<div dir="ltr" align="left"><span> <font face="Arial" size="2">oli</font></span></div><br>
<blockquote style="border-left: 2px solid rgb(0, 0, 0); padding-left: 5px; margin-left: 5px; margin-right: 0px;">
<div dir="ltr" align="left" lang="en-us">
<hr>
<font face="Tahoma" size="2"><b>From:</b> <a href="mailto:cisco-bba-bounces@puck.nether.net" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">cisco-bba-bounces@puck.nether.net</a> [mailto:<a href="mailto:cisco-bba-bounces@puck.nether.net" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
cisco-bba-bounces@puck.nether.net</a>] <b>On Behalf Of </b>Arie Vayner<br><b>Sent:</b> Thursday, January 25, 2007 8:38 AM<br><b>To:</b> Anthony Law; <a href="mailto:cisco-bba@puck.nether.net" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
cisco-bba@puck.nether.net</a><br><b>Subject:</b> Re: [cisco-bba] need help on troubleshooting high cpu on 7206NPE300 LNS<br></font><br></div>
<div></div><br><br>
<div><span>On 1/25/07, <b>Arie Vayner</b> <<a href="mailto:ariev@vayner.net" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">ariev@vayner.net</a>> wrote:</span>
<blockquote style="border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Anthony,<br><br>The high CPU on IP Input is normal, as this is where the L2TP work is being done.<br>Also note that you have a high rate of CPU being used in Interrupts (91%/44% means that 44% is used for Interrupts). Interrupts on Cisco routers are usually linked directly to a high rate of traffic (on centralized CPU devices).
<br>I would assume you box is very close to its limit of how much traffic it can handle. Could you please send some of the "show interface" outputs (for the FastEthernet/GigE/ATM ports you might have). This would allow us to get a better estimation.
<br><br>You need to take into account that this is a centralized CPU platform, and all traffic is handled by the CPU. This means that the scale factor is not only a question of how many sessions you have concurrently, but also how much traffic (mostly in PPS and not BPS) they transmit.
<br><br>Thanks<br><span>Arie</span>
<div><span><br><br>
<div><span>On 1/25/07, <b>Anthony Law</b> <<a href="mailto:antnada@hotmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> antnada@hotmail.com</a>> wrote:</span>
<blockquote style="border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div><br>Dear all<br> <br>Thank you for all of your input. I configured vpdn ip udp ignore checksum<br>& I have corrected a mis-config on our radius server (passing compression attribute to cisco) now that the L2TP data daemon is running normal, but I am still facing high cpu on Pool Manager & IP Input
<br>anymore suggestions?<br> <br> <br>CPU utilization for five seconds: 91%/44%; one minute: 91%; five minutes: 86%<br> PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process <br> 1 4 175 22
0.00% 0.00% 0.00% 0 Chunk Manager <br> 2 487964 5014024 97 0.00% 0.00% 0.00% 0 Load Meter <br> 3 1606476 870141 1846 0.00% 0.00% 0.00% 0 CEF Scanner <br> 4 22428792 3318958 6757
0.00% 0.06% 0.05% 0 Check heaps <br> 5 481842360 102963163 4679 9.05% 9.70% 7.90% 0 Pool Manager <br> 37 11275060121049358292 1074 36.02% 35.07% 32.40% 0 IP Input <br><br>Thank You<br>
<br>Anthony<br><br><br><br>
<hr>
<br>> Date: Wed, 24 Jan 2007 02:37:10 +0200<br>> From: <a href="mailto:nitzan.tzelniker@gmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">nitzan.tzelniker@gmail.com</a><br>> To: <a href="mailto:antnada@hotmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
antnada@hotmail.com</a><br>> Subject: Re: [cisco-bba] need help on troubleshooting high cpu on 7206 NPE300 LNS<br>> CC: <a href="mailto:cisco-bba@puck.nether.net" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
cisco-bba@puck.nether.net</a><br>> <br>> You can try<br>> <br>> vpdn ip udp ignore checksum<br>> <br>> Nitzan<br>> <br>> On 1/24/07, Anthony Law <<a href="mailto:antnada@hotmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
antnada@hotmail.com</a>> wrote:<br>> > Dear all,<br>> ><br>> > We have a 7206 w/NPE300 running as a LNS terminating pppoe sessions from our<br>> > telco. We are concurrently running around 360 pppoe sessions.
<br>> ><br>> > Recently. I noticed that our 7206 is having extremely high cpu, at times<br>> > going to 100%, please see below<br>> ><br>> > CPU utilization for five seconds: 99%/42%; one minute: 99%; five minutes:
<br>> > 99%<br>> > PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process<br>> > 1 0 75 0 0.00% 0.00% 0.00% 0 Chunk Manager<br>> ><br>> ><br>> > 5 472509060 101324023 4663 7.65% 8.80%
8.84% 0 Pool Manager<br>> ><br>> > 37 10810547881019294234 1060 22.79% 25.16% 25.51% 0 IP Input<br>> ><br>> ><br>> > 101 705044020 800103660 881 18.89% 21.35% 19.34% 0 L2TP data<br>> > daemon
<br>> > 102 53153196 10197928 5212 2.19% 0.46% 0.45% 0 L2TP mgmt<br>> > daemon<br>> ><br>> ><br>> > It seemed that Pool Manager + IP Input + L2TP data daemon together is<br>> > causing this issue. I was searching for documents regarding this on google
<br>> > and came to this mailing list. I am wondering if you guys can help me out by<br>> > identifying the mis-configuration that I have on my end as it is my<br>> > understanding that a 7206 should at least take close 1000 pppoe sessions.
<br>> > Thank You in advance for your input.<br>> ><br>> ><br>> > hostname LNS<br>> > !<br>> > boot system slot1:c7200-is-mz.122-32.bin<br>> > boot system slot1:c7200-is-mz.120-3.T3
<br>> > aaa new-model<br>> > aaa authentication login default local<br>> > aaa authentication login no_rad line<br>> > aaa authentication ppp default group radius local<br>> > aaa authentication ppp vpdn group radius
<br>> > aaa authorization network default group radius<br>> > aaa authorization configuration default group radius<br>> > aaa accounting delay-start<br>> > aaa accounting exec default start-stop group radius
<br>> > aaa accounting network default start-stop group radius<br>> > enable secret 5 XXXXXXXXXXXXXXXXXXXXXXXXXXX<br>> > !<br>> > clock timezone EST -5<br>> > clock summer-time EDT recurring
<br>> > ip subnet-zero<br>> > no ip source-route<br>> > ip cef<br>> > !<br>> > !<br>> > ip name-server XXXXXX<br>> > ip name-server XXXXXX<br>> > ip name-server XXXXXX<br>> > !
<br>> > vpdn enable<br>> > !<br>> > vpdn-group XXXXXXXX<br>> > accept-dialin<br>> > protocol l2tp<br>> > virtual-template 1<br>> > terminate-from hostname XXXXXX<br>> > local name XXXXXXX
<br>> > lcp renegotiation always<br>> > !<br>> > interface FastEthernet0/0<br>> > ip address X.X.X.X <a href="http://255.255.255.192/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
255.255.255.192</a><br>> > no ip mroute-cache<br>> > duplex full<br>> > !<br>> > interface FastEthernet1/0<br>> > no ip address<br>> > no ip mroute-cache<br>> > duplex full<br>> > !
<br>> > interface FastEthernet1/0.401<br>> > description !!XXXXXXXXXXXXXXXXXXXXXXXX!!<br>> > encapsulation dot1Q 401<br>> > ip address 10.70.X.X <a href="http://255.255.255.252/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
255.255.255.252</a><br>> > no ip mroute-cache<br>> > !<br>> > interface FastEthernet2/0<br>> > description !!Internet Feed!!<br>> > ip address Y.Y.Y.Y <a href="http://255.255.255.252/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
255.255.255.252</a><br>> > no ip mroute-cache<br>> > duplex full<br>> > !<br>> > interface Virtual-Template1<br>> > mtu 1492<br>> > ip unnumbered FastEthernet2/0<br>> > peer default ip address pool internet1 internet2
<br>> > ppp authentication pap vpdn<br>> > !<br>> > ip local pool internet1 A.A.A.A B.B.B.B<br>> > ip local pool internet2 C.C.C.C D.D.D.D<br>> > ip classless<br>> > ip route <a href="http://0.0.0.0/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
0.0.0.0</a> <a href="http://0.0.0.0/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">0.0.0.0</a> Y.Y.Y.Y<br>> > no ip http server<br>> > !<br>> > ip radius source-interface FastEthernet0/0
<br>> > radius-server host X.X.X.X auth-port 1645 acct-port 1646<br>> > radius-server host X.X.X.X auth-port 1645 acct-port 1646<br>> > radius-server key 7 ZZZZZZZZZZZZZZZ<br>> ><br>> > Anthony
<br>> ><br>> > ________________________________<br>> > Be one of the first to try Windows Live Mail.<br>> > _______________________________________________<br>> > cisco-bba mailing list<br>> >
<a href="mailto:cisco-bba@puck.nether.net" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">cisco-bba@puck.nether.net</a><br>> > <a href="https://puck.nether.net/mailman/listinfo/cisco-bba" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
https://puck.nether.net/mailman/listinfo/cisco-bba</a><br>> ><br>> ><br>> ><br><br><br>
<hr>
Be one of the first to try <a href="http://ideas.live.com/programpage.aspx?versionId=5d21c51a-b161-4314-9b0e-4911fb2b2e6d" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">Windows Live Mail.</a></div>
<br>_______________________________________________<br>cisco-bba mailing list<br><a href="mailto:cisco-bba@puck.nether.net" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">cisco-bba@puck.nether.net
</a><br><a href="https://puck.nether.net/mailman/listinfo/cisco-bba" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">https://puck.nether.net/mailman/listinfo/cisco-bba</a><br><br><br></blockquote></div>
<br></span></div></blockquote></div><br></blockquote></span></div></blockquote><div><span class="e" id="q_11059a7afd42a332_9"><br><hr>Be one of the first to try <a href="http://ideas.live.com/programpage.aspx?versionId=5d21c51a-b161-4314-9b0e-4911fb2b2e6d" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
Windows Live Mail.</a></span></div></div>
<br>_______________________________________________<br>cisco-bba mailing list<br><a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:cisco-bba@puck.nether.net">cisco-bba@puck.nether.net</a><br><a onclick="return top.js.OpenExtLink(window,event,this)" href="https://puck.nether.net/mailman/listinfo/cisco-bba" target="_blank">
https://puck.nether.net/mailman/listinfo/cisco-bba</a><br><br><br></blockquote></div><br>