[cisco-bba] cpu load due to PPP IPCP (Add Route)

Dennis Peng dpeng at cisco.com
Wed Jun 22 14:49:24 EDT 2005


In this version, you don't have the fix for CSCea87461, so even though
there is an error in the parsing of the IPCP attribute, the session
continues to stay up, but in a non-functioning state. Because the
client periodically tries to bring up IPCP, you see the long list of
AAA. If you fix the syntax or upgrade to 12.3(15), this problem will
stop happening. I reproduced the problem in my lab, but didn't see any
evidence that it would be related to your high CPU problem in IPCP.

Dennis

Tassos Chatzithomaoglou [achatz at forthnet.gr] wrote:
> Hi Dennis,
> 
> Here is another strange thing i noticed today which i don't know if it has 
> something to do with the IPCP * cpu load problem.
> 
> Check out the multiple AAA in the following va interface.
> 
> router#sh int vi704
> Virtual-Access704 is up, line protocol is up
>   Hardware is Virtual Access interface
>   Interface is unnumbered. Using address of Loopback0 (194.219.252.144)
>   MTU 1500 bytes, BW 622080 Kbit, DLY 100000 usec,
>      reliability 255/255, txload 1/255, rxload 1/255
>   Encapsulation PPP, LCP Open, multilink Closed
>   Closed: IPCP
>   PPPoVPDN vaccess, cloned from AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, 
>   AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, 
> AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, 
> AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, 
> AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, 
> AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, 
> AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, 
> AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, 
> AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, 
> AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, 
> AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, 
> AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, AAA, Virtual-Template1
>   Vaccess status 0x44
>   Protocol l2tp, tunnel id 807, session id 24697, loopback not set
>   Keepalive set (20 sec)
>   DTR is pulsed for 5 seconds on reset idle 00:02:42
>   Last input 00:00:09, output never, output hang never
>   Last clearing of "show interface" counters 02:10:17
>   Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
>   Queueing strategy: fifo
>   Output queue: 0/40 (size/max)
>   30 second input rate 0 bits/sec, 0 packets/sec
>   30 second output rate 0 bits/sec, 0 packets/sec
>      1187 packets input, 19310 bytes, 0 no buffer
>      Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
>      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
>      1189 packets output, 19333 bytes, 0 underruns
>      0 output errors, 0 collisions, 0 interface resets
>      0 output buffer failures, 0 output buffers swapped out
>      0 carrier transitions
> 
> 
> Although this particular user has got an ip address (shown with "sh ip int 
> vi704"), he hasn't got his per-user acl through radius, because radius was 
> returning a wrong attribute.
> 
> Jun 22 16:14:37: %AAA-3-PARSEERR: Error(2) parser is unable to parse permit 
> ip x.x.40.24 0.0.7 any per-user command
> 
> 
> 
> Tassos Chatzithomaoglou wrote on 21/6/2005 7:32:
> 
> >This 7200 is using 12.3(12b)...is the correct ios version ;-)
> >
> >Tassos Chatzithomaoglou wrote on 21/6/2005 7:29:
> >
> >>Hi Dennis,
> >>
> >>That's exactly what i also said to my colleague who did "clear ppp 
> >>queues".
> >>But according to his sayings and to our graphs the cpu problem was 
> >>resolved just afterwards.
> >>I don't know if this was just a coincidence.
> >>
> >>Also after a while (or maybe at the same time but we hadn't noticed 
> >>it) we had another problem.
> >>The vpdn tunnel which was terminating to this 7200 couldn't accept any 
> >>more L2TP sessions. It was like incoming ppp was blocked (continuesly 
> >>"LCP: O CONFREQ" with no answer from the peer). We had to tear down 
> >>the tunnel manually in order to make it work again.
> >>
> >>This 7200 is using 12.2(12b).
> >>
> >>Dennis Peng wrote on 21/6/2005 2:36:
> >>
> >>>What version was this? I'm not sure how this would happen. It's
> >>>strange that "clear ppp queues" did anything since that should only
> >>>clear the statistics (counters). If this occurs again, can you grab
> >>>"debug ppp events"?
> >>>
> >>>Dennis
> >>>
> >>>Tassos Chatzithomaoglou [achatz at forthnet.gr] wrote:
> >>>
> >>>>Today we had the cpu of one of our 7200's reaching 100%. We found 
> >>>>that this was due to the "IPCP Skipped - Add Route:" which was 
> >>>>constantly increasing (we supply routes per user through radius). 
> >>>>After we did a "clear ppp queue", it returned to normal.
> >>>>
> >>>>Any idea what might have happened?
> >>>>
> >>>>router#sh proc cpu sorted | e 0.00
> >>>>CPU utilization for five seconds: 99%/70%; one minute: 99%; five 
> >>>>minutes: 99%
> >>>> PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
> >>>>  77    14284408   3108213       4595 20.47% 21.03% 21.11%   0 PPP 
> >>>>IPCP         <---
> >>>> 163    84786456   2270413      37344  1.86%  0.79%  0.64%   0 
> >>>>VTEMPLATE Backgr
> >>>> 148    75342260   4779805      15762  1.19%  0.76%  0.76%   0 IGMP 
> >>>>Input
> >>>> 150    92048440  14330863       6423  1.12%  1.02%  1.00%   0 PIM 
> >>>>Process
> >>>>  31    42891736   4708264       9109  1.12%  0.42%  0.38%   0 
> >>>>Per-Second Jobs
> >>>> 156    32672884 184875306        176  1.04%  1.03%  1.04%   0 PPP 
> >>>>Events
> >>>>  79    84550988   7024517      12036  0.97%  0.69%  0.67%   0 CEF 
> >>>>process
> >>>> 129    48902960    889685      54966  0.59%  0.53%  0.53%   0 
> >>>>Compute load avg
> >>>>  50    56443568 247614090        227  0.29%  0.38%  0.37%   0 IP Input
> >>>>   5    32496292   1692766      19197  0.22%  0.17%  0.22%   0 Check 
> >>>>heaps
> >>>> 167    18315212  18625936        983  0.22%  0.21%  0.24%   0 OSPF 
> >>>>Router
> >>>>  63     4082388   5244432        778  0.14%  0.12%  0.11%   0 IP 
> >>>>Background
> >>>> 160     1580920 997329231          1  0.14%  0.04%  0.01%   0 SAA 
> >>>>Event Proces
> >>>>  76     9643460   1590124       6064  0.07%  1.58%  1.62%   0 PPP 
> >>>>IP Route
> >>>>
> >>>>
> >>>>router#sh ppp queues
> >>>>
> >>>>5 Event Queues
> >>>>                 size   max      kicks     starts    false   
> >>>>suspends ticks(ms)
> >>>> 1 PPP Events       0     9      27514      27512        7          
> >>>>0        20
> >>>> 5 PPP Bind         0     2        445        445        0          
> >>>>0        20
> >>>> 3 PPP IPCP         0     0          0          0        0          
> >>>>0        20
> >>>> 2 PPP IP Route     0     3        226        226        0          
> >>>>0       100
> >>>> 4 PPP Hooks        0     1          1          1        0          
> >>>>0        20
> >>>>
> >>>> 30 Events
> >>>> #  Q Name                 Events   Queued  MaxQueued Suspends  
> >>>>usec/evt max/evt
> >>>> 1  1 Setup                     1        0        1        0       
> >>>>654       654
> >>>> 2  1 Free PPP                661        0        3        0       
> >>>>111      3172
> >>>> 3* 1 Timer                 28072 4294967290 4294967295        
> >>>>1      1047      9575
> >>>> 4  1 Cstate                   18        0        3        0      
> >>>>1041      4249
> >>>> 5  1 Restart CP                0        0        0        0         
> >>>>0         0
> >>>> 6    UNREGISTERED
> >>>> 7  1 Hard Disc               409        0        2        0       
> >>>>893      2466
> >>>> 8  1 Soft Disc                77        0        2        2      
> >>>>3174      9366
> >>>> 9  1 Packet                    2 4294967294 4294967295        
> >>>>0       380       475
> >>>>10  1 Auth Packet               0        0        0        0         
> >>>>0         0
> >>>>11* 3 IPCP Packet               0        0        0        0         
> >>>>0         0
> >>>>12* 2 Add Route                 0        0        0        0         
> >>>>0         0
> >>>>13* 2 Remove Route            247        0        3        0       
> >>>>293      1358
> >>>>14* 2 Remove Top                0        0        0        0         
> >>>>0         0
> >>>>15* 2 Remove Flag               0        0        0        0         
> >>>>0         0
> >>>>16* 2 Neg Addr Add              0        0        0        0         
> >>>>0         0
> >>>>17* 2 Neg Addr Remov            0        0        0        0         
> >>>>0         0
> >>>>18* 2 DHCP Opt Add              0        0        0        0         
> >>>>0         0
> >>>>19* 2 DHCP Opt Del              0        0        0        0         
> >>>>0         0
> >>>>20* 2 IPCP UP                   0        0        0        0         
> >>>>0         0
> >>>>21  1 Set LCP Open            448        0        2        0       
> >>>>497      1794
> >>>>22  1 Virtualize                0        0        0        0         
> >>>>0         0
> >>>>23  1 Redirect                  0        0        0        0         
> >>>>0         0
> >>>>24  1 Forwarded                 1        0        1        0       
> >>>>317       317
> >>>>25  4 Hook                      1        0        1        0        
> >>>>17        17
> >>>>26  1 AAA Response              3        0        2        0       
> >>>>994      1568
> >>>>27  1 Static Bind               1        0        1        0       
> >>>>379       379
> >>>>28  5 Dynamic Bind            448        0        2        0       
> >>>>585      1893
> >>>>29  1 Bound                     0        0        0        0         
> >>>>0         0
> >>>>30  1 Virtual Profil            0        0        0        0         
> >>>>0         0
> >>>>
> >>>>Pre-processed Requests  LCP:         0  IPCP:         0
> >>>>PPP Request Failures: 0
> >>>>PPP Hold Queue Drops: 14248
> >>>>IPCP Skipped - Add Route:         545  AddrAdd:         0  
> >>>>AddrRem:         0
> >>>>PPP Handles:  alloc[5719489] free[5717015] outstanding[2474]
> >>>>
> >>>>_______________________________________________
> >>>>cisco-bba mailing list
> >>>>cisco-bba at puck.nether.net
> >>>>https://puck.nether.net/mailman/listinfo/cisco-bba
> >>>
> >>>
> >>>
> >>>
> >>
> >


More information about the cisco-bba mailing list