[nsp] High "IP Input"/process switching on 5800
Steven W. Raymond
steven_raymond at eli.net
Thu Jan 9 12:18:25 EST 2003
Hello,
Have 5800s suffering from high CPU utilization (up to say 60% at peak
times). Seems to be causing packet loss across the directly-connected
ethernet (no errors on interfaces) except for accumulating "flushes":
#sh int f0/3/0
FastEthernet0/3/0 is up, line protocol is up
Hardware is DEC21140A, address is 0001.97ef.6c54 (bia 0001.97ef.6c54)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 3/255, rxload 12/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 01:56:59
Input queue: 10/75/134/85106 (size/max/drops/flushes); Total output
drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 4801000 bits/sec, 1160 packets/sec
5 minute output rate 1313000 bits/sec, 1070 packets/sec
7552501 packets input, 4099183949 bytes
Received 4450 broadcasts, 0 runts, 0 giants, 126 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
6958987 packets output, 1093618763 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Output of "show proc cpu":
CPU utilization for five seconds: 30%/19%; one minute: 38%; five
minutes: 36%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 6804 42647 159 0.08% 0.00% 0.00% 0 Load
Meter
2 4 18 222 0.00% 0.00% 0.00% 0 MAI Action
Proce
3 1592180 94311 16882 0.00% 0.73% 0.66% 0 Check
heaps
4 0 1 0 0.00% 0.00% 0.00% 0 Chunk
Manager
5 372 755 492 0.00% 0.00% 0.00% 0 Pool
Manager
6 0 2 0 0.00% 0.00% 0.00% 0
Timers
7 5128 38611 132 0.00% 0.00% 0.00% 0 Serial
Backgroun
8 1536 213387 7 0.00% 0.00% 0.00% 0
EnvMon
9 0 1 0 0.00% 0.00% 0.00% 0 OIR
Handler
10 4504484 919350 4899 0.48% 0.26% 0.24% 0 ARP
Input
11 348 10767 32 0.00% 0.00% 0.00% 0 HC Counter
Timer
12 10828 135619 79 0.00% 0.00% 0.00% 0 DDR
Timers
13 0 2 0 0.00% 0.00% 0.00% 0 Dialer
event
14 236 5 47200 0.00% 0.00% 0.00% 0 Entity MIB
API
15 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone
Manager
16 1504 213388 7 0.00% 0.00% 0.00% 0 IPC Periodic
Tim
17 132408 2539640 52 0.08% 0.02% 0.02% 0 IPC Seat
Manager
18 0 2 0 0.00% 0.00% 0.00% 0 DS OIR
Handler o
19 0 1 0 0.00% 0.00% 0.00% 0 RM
PROCESS
20 0 1 0 0.00% 0.00% 0.00% 0 RM
PROCESS
21 0 1 0 0.00% 0.00% 0.00% 0 RM
PROCESS
22 0 1 0 0.00% 0.00% 0.00% 0 RM
PROCESS
23 0 2 0 0.00% 0.00% 0.00% 0 CAS
Process
24 0 1 0 0.00% 0.00% 0.00% 0 SERIAL
A'detect
25 0 1 0 0.00% 0.00% 0.00% 0 Critical
Bkgnd
26 500944 828542 604 0.08% 0.12% 0.15% 0 Net
Background
27 8 211 37 0.00% 0.00% 0.00% 0
Logger
28 16180 213220 75 0.00% 0.00% 0.00% 0 TTY
Background
29 71160 213387 333 0.00% 0.02% 0.01% 0 Per-Second
Jobs
30 0 1 0 0.00% 0.00% 0.00% 0 CSP
Timer
31 3020 21326 141 0.00% 0.00% 0.00% 0 NIP Boot
Daemon
32 1664 132124 12 0.00% 0.00% 0.00% 0 DSIP
Daemon
33 0 7 0 0.00% 0.00% 0.00% 0 DSIP INTRAPI
Dae
34 0 1 0 0.00% 0.00% 0.00% 0 CES Line
Conditi
35 0 2 0 0.00% 0.00% 0.00% 0 Flash MIB
Update
36 0 2 0 0.00% 0.00% 0.00% 0 CSM
Periodic
37 0 1 0 0.00% 0.00% 0.00% 0 COT Timer
proces
38 0 1 0 0.00% 0.00% 0.00% 0 COT Queue
proces
39 31764 6067 5235 0.00% 0.00% 0.00% 0 Portware
Downloa
40 6716 107706 62 0.00% 0.00% 0.00% 0 DSIP
Redundancy
41 23880 456278 52 0.00% 0.00% 0.00% 0 Call
Management
42 16 3590 4 0.00% 0.00% 0.00% 0 VRM reset
proces
43 0 1 0 0.00% 0.00% 0.00% 0
VRM
44 6032 43279 139 0.00% 0.00% 0.00% 0 MM/CSM Event
Bkg
45 0 6 0 0.00% 0.00% 0.00% 0 MICA Modem
Recov
46 51228 129225 396 0.00% 0.01% 0.00% 0 MM/Mica
Event
47 41772 406939 102 0.08% 0.01% 0.00% 0 MLPVT Hello
Serv
48 163260 28549 5718 0.00% 0.00% 0.00% 0 IP
SNMP
49 0 1 0 0.00% 0.00% 0.00% 0 SNMP
ConfCopyPro
50 0 1 0 0.00% 0.00% 0.00% 0 SNMP
Traps
51 5136 1062301 4 0.00% 0.00% 0.00% 0 MLPVT MQ
Process
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
52 9520 78803 120 0.00% 0.00% 0.00% 0 L2F
management d
53 13068 139945 93 0.00% 0.01% 0.00% 0 L2F data
daemon
54 0 1 0 0.00% 0.00% 0.00% 0 L2TP data
daemon
55 0 1 0 0.00% 0.00% 0.00% 0 L2TP mgmt
daemon
56 32567344 214077966 152 5.51% 6.63% 6.39% 0 IP
Input
57 4 1224 3 0.00% 0.00% 0.00% 0
TACACS+
58 9460 115807 81 0.00% 0.00% 0.00% 0 PPP IP Add
Route
59 189296 471900 401 0.16% 0.04% 0.05% 0 IP
Background
60 0 1 0 0.00% 0.00% 0.00% 0 SNMP
Timers
61 12852 3663 3508 0.00% 0.00% 0.00% 0 Adj
Manager
62 0 1 0 0.00% 0.00% 0.00% 0 HTTP
Timer
63 40 1678 23 0.00% 0.00% 0.00% 0 TCP
Timer
64 32 111 288 0.00% 0.00% 0.00% 0 TCP
Protocols
65 0 1 0 0.00% 0.00% 0.00% 0 Probe
Input
66 0 1 0 0.00% 0.00% 0.00% 0 RARP
Input
67 0 1 0 0.00% 0.00% 0.00% 0 Socket
Timers
68 7184 30483 235 0.00% 0.00% 0.00% 0 DHCPD
Receive
69 47200 3589 13151 0.00% 0.00% 0.00% 0 IP Cache
Ager
70 0 1 0 0.00% 0.00% 0.00% 0 PAD
InCall
71 0 2 0 0.00% 0.00% 0.00% 0 X.25
Background
72 0 2 0 0.00% 0.00% 0.00% 0 RLM groups
Proce
73 16 3590 4 0.00% 0.00% 0.00% 0 TCP
Intercept Ti
74 1420 213216 6 0.00% 0.00% 0.00% 0 DS
Manager
75 0 2 0 0.00% 0.00% 0.00% 0 tcl ivr
app
76 0 2 0 0.00% 0.00% 0.00% 0 tcl ivr
app
77 0 2 0 0.00% 0.00% 0.00% 0 tcl ivr
app
78 0 2 0 0.00% 0.00% 0.00% 0 tcl ivr
app
79 0 2 0 0.00% 0.00% 0.00% 0 tcl ivr
app
80 0 2 0 0.00% 0.00% 0.00% 0 tcl ivr
app
81 0 2 0 0.00% 0.00% 0.00% 0 tcl ivr
app
82 0 2 0 0.00% 0.00% 0.00% 0 tcl ivr
app
83 0 1 0 0.00% 0.00% 0.00% 0 Syslog
Traps
84 14660 869858 16 0.00% 0.00% 0.00% 0 Net
Input
85 1945200 42647 45611 0.97% 0.76% 0.79% 0 Compute load
avg
86 94376 3589 26295 0.00% 0.01% 0.00% 0 Per-minute
Jobs
87 0 3 0 0.00% 0.00% 0.00% 0 Modem
Recovery W
88 188 20687 9 0.00% 0.00% 0.00% 0 Portware
Watcher
89 24 2391 10 0.00% 0.00% 0.00% 0 Portware
Sanity
90 724080 2584629 280 0.16% 0.06% 0.17% 0
ISDN
91 0 2 0 0.00% 0.00% 0.00% 0 ISDN
Timer
92 40 3250 12 0.00% 0.00% 0.00% 0 ISDNMIB
Backgrou
93 892 74646 11 0.00% 0.00% 0.00% 0 CallMIB
Backgrou
94 0 2 0 0.00% 0.00% 0.00% 0
CC-API_VCM
95 0 2 0 0.00% 0.00% 0.00% 0 Session
Applicat
96 0 1 0 0.00% 0.00% 0.00% 0
VTSP
97 0 2 0 0.00% 0.00% 0.00% 0
VOIP_RTCP
98 4 2 2000 0.00% 0.00% 0.00% 0 Voice
Player
99 0 1 0 0.00% 0.00% 0.00% 0
pcapp
100 0 1 0 0.00% 0.00% 0.00% 0
placecall
101 0 1 0 0.00% 0.00% 0.00% 0 ivr
app
102 0 1 0 0.00% 0.00% 0.00% 0 ivr
app
103 0 1 0 0.00% 0.00% 0.00% 0 ivr
app
104 0 1 0 0.00% 0.00% 0.00% 0 ivr
app
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
105 0 1 0 0.00% 0.00% 0.00% 0 ivr
app
106 0 1 0 0.00% 0.00% 0.00% 0 ivr
app
107 0 1 0 0.00% 0.00% 0.00% 0 ivr
app
108 0 2 0 0.00% 0.00% 0.00% 0
CCH323_CT
109 0 2 0 0.00% 0.00% 0.00% 0
CCH323_RTCP
110 0 1 0 0.00% 0.00% 0.00% 0
CCH323_DNS
111 0 2 0 0.00% 0.00% 0.00% 0 CCP
manager
112 646552 1567406 412 0.08% 0.03% 0.22% 0 PPP
manager
113 61424 507768 120 0.00% 0.00% 0.00% 0 Multilink
PPP
114 0 3 0 0.00% 0.00% 0.00% 0 Multilink
PPP ou
115 1173512 39211 29928 0.08% 0.03% 0.51% 0 Multilink
event
116 114280 1004861 113 0.00% 0.01% 0.03% 0
RADIUS
117 1044 216856 4 0.00% 0.00% 0.00% 0
NTP
118 18976 509792 37 0.00% 0.00% 0.00% 0 Periodic
Account
119 35272 338727 104 0.00% 0.00% 0.00% 0 AAA
Accounting
120 3828 35840 106 0.00% 0.00% 0.00% 0 TTY
Autoconfig
121 8 1794 4 0.00% 0.00% 0.00% 0 DHCPD
Timer
122 404 60947 6 0.00% 0.00% 0.00% 0 DHCPD
Database
123 1338576 17684 75694 2.92% 0.74% 0.59% 0 VTEMPLATE
Backgr
124 196 1739 112 0.00% 0.00% 0.00% 0 VPDN failure
man
125 24892 732059 34 0.00% 0.00% 0.00% 2 Virtual
Exec
126 0 3 0 0.00% 0.00% 0.00% 0 TCP
Listener
127 3116 102899 30 0.00% 0.00% 0.00% 0 BGP
Router
128 100 2773 36 0.00% 0.00% 0.00% 0 BGP
I/O
129 3073460 342126 8983 0.00% 0.95% 1.64% 0 PPP
auth
130 810596 311711 2600 0.00% 0.00% 0.27% 0 PPP
auth
131 419564 308662 1359 0.00% 0.00% 0.13% 0 PPP
auth
132 227204 318816 712 0.00% 0.00% 0.06% 0 PPP
auth
133 133384 325783 409 0.00% 0.00% 0.04% 0 PPP
auth
134 74736 333903 223 0.00% 0.00% 0.00% 0 PPP
auth
135 48408 340013 142 0.00% 0.00% 0.00% 0 PPP
auth
136 27796 344860 80 0.00% 0.00% 0.00% 0 PPP
auth
137 14180 347094 40 0.00% 0.00% 0.00% 0 PPP
auth
138 8784 347717 25 0.00% 0.00% 0.00% 0 PPP
auth
139 5820 349903 16 0.00% 0.00% 0.00% 0 PPP
auth
140 4076 350182 11 0.00% 0.00% 0.00% 0 PPP
auth
141 3748 351274 10 0.00% 0.00% 0.00% 0 PPP
auth
142 2436 351114 6 0.00% 0.00% 0.00% 0 PPP
auth
143 2296 351268 6 0.00% 0.00% 0.00% 0 PPP
auth
144 2036 351265 5 0.00% 0.00% 0.00% 0 PPP
auth
145 2088 351261 5 0.00% 0.00% 0.00% 0 PPP
auth
146 2292 351266 6 0.00% 0.00% 0.00% 0 PPP
auth
147 2248 351266 6 0.00% 0.00% 0.00% 0 PPP
auth
148 2000 351266 5 0.00% 0.00% 0.00% 0 PPP
auth
149 2216 351266 6 0.00% 0.00% 0.00% 0 PPP
auth
150 2300 351265 6 0.00% 0.00% 0.00% 0 PPP
auth
151 2436 351263 6 0.00% 0.00% 0.00% 0 PPP
auth
152 2284 351264 6 0.00% 0.00% 0.00% 0 PPP
auth
153 2176 351261 6 0.00% 0.00% 0.00% 0 PPP
auth
154 2216 351261 6 0.00% 0.00% 0.00% 0 PPP
auth
155 2316 351259 6 0.00% 0.00% 0.00% 0 PPP
auth
156 2144 351259 6 0.00% 0.00% 0.00% 0 PPP
auth
157 1968 351259 5 0.00% 0.00% 0.00% 0 PPP
auth
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
158 2056 351258 5 0.00% 0.00% 0.00% 0 PPP
auth
159 25904 2995 8649 0.00% 0.03% 0.05% 0 BGP
Scanner
160 352 50653 6 1.86% 0.18% 0.04% 3 Virtual
Exec
161 6660 1866732 3 0.00% 2.01% 0.99% 4 Virtual
Exec
162 2308 7289 316 0.00% 0.00% 0.00% 0 CEF
process
163 412 167 2467 0.00% 0.00% 0.00% 0 CEF
Scanner
165 4 39 102 0.00% 0.00% 0.00% 1/9 Modem
Autoconfig
Do have global CEF configured, but the group async interface seems to be
unable to participate:
#sh cef interface
FastEthernet0/0/0 is down (if_number 1302)
Internet Protocol processing disabled
Hardware idb is FastEthernet0/0/0
Fast switching type 1, interface type 18
IP CEF switching enabled
IP Feature Fast switching turbo vector
IP Feature CEF switching turbo vector
Input fast flags 0x0, Output fast flags 0x0
ifindex 1298(1298)
Slot 0 Slot unit 0 VC -1
Transmit limit accumulator 0x0 (0x0)
IP MTU 1500
FastEthernet0/3/0 is up (if_number 1305)
Internet address is ...
ICMP redirects are always sent
Per packet load-sharing is disabled
IP unicast RPF check is disabled
Inbound access list is not set
Outbound access list is not set
IP policy routing is disabled
Hardware idb is FastEthernet0/3/0
Fast switching type 1, interface type 18
IP CEF switching enabled
IP Feature Fast switching turbo vector
IP Feature CEF switching turbo vector
Input fast flags 0x0, Output fast flags 0x0
ifindex 1299(1299)
Slot 3 Slot unit 0 VC -1
Transmit limit accumulator 0x0 (0x0)
IP MTU 1500
Group-Async0 is up (if_number 5)
Internet address is 0.0.0.0/0
Unnumbered Interface, using Group-Async0 .....
ICMP redirects are always sent
Per packet load-sharing is disabled
IP unicast RPF check is disabled
Inbound access list is not set
Outbound access list is not set
IP policy routing is disabled
Interface is marked as point to point interface
Packets switched to this interface are dropped to the next slow path
Virtual-Template1 is down (if_number 2707)
Internet address is 0.0.0.0/0
Unnumbered Interface, using Virtual-Template1 (address ...)
ICMP redirects are always sent
Per packet load-sharing is disabled
IP unicast RPF check is disabled
Inbound access list is not set
Outbound access list is not set
IP policy routing is disabled
Interface is marked as point to point interface
Hardware idb is Virtual-Template1
Fast switching type 7, interface type 20
IP CEF switching enabled
IP Feature Fast switching turbo vector
IP Feature CEF switching turbo vector
Input fast flags 0x0, Output fast flags 0x0
ifindex 2644(2644)
Slot -1 Slot unit 1 VC -1
Transmit limit accumulator 0x0 (0x0)
IP MTU 1500
Virtual-Access1 is down (if_number 2709)
Internet address is 0.0.0.0/0
ICMP redirects are always sent
Per packet load-sharing is disabled
IP unicast RPF check is disabled
Inbound access list is not set
Outbound access list is not set
IP policy routing is disabled
Interface is marked as point to point interface
Hardware idb is Virtual-Access1
Fast switching type 7, interface type 21
IP CEF switching enabled
IP CEF Feature Fast switching turbo vector
IP Null turbo vector
Input fast flags 0x0, Output fast flags 0x0
ifindex 2646(2646)
Slot -1 Slot unit 1 VC -1
Transmit limit accumulator 0x0 (0x0)
IP MTU 1500
<snip>
Should Cisco 5800-series have the capability to CEF switch across the
entire box? Am finding that although global CEF can be applied to the
fast e interface, but not group async (it takes the command, but doesn't
seem to stick).
Am I barking up the wrong tree here? Basically it appears that even
under half load the box is process switching most packets, perhaps
causing flushes which drops packets. Any suggestions for improvement?
Is CEF doing it's job here (as designed for the 5800?) Is it possible
to add "route-cache cef" to interface Virtual-Template1 ?
Thanks
More output below...
#sh cef not-cef-switched
CEF Packets passed on to next switching layer
Slot No_adj No_encap Unsupp'ted Redirect Receive Options Access
Frag
RP 0 0 5485989 0 44835 0
0 0
#sh int stat
Interface FastEthernet0/0/0 is disabled
FastEthernet0/3/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 6524787 3251382751 5757660 995781740
Route cache 1448820 1069390636 1585058 160664971
Total 7973607 25806091 7342718 1156446711
Group-Async0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 0 0 0 0
Route cache 0 0 0 0
Total 0 0 0 0
Virtual-Template1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 0 0 0 0
Route cache 0 0 0 0
Total 0 0 0 0
Virtual-Access1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 0 0 0 0
Route cache 0 0 0 0
Total 0 0 0 0
Virtual-Access2
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 6591 784766 10 3077
Route cache 0 0 6084 5731206
Total 6591 784766 6094 5734283
Virtual-Access3
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 12937 1038735 25 2633
Route cache 0 0 11553 12297932
Total 12937 1038735 11578 12300565
Virtual-Access4
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 7 140 9 209
Route cache 280 41345 287 82373
Total 287 41485 296 82582
Virtual-Access5
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 0 0 0 0
Route cache 0 0 0 0
Total 0 0 0 0
Virtual-Access6
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 7738 700549 24 1467
Route cache 0 0 7023 4300216
Total 7740 700647 7048 4301727
Virtual-Access7
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 1670 89792 7 214
Route cache 0 0 2123 2558885
Total 1670 89792 2130 2559099
Virtual-Access8
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 280 34504 8 334
Route cache 1263 87454 1095 286699
Total 1543 121958 1103 287033
Virtual-Access9
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 1434 159376 11 783
Route cache 0 0 1395 1127836
Total 1434 159376 1406 1128619
Virtual-Access10
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 2595 453640 35 3126
Route cache 0 0 2231 1810482
Total 2595 453640 2266 1813608
More information about the cisco-nsp
mailing list