[c-nsp] 4006 RSM Directional Performance
Adam Peterson
Adam.Peterson at splwg.com
Wed Apr 19 18:09:31 EDT 2006
I'm having the same issue this guy was having:
http://groups.google.com/group/comp.dcom.sys.cisco/browse_thread/thread/
5153336ed7d91e99/b1e80b4bf52e0f02?lnk=st&q=cisco+cef+debug+slow&rnum=1&h
l=en#b1e80b4bf52e0f02
To summarize, traffic in 1 direction is crushing the CPU (easily 98%
under load, 50% average with less than 200 machines and less than 8k pps
peak) but in the other direction there is no real effect on the CPU and
throughput is acceptable.
For a while the high CPU was happening on both (identical) routers but
now it's only happening on 1 of them. Both sides have switch CPU around
45% constant.
I was positive my problem was specifically related to my multi-homed
config - routers on both sides have multiple addresses on multiple
subnets on a single interface. After reading the post above, this can't
be. The most interesting things I've come across are the fact that cef
can't be disabled (ip cef isn't a valid command) on this RSM and yet the
cef cache is getting 0 hits. All of my routing is being done by the
processor and there's nothing I've been able to do about it.
Even in the direction that "works," it takes 15 seconds to copy an 85MB
file. On the same router/switch, even across those multi-homed subnets,
that file copies in about 2 seconds. Huge pings of 18k will get dropped
periodically and generally the performance is about 30ms across the gbps
link.
Here's where it gets really weird. I added a NEW secondary address to
router2 (below, 10.254.253.1) - identical setup to existing secondary
addresses on the same router including static route at the other end -
and that new subnet running on the same physical LAN performs perfectly
in both directions. Huge pings are always around 2ms and never get
dropped. There's no difference between this new subnet and the existing
secondary subnets other than the fact that I'm the only one using it so
if this is some sort of table/cache/arp(?) issue, that might be
important.
Over the last few days I've been able to solve some unrelated issues
that appeared to be related such as adding "no ip redirects" to the g3
interfaces on both routers. That fixed the issue within the same
switch/router that seemed to be the same problem but must be unrelated
because the "no ip redirects" solves that problem like the flick of a
switch but this only applies to the subnets being routed on the g3
interfaces. It behaved exactly like the current problem though.
I think I'm doing a horrible job of explaining this because I've been in
it for the last 3 days solid. Absolutely any help is greatly
appreciated.
router1#sh ver
Cisco Internetwork Operating System Software
IOS (tm) L3 Switch/Router Software (CAT4232-IN-M), Version
12.0(10)W5(18f) RELE
ASE SOFTWARE
Copyright (c) 1986-2000 by cisco Systems, Inc.
Compiled Mon 04-Dec-00 22:07 by integ
Image text-base: 0x60010928, data-base: 0x605F6000
ROM: System Bootstrap, Version 12.0(7)W5(15b) RELEASE SOFTWARE
router1#sh run
Current configuration:
!
version 12.0
no service pad
service timestamps debug uptime
service timestamps log uptime
service password-encryption
!
hostname router1
!
enable secret router1
!
ip subnet-zero
ip name-server xxx.xxx.xxx.xxx
!
!
!
interface FastEthernet1
no ip address
no ip directed-broadcast
shutdown
!
interface GigabitEthernet1
bandwidth 10000000
ip address 10.90.90.1 255.255.255.252
no ip redirects
no ip directed-broadcast
no ip proxy-arp
no negotiation auto
random-detect
!
interface GigabitEthernet2
no ip address
no ip directed-broadcast
shutdown
!
interface GigabitEthernet3
ip address xxx.xxx.75.175 255.255.255.0 secondary
ip address xxx.xxx.254.1 255.255.255.0 secondary
ip address xxx.xxx.75.1 255.255.255.0
no ip redirects
no ip directed-broadcast
no ip proxy-arp
no negotiation auto
random-detect
!
interface GigabitEthernet4
no ip address
no ip directed-broadcast
no negotiation auto
!
ip classless
ip route 0.0.0.0 0.0.0.0 xxx.xxx.75.10
ip route xxx.xxx.117.0 255.255.255.0 10.90.90.2
<edited>
!
!
line con 0
transport input none
line aux 0
line vty 0 4
password 7 router1
login
!
end
router1#sh proc cpu
CPU utilization for five seconds: 52%/46%; one minute: 34%; five
minutes: 39%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 76 119429 0 0.00% 0.00% 0.00% 0 Load Meter
2 140 129 1085 0.00% 0.00% 0.00% 2 Virtual Exec
3 602984 73714 8180 0.81% 0.11% 0.06% 0 Check heaps
4 40 68 588 0.00% 0.00% 0.00% 0 Pool Manager
5 0 2 0 0.00% 0.00% 0.00% 0 Timers
6 294040 1209588 243 0.00% 0.02% 0.00% 0 ARP Input
7 0 2 0 0.00% 0.00% 0.00% 0 Serial
Backgroun
8 0 2 0 0.00% 0.00% 0.00% 0 DDR Timers
9 0 1 0 0.00% 0.00% 0.00% 0 Entity MIB
API
10 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler
11 0 2 0 0.00% 0.00% 0.00% 0 EPIF reset
proce
12 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone
Manager
13 1332 596879 2 0.00% 0.00% 0.00% 0 IPC Periodic
Tim
14 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat
Manager
15 3168 122867 25 0.00% 0.00% 0.00% 0 HC Counter
Timer
16 0 1 0 0.00% 0.00% 0.00% 0 SERIAL
A'detect
17 676 21 32190 0.00% 0.00% 0.00% 0 EPAM Card
Manage
18 4 10 400 0.00% 0.00% 0.00% 0 EPIF Enable
If p
19 0 1 0 0.00% 0.00% 0.00% 0 Critical
Bkgnd
20 10912 74417 146 0.00% 0.00% 0.00% 0 Net
Background
21 140 1111 126 0.00% 0.00% 0.00% 0 Logger
22 1840 596916 3 0.00% 0.00% 0.00% 0 TTY
Background
23 892 596878 1 0.00% 0.00% 0.00% 0 Per-Second
Jobs
24 440 597139 0 0.00% 0.00% 0.00% 0 LINE Monitor
Dea
25 0 1 0 0.00% 0.00% 0.00% 0 marker
26 382264 19906 19203 0.00% 0.05% 0.05% 0 THERMOSTAT
monit
27 1924 596967 3 0.00% 0.00% 0.00% 0 LSIPC
Periodic T
28 56872 671972 84 0.00% 0.00% 0.00% 0 LSIPC RECV
PROCE
29 224 634 353 0.00% 0.00% 0.00% 0 LSIPC BCAST
PROC
30 2804 57 49192 0.00% 0.00% 0.00% 0 LSS ACL/QOS
Proc
31 680 198960 3 0.00% 0.00% 0.00% 0 Envirnment
Monit
32 0 1 0 0.00% 0.00% 0.00% 0 PPP IP Add
Route
33 17520 82437 212 0.00% 0.00% 0.00% 0 CDP Protocol
34 57120808 126850201 450 5.24% 3.07% 3.48% 0 IP Input
35 12408 73 169972 0.00% 0.00% 0.00% 0 SCP Process
36 0 2 0 0.00% 0.00% 0.00% 0 DOMINO PORT
QOS
37 0 7 0 0.00% 0.00% 0.00% 0 LSS VLAN
Reload
38 220416 3910101 56 0.00% 0.01% 0.00% 0 LSS L2STAT
RECV
router1#sh ip cef g3 detail
IP CEF with switching (Table Version 738)
211 routes, 0 reresolve, 0 unresolved (0 old, 0 new)
211 leaves, 54 nodes, 82520 bytes, 738 inserts, 527 invalidations
0 load sharing elements, 0 bytes, 0 references
1 CEF resets, 18 revisions of existing leaves
refcounts: 14447 leaf, 14080 node
Adjacency Table has 145 adjacencies
0.0.0.0/0, version 6, cached adjacency xxx.xxx.75.10
0 packets, 0 bytes
via xxx.xxx.75.10, 0 dependencies, recursive
next hop xxx.xxx.75.10, GigabitEthernet3 via xxx.xxx.75.10/32
valid cached adjacency
10.1.7.0/24, version 7, cached adjacency xxx.xxx.75.8
0 packets, 0 bytes
via xxx.xxx.75.8, 0 dependencies, recursive
next hop xxx.xxx.75.8, GigabitEthernet3 via xxx.xxx.75.8/32
valid cached adjacency
10.1.10.43/32, version 8, cached adjacency xxx.xxx.75.8
0 packets, 0 bytes
via xxx.xxx.75.8, 0 dependencies, recursive
next hop xxx.xxx.75.8, GigabitEthernet3 via xxx.xxx.75.8/32
valid cached adjacency
10.1.10.109/32, version 9, cached adjacency xxx.xxx.75.8
0 packets, 0 bytes
via xxx.xxx.75.8, 0 dependencies, recursive
next hop xxx.xxx.75.8, GigabitEthernet3 via xxx.xxx.75.8/32
valid cached adjacency
10.1.13.0/24, version 10, cached adjacency xxx.xxx.75.8
0 packets, 0 bytes
via xxx.xxx.75.8, 0 dependencies, recursive
next hop xxx.xxx.75.8, GigabitEthernet3 via xxx.xxx.75.8/32
valid cached adjacency
router1#sh ip cef g1
Prefix Next Hop Interface
10.10.11.0/24 attached GigabitEthernet1
10.90.90.0/30 attached GigabitEthernet1
10.90.90.2/32 10.90.90.2 GigabitEthernet1
10.254.253.0/24 10.90.90.2 GigabitEthernet1
192.168.10.0/24 attached GigabitEthernet1
xxx.xxx.117.0/24 attached GigabitEthernet1
router2#sh ver
Cisco Internetwork Operating System Software
IOS (tm) L3 Switch/Router Software (CAT4232-IN-M), Version
12.0(10)W5(18f) RELE
ASE SOFTWARE
Copyright (c) 1986-2000 by cisco Systems, Inc.
Compiled Mon 04-Dec-00 22:07 by integ
Image text-base: 0x60010928, data-base: 0x605F6000
ROM: System Bootstrap, Version 12.0(7)W5(15b) RELEASE SOFTWARE
router2#sh run
Current configuration:
!
version 12.0
no service pad
service timestamps debug uptime
service timestamps log uptime
service password-encryption
!
hostname router2
!
enable secret router2
!
ip subnet-zero
ip name-server xxx.xxx.117.23
!
!
!
interface FastEthernet1
no ip address
no ip directed-broadcast
shutdown
!
interface GigabitEthernet1
bandwidth 10000000
ip address 10.90.90.2 255.255.255.252
no ip redirects
no ip directed-broadcast
no ip proxy-arp
no negotiation auto
random-detect
!
interface GigabitEthernet2
no ip address
no ip redirects
no ip directed-broadcast
shutdown
!
interface GigabitEthernet3
ip address 192.168.10.1 255.255.255.0 secondary
ip address 10.254.253.1 255.255.255.0 secondary
ip address xxx.xxx.117.2 255.255.255.0
no ip redirects
no ip directed-broadcast
no ip proxy-arp
no negotiation auto
random-detect
!
interface GigabitEthernet4
no ip address
no ip redirects
no ip directed-broadcast
no ip proxy-arp
no negotiation auto
random-detect
!
ip classless
ip route 0.0.0.0 0.0.0.0 10.90.90.1
ip route xxx.xxx.75.0 255.255.252.0 10.90.90.1
<edited>
!
!
!
line con 0
transport input none
line aux 0
line vty 0 4
password router2
login
!
end
router2#sh proc cpu
CPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes:
0%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 0 28395 0 0.00% 0.00% 0.00% 0 Load Meter
2 200 166 1204 0.08% 0.01% 0.01% 2 Virtual Exec
3 125204 16803 7451 0.00% 0.06% 0.05% 0 Check heaps
4 16 29 551 0.00% 0.00% 0.00% 0 Pool Manager
5 0 2 0 0.00% 0.00% 0.00% 0 Timers
6 34064 161059 211 0.00% 0.02% 0.00% 0 ARP Input
7 0 2 0 0.00% 0.00% 0.00% 0 Serial
Backgroun
8 0 2 0 0.00% 0.00% 0.00% 0 DDR Timers
9 0 1 0 0.00% 0.00% 0.00% 0 Entity MIB
API
10 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler
11 0 2 0 0.00% 0.00% 0.00% 0 EPIF reset
proce
12 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone
Manager
13 24 141912 0 0.00% 0.00% 0.00% 0 IPC Periodic
Tim
14 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat
Manager
15 292 30807 9 0.00% 0.00% 0.00% 0 HC Counter
Timer
16 0 1 0 0.00% 0.00% 0.00% 0 SERIAL
A'detect
17 680 16 42500 0.00% 0.00% 0.00% 0 EPAM Card
Manage
18 4 9 444 0.00% 0.00% 0.00% 0 EPIF Enable
If p
19 0 1 0 0.00% 0.00% 0.00% 0 Critical
Bkgnd
20 2672 18091 147 0.00% 0.00% 0.00% 0 Net
Background
21 4 140 28 0.00% 0.00% 0.00% 0 Logger
22 120 141944 0 0.00% 0.00% 0.00% 0 TTY
Background
23 24 141911 0 0.00% 0.00% 0.00% 0 Per-Second
Jobs
24 84 141975 0 0.00% 0.00% 0.00% 0 LINE Monitor
Dea
25 0 1 0 0.00% 0.00% 0.00% 0 marker
26 75904 4734 16033 0.00% 0.05% 0.05% 0 THERMOSTAT
monit
27 32 141925 0 0.00% 0.00% 0.00% 0 LSIPC
Periodic T
28 148 154580 0 0.00% 0.00% 0.00% 0 LSIPC RECV
PROCE
29 52 703 73 0.00% 0.00% 0.00% 0 LSIPC BCAST
PROC
30 3892 55 70763 0.00% 0.00% 0.00% 0 LSS ACL/QOS
Proc
31 536 47304 11 0.00% 0.00% 0.00% 0 Envirnment
Monit
32 0 1 0 0.00% 0.00% 0.00% 0 PPP IP Add
Route
33 2728 20002 136 0.00% 0.00% 0.00% 0 CDP Protocol
34 57936 264025 219 0.00% 0.01% 0.00% 0 IP Input
35 11800 70 168571 0.00% 0.00% 0.00% 0 SCP Process
36 0 2 0 0.00% 0.00% 0.00% 0 DOMINO PORT
QOS
router2#sh ip cef g3 detail
IP CEF with switching (Table Version 599)
276 routes, 0 reresolve, 0 unresolved (0 old, 0 new)
276 leaves, 73 nodes, 110372 bytes, 599 inserts, 323 invalidations
0 load sharing elements, 0 bytes, 0 references
1 CEF resets, 0 revisions of existing leaves
refcounts: 19422 leaf, 18944 node
Adjacency Table has 185 adjacencies
10.10.8.0/22, version 6, cached adjacency xxx.xxx.117.199
0 packets, 0 bytes
via xxx.xxx.117.199, 0 dependencies, recursive
next hop xxx.xxx.117.199, GigabitEthernet3 via xxx.xxx.117.199/32
valid cached adjacency
10.10.10.0/24, version 7, cached adjacency xxx.xxx.117.1
0 packets, 0 bytes
via xxx.xxx.117.1, 0 dependencies, recursive
next hop xxx.xxx.117.1, GigabitEthernet3 via xxx.xxx.117.1/32
valid cached adjacency
10.10.11.0/24, version 8, cached adjacency xxx.xxx.117.199
0 packets, 0 bytes
via xxx.xxx.117.199, 0 dependencies, recursive
next hop xxx.xxx.117.199, GigabitEthernet3 via xxx.xxx.117.199/32
valid cached adjacency
10.10.16.0/22, version 9, cached adjacency xxx.xxx.117.1
0 packets, 0 bytes
via xxx.xxx.117.1, 0 dependencies, recursive
next hop xxx.xxx.117.1, GigabitEthernet3 via xxx.xxx.117.1/32
valid cached adjacency
router2#sh ip cef g1
Prefix Next Hop Interface
0.0.0.0/0 10.90.90.1 GigabitEthernet1
10.90.90.0/30 attached GigabitEthernet1
10.90.90.1/32 10.90.90.1 GigabitEthernet1
10.254.254.0/24 10.90.90.1 GigabitEthernet1
xxx.xxx.75.0/24 10.90.90.1 GigabitEthernet1
NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies and attachments of the original message.
More information about the cisco-nsp
mailing list