[c-nsp] OSPF and iBGP session drops between 3640s
Robert Johnson
fasterfourier at gmail.com
Tue Mar 24 10:55:11 EDT 2009
Hello list,
I have a small network with four 3640s. Each router has 128/32MB ram, and a
single FE interface connected to a catalyst 2924. Two of the routers are
running BGP, each with a session to a (single) other provider, and a session
between themselves. These are not carrying full tables. All four routers are
running OSPF between each other. The problem is that occasionally (from once
a week to 3x/day) OSPF neighbor relationships will bounce due to hello
timers expiring. Just recently the iBGP session between two of the routers
also bounced.
There do not appear to be any layer 1 or 2 connectivity problems that would
cause this behavior. However, CPU usage on the 3640s seems high- 30%
sustained and up to 90% peak, with only 1-2k max PPS. Also, I'm seeing
buffer misses and failures.
CEF is enabled. There are several relatively long access lists that are
being processed, and the routers are doing QoS classifying and tagging at
layers 2 and 3 for VoIP performance.
Without any major hardware changes, where do I begin here?
Thanks in advance.
The fun stuff (sho buffers, sho proc cpu hist, sho proc cpu, sho run):
router1#sho buffers
Buffer elements:
1118 in free list (500 max allowed)
707983613 hits, 0 misses, 1119 created
Public buffer pools:
Small buffers, 104 bytes (total 78, permanent 50, peak 104 @ 4w0d):
42 in free list (20 min, 150 max allowed)
18990955 hits, 3598 misses, 4408 trims, 4436 created
312 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent 25, peak 176 @ 7w0d):
22 in free list (10 min, 150 max allowed)
651012877 hits, 12602 misses, 30744 trims, 30744 created
2744 failures (0 no memory)
Big buffers, 1536 bytes (total 50, permanent 50, peak 63 @ 2d19h):
50 in free list (5 min, 150 max allowed)
4658228 hits, 1005 misses, 102 trims, 102 created
936 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 12 @ 7w0d):
10 in free list (0 min, 100 max allowed)
129 hits, 807 misses, 13 trims, 13 created
807 failures (0 no memory)
Large buffers, 5024 bytes (total 1, permanent 0, peak 3 @ 7w0d):
1 in free list (0 min, 10 max allowed)
14 hits, 793 misses, 2764 trims, 2765 created
793 failures (0 no memory)
Huge buffers, 18024 bytes (total 1, permanent 0, peak 3 @ 7w0d):
1 in free list (0 min, 4 max allowed)
16 hits, 779 misses, 3858 trims, 3859 created
778 failures (0 no memory)
Interface buffer pools:
CD2430 I/O buffers, 1536 bytes (total 0, permanent 0):
0 in free list (0 min, 0 max allowed)
0 hits, 0 fallbacks
Header pools:
Header buffers, 0 bytes (total 265, permanent 256, peak 265 @ 7w0d):
9 in free list (10 min, 512 max allowed)
253 hits, 3 misses, 0 trims, 9 created
0 failures (0 no memory)
256 max cache size, 256 in cache
7674266 hits in cache, 0 misses in cache
Particle Clones:
1024 clones, 0 hits, 0 misses
Public particle pools:
F/S buffers, 256 bytes (total 384, permanent 384):
128 in free list (128 min, 1024 max allowed)
256 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
256 max cache size, 256 in cache
0 hits in cache, 0 misses in cache
Normal buffers, 1548 bytes (total 512, permanent 512):
384 in free list (128 min, 1024 max allowed)
21114 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
128 max cache size, 128 in cache
0 hits in cache, 0 misses in cache
Private particle pools:
IDS SM buffers, 240 bytes (total 128, permanent 128):
0 in free list (0 min, 128 max allowed)
128 hits, 0 fallbacks
128 max cache size, 128 in cache
0 hits in cache, 0 misses in cache
FastEthernet0/0 buffers, 1548 bytes (total 192, permanent 192):
0 in free list (0 min, 192 max allowed)
192 hits, 0 fallbacks
192 max cache size, 128 in cache
694772430 hits in cache, 20986 misses in cache
router1#sho proc cpu hist
router1 02:40:53 PM Tuesday Mar 24 2009 UTC
4444444444444444444444444444444555554444444444444443333355
8333332222200000000004444411111111110000000000222227777722
100
90
80
70
60
50 * ***** ****
40 ************************************************************
30 ************************************************************
20 ************************************************************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
5656435544454334664445454566532243344446444645545774454545
2900663259495363238448467911347711166900544033873220265057
100
90
80
70 * **
60 * * * ** * * *** * * * ** * *
50 ***#* *#** ** *** * **###* *** ** * ****#* *******
40 ##*##*####*#* **####*#***###* * **********######****#####
30 ##############################**#***##*#**##################
20 ############################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
7664665666756555557666776555554545664455555555455555654555556565545654
2734555279005332498657259890379052808981353640965081868135475217086638
100
90
80 * *
70 ** ** *** * ******* * * * * *
60 *** ******* * ********** * ** * * * ** * ** * ** ** ** **
50 *** ********************************************************************
40 ****##########****#***##****########################*###################
30 ##**###########***########**############################################
20 ###*############*##########*############################################
10 ########################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
sho proc cpu:
CPU utilization for five seconds: 42%/39%; one minute: 43%; five minutes:
40%
router1#sho run
Building configuration...
Current configuration : 8460 bytes
!
! Last configuration change at 01:54:37 UTC Tue Mar 24 2009
! NVRAM config last updated at 22:25:51 UTC Thu Mar 5 2009
!
version 12.4
service timestamps debug uptime
service timestamps log uptime
service password-encryption
!
hostname router1
!
boot-start-marker
boot system flash c3640-jk9o3s-mz.124-3.bin
boot-end-marker
!
no logging console
enable secret 5 **
!
no aaa new-model
!
resource policy
!
ip subnet-zero
no ip source-route
!
!
ip cef
!
!
class-map match-all assure
match ip dscp af31
class-map match-all critical
match ip dscp cs6
class-map match-all expedite
match ip dscp ef
class-map match-any rtp-vox
match ip rtp 13456 13462
match ip rtp 13556 13560
match ip rtp 13656 13660
match ip rtp 13756 13760
class-map match-all sip
match protocol sip
class-map match-all voice
match packet length min 1 max 200
match class-map rtp-vox
!
!
policy-map output-cos
class expedite
set cos 6
class assure
set cos 5
class critical
set cos 7
policy-map input-mark
class sip
set ip dscp af31
class voice
set dscp ef
!
!
!
!
!
!
interface FastEthernet0/0
description Trunk to cat2924-pri
no ip address
full-duplex
!
interface FastEthernet0/0.5
description Switch management segment
encapsulation dot1Q 5
ip address 10.1.5.254 255.255.255.0
ip access-group mgmt-only in
ip access-group mgmt-only out
no snmp trap link-status
!
interface FastEthernet0/0.15
description AP management segment
encapsulation dot1Q 15
ip address 10.1.15.254 255.255.255.0
ip access-group mgmt-only in
ip access-group mgmt-only out
no snmp trap link-status
!
interface FastEthernet0/0.25
description CTM management segment
encapsulation dot1Q 25
ip address 10.1.25.254 255.255.255.0
ip access-group mgmt-only in
ip access-group mgmt-only out
no snmp trap link-status
!
interface FastEthernet0/0.35
description UPS management segment
encapsulation dot1Q 35
ip address 10.1.35.254 255.255.255.0
ip access-group mgmt-only in
ip access-group mgmt-only out
no snmp trap link-status
!
interface FastEthernet0/0.50
description Management link to router3
bandwidth 9850
encapsulation dot1Q 50
ip address 10.1.50.254 255.255.255.0
ip access-group mgmt-only in
ip access-group mgmt-only out
ip ospf message-digest-key 1 md5 7 *secret*
ip ospf hello-interval 1
ip ospf dead-interval 5
no snmp trap link-status
!
interface FastEthernet0/0.51
description Management link to router2
encapsulation dot1Q 51
ip address 10.1.51.254 255.255.255.0
ip access-group mgmt-only in
ip access-group mgmt-only out
ip ospf message-digest-key 1 md5 7 **
ip ospf hello-interval 1
ip ospf dead-interval 5
no snmp trap link-status
!
interface FastEthernet0/0.52
description Management link to **
bandwidth 10610
encapsulation dot1Q 52
ip address 10.1.52.254 255.255.255.0
ip access-group mgmt-only in
ip access-group mgmt-only out
no snmp trap link-status
!
interface FastEthernet0/0.300
description Production traffic link to router3
bandwidth 9850
encapsulation dot1Q 300
ip address xxx.xxx.xxx.xxx 255.255.255.252
ip ospf message-digest-key 10 md5 7 **
ip ospf dead-interval minimal hello-multiplier 4
no snmp trap link-status
service-policy output output-cos
!
interface FastEthernet0/0.301
description Production traffic link to router2
encapsulation dot1Q 301
ip address xxx.xxx.xxx.xxx 255.255.255.252
ip ospf message-digest-key 10 md5 7 **
ip ospf dead-interval minimal hello-multiplier 4
no snmp trap link-status
service-policy output output-cos
!
interface FastEthernet0/0.302
description Production traffic link to **
bandwidth 10610
encapsulation dot1Q 302
ip address xxx.xxx.xxx.xxx 255.255.255.252
ip access-group internet-edge-ingress in
ip access-group internet-edge-egress out
no snmp trap link-status
service-policy input input-mark
service-policy output output-cos
!
interface FastEthernet0/0.500
description Customer access subnet
encapsulation dot1Q 500
ip address xxx.xxx.xxx.xxx 255.255.255.240
ip access-group block-customercrap in
ip verify unicast reverse-path
rate-limit input access-group 100 768000 10000 200000 conform-action
transmit e
xceed-action drop
rate-limit output access-group 100 768000 40000000 80000000 conform-action
tran
smit exceed-action drop
no snmp trap link-status
service-policy output output-cos
!
router ospf 1000
log-adjacency-changes
area 0.0.0.0 authentication message-digest
passive-interface default
no passive-interface FastEthernet0/0.300
no passive-interface FastEthernet0/0.301
network xxx.xxx.xxx.xxx 0.0.0.63 area 0.0.0.0
network xxx.xxx.xxx.xxx 0.0.0.63 area 0.0.0.0
network xxx.xxx.xxx.xxx 0.0.0.63 area 0.0.0.0
network xxx.xxx.xxx.xxx 0.0.0.63 area 0.0.0.0
default-information originate metric-type 1
!
router ospf 100
log-adjacency-changes
area 10.0.0.0 authentication message-digest
area 10.0.0.0 stub no-summary
passive-interface default
no passive-interface FastEthernet0/0.50
no passive-interface FastEthernet0/0.51
network 10.0.0.0 0.255.255.255 area 10.0.0.0
!
router bgp *****
no synchronization
bgp log-neighbor-changes
network xxx.xxx.xxx.xxx mask 255.255.255.192
network xxx.xxx.xxx.xxx mask 255.255.255.192
network xxx.xxx.xxx.xxx mask 255.255.255.192
network xxx.xxx.xxx.xxx mask 255.255.255.192
aggregate-address xxx.xxx.xxx.xxx 255.255.255.192 as-set summary-only
aggregate-address xxx.xxx.xxx.xxx 255.255.255.192 as-set summary-only
aggregate-address xxx.xxx.xxx.xxx 255.255.255.192 as-set summary-only
aggregate-address xxx.xxx.xxx.xxx 255.255.255.192 as-set summary-only
redistribute ospf 1000
neighbor xxx.xxx.xxx.xxx remote-as ****
neighbor xxx.xxx.xxx.xxx route-map pri-map out
neighbor xxx.xxx.xxx.xxx remote-as *****
neighbor xxx.xxx.xxx.xxx next-hop-self
no auto-summary
!
no ip http server
no ip http secure-server
ip classless
!
!
!
!
ip access-list standard mgmt-only
permit 10.0.0.0 0.255.255.255
permit 192.168.101.0 0.0.0.255
!
ip access-list extended block-customercrap
deny udp any any eq bootps
deny tcp any any eq 139
deny tcp any any eq 445
deny udp any any eq netbios-ns
deny udp any any eq netbios-dgm
permit ip any any
ip access-list extended internet-edge-egress
deny ip 10.0.0.0 0.255.255.255 any
deny ip 172.16.0.0 0.15.255.255 any
deny ip 192.168.0.0 0.0.255.255 any
deny ip 127.0.0.0 0.0.255.255 any
deny ip 224.0.0.0 31.255.255.255 any
deny ip 169.254.0.0 0.0.255.255 any
deny udp any any eq bootps
deny udp any any eq bootpc
deny tcp any any eq 139
deny tcp any any eq 445
deny udp any any eq netbios-ns
deny udp any any eq netbios-dgm
deny ip any xxx.xxx.xxx.xxx 0.0.0.63
deny ip any xxx.xxx.xxx.xxx 0.0.0.63
deny ip any xxx.xxx.xxx.xxx 0.0.0.63
deny ip any xxx.xxx.xxx.xxx 0.0.0.63
permit ip any any
ip access-list extended internet-edge-ingress
deny ip 10.0.0.0 0.255.255.255 any
deny ip 172.16.0.0 0.15.255.255 any
deny ip 192.168.0.0 0.0.255.255 any
deny ip 127.0.0.0 0.0.255.255 any
deny ip 224.0.0.0 31.255.255.255 any
deny ip 169.254.0.0 0.0.255.255 any
deny udp any any eq bootps
deny udp any any eq bootpc
deny tcp any any eq 139
deny tcp any any eq 445
deny udp any any eq netbios-ns
deny udp any any eq netbios-dgm
deny ip xxx.xxx.xxx.xxx 0.0.0.63 any
deny ip xxx.xxx.xxx.xxx 0.0.0.63 any
deny ip xxx.xxx.xxx.xxx 0.0.0.63 any
deny ip xxx.xxx.xxx.xxx 0.0.0.63 any
permit ip any any
logging facility local5
logging 10.3.40.105
access-list 1 permit xxx.xxx.xxx.xxx 0.0.0.63
access-list 1 permit xxx.xxx.xxx.xxx 0.0.0.63
access-list 2 permit xxx.xxx.xxx.xxx 0.0.0.63
access-list 2 permit xxx.xxx.xxx.xxx 0.0.0.63
access-list 100 permit ip host xxx.xxx.xxx.xxx any
access-list 100 permit ip any host xxx.xxx.xxx.xxx
snmp-server community ** RO mgmt-only
!
route-map pri-map permit 10
match ip address 1
!
route-map pri-map permit 20
match ip address 2
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
banner login ^C Property of **. Unauthorized access attempt
s will be prosecuted. ^C
!
line con 0
password 7 **
login
line aux 0
password 7 **
login
line vty 0 4
access-class mgmt-only in
password 7 **
login
!
ntp clock-period 17179597
ntp server 10.3.40.105
!
end
More information about the cisco-nsp
mailing list