[c-nsp] OSPF and iBGP session drops between 3640s

Church, Charles cchurc05 at harris.com
Tue Mar 24 11:31:54 EDT 2009


That 12.4(3) IOS is pretty old.  Trying a newer one might help, as
you're vulnerable to many things.  It's possible there are bugs you're
hitting that are affecting performance.  If you could consolidate some
things, that may help.  You're matching RTP, but also matching packet
length, that might be overkill.  The fast hellos for OSPF probably
aren't helping either.  Another thought might be to score a 2950 or 3550
L2 switch, and put that in place of the 2924.  Then move all the ACls to
that, as it can do them in hardware.  You could probably do a little
buffer tuning, middle ones look pretty ugly.  Probably not long term
solution.  I think MCQ is more efficient than CAR, might want to move to
that completely.

Chuck


-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net
[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Robert Johnson
Sent: Tuesday, March 24, 2009 10:55 AM
To: cisco-nsp at puck.nether.net
Subject: [c-nsp] OSPF and iBGP session drops between 3640s


Hello list,
I have a small network with four 3640s. Each router has 128/32MB ram,
and a
single FE interface connected to a catalyst 2924. Two of the routers are
running BGP, each with a session to a (single) other provider, and a
session
between themselves. These are not carrying full tables. All four routers
are
running OSPF between each other. The problem is that occasionally (from
once
a week to 3x/day) OSPF neighbor relationships will bounce due to hello
timers expiring. Just recently the iBGP session between two of the
routers
also bounced.

There do not appear to be any layer 1 or 2 connectivity problems that
would
cause this behavior. However, CPU usage on the 3640s seems high- 30%
sustained and up to 90% peak, with only 1-2k max PPS. Also, I'm seeing
buffer misses and failures.

CEF is enabled. There are several relatively long access lists that are
being processed, and the routers are doing QoS classifying and tagging
at
layers 2 and 3 for VoIP performance.

Without any major hardware changes, where do I begin here?

Thanks in advance.



The fun stuff (sho buffers, sho proc cpu hist, sho proc cpu, sho run):

router1#sho buffers
Buffer elements:
     1118 in free list (500 max allowed)
     707983613 hits, 0 misses, 1119 created

Public buffer pools:
Small buffers, 104 bytes (total 78, permanent 50, peak 104 @ 4w0d):
     42 in free list (20 min, 150 max allowed)
     18990955 hits, 3598 misses, 4408 trims, 4436 created
     312 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent 25, peak 176 @ 7w0d):
     22 in free list (10 min, 150 max allowed)
     651012877 hits, 12602 misses, 30744 trims, 30744 created
     2744 failures (0 no memory)
Big buffers, 1536 bytes (total 50, permanent 50, peak 63 @ 2d19h):
     50 in free list (5 min, 150 max allowed)
     4658228 hits, 1005 misses, 102 trims, 102 created
     936 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 12 @ 7w0d):
     10 in free list (0 min, 100 max allowed)
     129 hits, 807 misses, 13 trims, 13 created
     807 failures (0 no memory)
Large buffers, 5024 bytes (total 1, permanent 0, peak 3 @ 7w0d):
     1 in free list (0 min, 10 max allowed)
     14 hits, 793 misses, 2764 trims, 2765 created
     793 failures (0 no memory)
Huge buffers, 18024 bytes (total 1, permanent 0, peak 3 @ 7w0d):
     1 in free list (0 min, 4 max allowed)
     16 hits, 779 misses, 3858 trims, 3859 created
     778 failures (0 no memory)

Interface buffer pools:
CD2430 I/O buffers, 1536 bytes (total 0, permanent 0):
     0 in free list (0 min, 0 max allowed)
     0 hits, 0 fallbacks

Header pools:
Header buffers, 0 bytes (total 265, permanent 256, peak 265 @ 7w0d):
     9 in free list (10 min, 512 max allowed)
     253 hits, 3 misses, 0 trims, 9 created
     0 failures (0 no memory)
     256 max cache size, 256 in cache
     7674266 hits in cache, 0 misses in cache

Particle Clones:
     1024 clones, 0 hits, 0 misses

Public particle pools:
F/S buffers, 256 bytes (total 384, permanent 384):
     128 in free list (128 min, 1024 max allowed)
     256 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
     256 max cache size, 256 in cache
     0 hits in cache, 0 misses in cache
Normal buffers, 1548 bytes (total 512, permanent 512):
     384 in free list (128 min, 1024 max allowed)
     21114 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
     128 max cache size, 128 in cache
     0 hits in cache, 0 misses in cache

Private particle pools:
IDS SM buffers, 240 bytes (total 128, permanent 128):
     0 in free list (0 min, 128 max allowed)
     128 hits, 0 fallbacks
     128 max cache size, 128 in cache
     0 hits in cache, 0 misses in cache
FastEthernet0/0 buffers, 1548 bytes (total 192, permanent 192):
     0 in free list (0 min, 192 max allowed)
     192 hits, 0 fallbacks
     192 max cache size, 128 in cache
     694772430 hits in cache, 20986 misses in cache

router1#sho proc cpu hist

router1   02:40:53 PM Tuesday Mar 24 2009 UTC


    4444444444444444444444444444444555554444444444444443333355
    8333332222200000000004444411111111110000000000222227777722
100
 90
 80
 70
 60
 50 *                              *****                    ****
 40 ************************************************************
 30 ************************************************************
 20 ************************************************************
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

    5656435544454334664445454566532243344446444645545774454545
    2900663259495363238448467911347711166900544033873220265057
100
 90
 80
 70  *                                               **
 60  * *       *    **   * * ***           *   *  *  **  *   *
 50 ***#* *#** **   ***  * **###*      *** **  * ****#*  *******
 40 ##*##*####*#* **####*#***###*   *  **********######****#####
 30 ##############################**#***##*#**##################
 20 ############################################################
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

 
7664665666756555557666776555554545664455555555455555654555556565545654
 
2734555279005332498657259890379052808981353640965081868135475217086638
100
 90
 80                   *    *
 70 **  **  *** *     *******         *                 *       *      *
*
 60 *** ******* *    **********  *    **  *  * *   ** * **   * ** **  **
**
 50 ***
********************************************************************
 40
****##########****#***##****########################*###################
 30
##**###########***########**############################################
 20
###*############*##########*############################################
 10
########################################################################
 
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5
0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%


sho proc cpu:

CPU utilization for five seconds: 42%/39%; one minute: 43%; five
minutes:
40%

router1#sho run
Building configuration...

Current configuration : 8460 bytes
!
! Last configuration change at 01:54:37 UTC Tue Mar 24 2009
! NVRAM config last updated at 22:25:51 UTC Thu Mar 5 2009
!
version 12.4
service timestamps debug uptime
service timestamps log uptime
service password-encryption
!
hostname router1
!
boot-start-marker
boot system flash c3640-jk9o3s-mz.124-3.bin
boot-end-marker
!
no logging console
enable secret 5 **
!
no aaa new-model
!
resource policy
!
ip subnet-zero
no ip source-route
!
!
ip cef
!
!
class-map match-all assure
 match ip dscp af31
class-map match-all critical
 match ip dscp cs6
class-map match-all expedite
 match ip dscp ef
class-map match-any rtp-vox
 match ip rtp 13456 13462
 match ip rtp 13556 13560
 match ip rtp 13656 13660
 match ip rtp 13756 13760
class-map match-all sip
 match protocol sip
class-map match-all voice
 match packet length min 1 max 200
 match class-map rtp-vox
!
!
policy-map output-cos
 class expedite
  set cos 6
 class assure
  set cos 5
 class critical
  set cos 7
policy-map input-mark
 class sip
  set ip dscp af31
 class voice
  set dscp ef
!
!
!
!
!
!
interface FastEthernet0/0
 description Trunk to cat2924-pri
 no ip address
 full-duplex
!
interface FastEthernet0/0.5
 description Switch management segment
 encapsulation dot1Q 5
 ip address 10.1.5.254 255.255.255.0
 ip access-group mgmt-only in
 ip access-group mgmt-only out
 no snmp trap link-status
!
interface FastEthernet0/0.15
 description AP management segment
 encapsulation dot1Q 15
 ip address 10.1.15.254 255.255.255.0
 ip access-group mgmt-only in
 ip access-group mgmt-only out
 no snmp trap link-status
!
interface FastEthernet0/0.25
 description CTM management segment
 encapsulation dot1Q 25
 ip address 10.1.25.254 255.255.255.0
 ip access-group mgmt-only in
 ip access-group mgmt-only out
 no snmp trap link-status
!
interface FastEthernet0/0.35
 description UPS management segment
 encapsulation dot1Q 35
 ip address 10.1.35.254 255.255.255.0
 ip access-group mgmt-only in
 ip access-group mgmt-only out
 no snmp trap link-status
!
interface FastEthernet0/0.50
 description Management link to router3
 bandwidth 9850
 encapsulation dot1Q 50
 ip address 10.1.50.254 255.255.255.0
 ip access-group mgmt-only in
 ip access-group mgmt-only out
 ip ospf message-digest-key 1 md5 7 *secret*
 ip ospf hello-interval 1
 ip ospf dead-interval 5
 no snmp trap link-status
!
interface FastEthernet0/0.51
 description Management link to router2
 encapsulation dot1Q 51
 ip address 10.1.51.254 255.255.255.0
 ip access-group mgmt-only in
 ip access-group mgmt-only out
 ip ospf message-digest-key 1 md5 7 **
 ip ospf hello-interval 1
 ip ospf dead-interval 5
 no snmp trap link-status
!
interface FastEthernet0/0.52
 description Management link to **
 bandwidth 10610
 encapsulation dot1Q 52
 ip address 10.1.52.254 255.255.255.0
 ip access-group mgmt-only in
 ip access-group mgmt-only out
 no snmp trap link-status
!
interface FastEthernet0/0.300
 description Production traffic link to router3
 bandwidth 9850
 encapsulation dot1Q 300
 ip address xxx.xxx.xxx.xxx 255.255.255.252
 ip ospf message-digest-key 10 md5 7 **
 ip ospf dead-interval minimal hello-multiplier 4
 no snmp trap link-status
 service-policy output output-cos
!
interface FastEthernet0/0.301
 description Production traffic link to router2
 encapsulation dot1Q 301
 ip address xxx.xxx.xxx.xxx 255.255.255.252
 ip ospf message-digest-key 10 md5 7 **
 ip ospf dead-interval minimal hello-multiplier 4
 no snmp trap link-status
 service-policy output output-cos
!
interface FastEthernet0/0.302
 description Production traffic link to **
 bandwidth 10610
 encapsulation dot1Q 302
 ip address xxx.xxx.xxx.xxx 255.255.255.252
 ip access-group internet-edge-ingress in
 ip access-group internet-edge-egress out
 no snmp trap link-status
 service-policy input input-mark
 service-policy output output-cos
!
interface FastEthernet0/0.500
 description Customer access subnet
 encapsulation dot1Q 500
 ip address xxx.xxx.xxx.xxx 255.255.255.240
 ip access-group block-customercrap in
 ip verify unicast reverse-path
 rate-limit input access-group 100 768000 10000 200000 conform-action
transmit e
xceed-action drop
 rate-limit output access-group 100 768000 40000000 80000000
conform-action
tran
smit exceed-action drop
 no snmp trap link-status
 service-policy output output-cos
!
router ospf 1000
 log-adjacency-changes
 area 0.0.0.0 authentication message-digest
 passive-interface default
 no passive-interface FastEthernet0/0.300
 no passive-interface FastEthernet0/0.301
 network xxx.xxx.xxx.xxx 0.0.0.63 area 0.0.0.0
 network xxx.xxx.xxx.xxx 0.0.0.63 area 0.0.0.0
 network xxx.xxx.xxx.xxx 0.0.0.63 area 0.0.0.0
 network xxx.xxx.xxx.xxx 0.0.0.63 area 0.0.0.0
 default-information originate metric-type 1
!
router ospf 100
 log-adjacency-changes
 area 10.0.0.0 authentication message-digest
 area 10.0.0.0 stub no-summary
 passive-interface default
 no passive-interface FastEthernet0/0.50
 no passive-interface FastEthernet0/0.51
 network 10.0.0.0 0.255.255.255 area 10.0.0.0
!
router bgp *****
 no synchronization
 bgp log-neighbor-changes
 network xxx.xxx.xxx.xxx mask 255.255.255.192
 network xxx.xxx.xxx.xxx mask 255.255.255.192
 network xxx.xxx.xxx.xxx mask 255.255.255.192
 network xxx.xxx.xxx.xxx mask 255.255.255.192
 aggregate-address xxx.xxx.xxx.xxx 255.255.255.192 as-set summary-only
 aggregate-address xxx.xxx.xxx.xxx 255.255.255.192 as-set summary-only
 aggregate-address xxx.xxx.xxx.xxx 255.255.255.192 as-set summary-only
 aggregate-address xxx.xxx.xxx.xxx 255.255.255.192 as-set summary-only
 redistribute ospf 1000
 neighbor xxx.xxx.xxx.xxx remote-as ****
 neighbor xxx.xxx.xxx.xxx route-map pri-map out
 neighbor xxx.xxx.xxx.xxx remote-as *****
 neighbor xxx.xxx.xxx.xxx next-hop-self
 no auto-summary
!
no ip http server
no ip http secure-server
ip classless
!
!
!
!
ip access-list standard mgmt-only
 permit 10.0.0.0 0.255.255.255
 permit 192.168.101.0 0.0.0.255
!
ip access-list extended block-customercrap
 deny   udp any any eq bootps
 deny   tcp any any eq 139
 deny   tcp any any eq 445
 deny   udp any any eq netbios-ns
 deny   udp any any eq netbios-dgm
 permit ip any any
ip access-list extended internet-edge-egress
 deny   ip 10.0.0.0 0.255.255.255 any
 deny   ip 172.16.0.0 0.15.255.255 any
 deny   ip 192.168.0.0 0.0.255.255 any
 deny   ip 127.0.0.0 0.0.255.255 any
 deny   ip 224.0.0.0 31.255.255.255 any
 deny   ip 169.254.0.0 0.0.255.255 any
 deny   udp any any eq bootps
 deny   udp any any eq bootpc
 deny   tcp any any eq 139
 deny   tcp any any eq 445
 deny   udp any any eq netbios-ns
 deny   udp any any eq netbios-dgm
 deny   ip any xxx.xxx.xxx.xxx 0.0.0.63
 deny   ip any xxx.xxx.xxx.xxx 0.0.0.63
 deny   ip any xxx.xxx.xxx.xxx 0.0.0.63
 deny   ip any xxx.xxx.xxx.xxx 0.0.0.63
 permit ip any any
ip access-list extended internet-edge-ingress
 deny   ip 10.0.0.0 0.255.255.255 any
 deny   ip 172.16.0.0 0.15.255.255 any
 deny   ip 192.168.0.0 0.0.255.255 any
 deny   ip 127.0.0.0 0.0.255.255 any
 deny   ip 224.0.0.0 31.255.255.255 any
 deny   ip 169.254.0.0 0.0.255.255 any
 deny   udp any any eq bootps
 deny   udp any any eq bootpc
 deny   tcp any any eq 139
 deny   tcp any any eq 445
 deny   udp any any eq netbios-ns
 deny   udp any any eq netbios-dgm
 deny   ip xxx.xxx.xxx.xxx 0.0.0.63 any
 deny   ip xxx.xxx.xxx.xxx 0.0.0.63 any
 deny   ip xxx.xxx.xxx.xxx 0.0.0.63 any
 deny   ip xxx.xxx.xxx.xxx 0.0.0.63 any
 permit ip any any
logging facility local5
logging 10.3.40.105
access-list 1 permit xxx.xxx.xxx.xxx 0.0.0.63
access-list 1 permit xxx.xxx.xxx.xxx 0.0.0.63
access-list 2 permit xxx.xxx.xxx.xxx 0.0.0.63
access-list 2 permit xxx.xxx.xxx.xxx 0.0.0.63
access-list 100 permit ip host xxx.xxx.xxx.xxx any
access-list 100 permit ip any host xxx.xxx.xxx.xxx
snmp-server community ** RO mgmt-only
!
route-map pri-map permit 10
 match ip address 1
!
route-map pri-map permit 20
 match ip address 2
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
banner login ^C Property of **. Unauthorized access attempt
s will be prosecuted. ^C
!
line con 0
 password 7 **
 login
line aux 0
 password 7 **
 login
line vty 0 4
 access-class mgmt-only in
 password 7 **
 login
!
ntp clock-period 17179597
ntp server 10.3.40.105
!
end
_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


More information about the cisco-nsp mailing list