[nsp] High CPU question (update)

[Bryan]

High all, so this is getting more interesting.

Seems the customer is pushing 85% UDP traffic w/ packet sizes less than
128, more like 64byte.

They pulled a 7206VXR NPE-400 from another environment and loaded the same
config.  The router saw maybe a 10% improvement over the 7206non-VXR w/
NPE-225.

At bout 40Mb/s the router is seeing about 70% cpu, when it goes to ~50-55
and above it hits 100% (100/100 on sh proc cpu) very often and average
over 5hrs was around 95%

I've got CEF enabled and verified it's operational with very few
"non-cef-switched" packets going through the router.

I've gone through Cisco's docs on "troubleshooting high cpu" and
"troubleshooting high cpu due to interrupts".  The router isn't
experiencing any "problems" according to these.

Here is a config to take a peek at:

Using 3904 out of 129016 bytes
!
! Last configuration change at 13:32:22 PDT Thu Apr 29 2004
! NVRAM config last updated at 13:32:28 PDT Thu Apr 29 2004
!
version 12.3
service nagle
no service pad
service timestamps debug uptime
service timestamps log datetime localtime
service password-encryption
!
hostname <blah>
!
boot-start-marker
boot-end-marker
!
no logging console
no logging monitor
enable secret 5
enable password 7
!
username <blah>

clock timezone PST -8
clock summer-time PDT recurring
regexp optimize
no aaa new-model
ip subnet-zero
no ip source-route
no ip icmp rate-limit unreachable
no ip icmp rate-limit unreachable DF
!
!
ip tcp path-mtu-discovery
ip name-server x.x.62.1
!
ip cef
!
!
!!
!
interface Loopback0
 ip address 2.2.2.179 255.255.255.255
 ip route-cache flow
 load-interval 120
!
interface FastEthernet0/0
  no ip redirects
 no ip mroute-cache
 load-interval 120
 shutdown
 duplex full
!
interface FastEthernet1/0
 description TO-NSP#1
 ip address 100.0.220.210 255.255.255.252
 no ip redirects
 no ip proxy-arp
 ip route-cache flow
 no ip mroute-cache
 load-interval 120
 duplex full
!
interface FastEthernet2/0
 ip address 100.0.220.214 255.255.255.252
 no ip redirects
 no ip proxy-arp
 ip route-cache flow
 no ip mroute-cache
 load-interval 120
 duplex full
!
interface FastEthernet3/0
 ip address 10.100.101.254 255.255.254.0
 no ip redirects
 no ip proxy-arp
 ip route-cache flow
 no ip mroute-cache
 load-interval 120
 duplex full
!
interface FastEthernet4/0
 ip address 10.10.155.254 255.255.255.0
 no ip redirects
 no ip proxy-arp
 ip route-cache flow
 no ip mroute-cache
 load-interval 120
 duplex full
!
interface FastEthernet6/0
 ip address 10.10.188.254 255.255.255.0
 no ip redirects
 no ip proxy-arp
 ip route-cache flow
 no ip mroute-cache
 duplex full
!
router bgp 1234
 no synchronization
 no bgp fast-external-fallover
 bgp log-neighbor-changes
 network 10.100.100.0 mask 255.255.254.0
 network 10.10.155.0 mask 255.255.255.0
 network 10.10.188.0 mask 255.255.255.0
 neighbor 1.1.255.158 remote-as 4567
 neighbor 1.1.255.158 description TO-NSP
 neighbor 1.1.255.158 ebgp-multihop 2
 neighbor 1.1.255.158 update-source Loopback0
 neighbor 1.1.255.158 send-community
 neighbor 1.1.255.158 remove-private-AS
 neighbor 1.1.255.158 version 4
 neighbor 1.1.255.158 soft-reconfiguration inbound
 neighbor 1.1.255.158 distribute-list BGP-TO-NSP out
 neighbor 1.1.255.158 route-map OTHER-OUT out
 neighbor 1.1.255.158 password 7
 neighbor 1.1.255.158 maximum-prefix 175000
 no auto-summary
!
ip classless
ip route 10.100.100.0 255.255.254.0 1.1.255.158
ip route 10.10.155.0 255.255.255.0 1.1.255.158
ip route 10.10.188.0 255.255.255.0 1.1.255.158
ip route 100.0.220.208 255.255.255.248 100.0.220.209
ip route 100.0.220.208 255.255.255.248 100.0.220.213
ip route 1.1.255.158 255.255.255.255 100.0.220.209
ip route 1.1.255.158 255.255.255.255 100.0.220.213
no ip http server
!
ip bgp-community new-format
ip as-path access-list 2 permit ^$
ip as-path access-list 10 deny ^$
!
!
ip access-list extended BGP-TO-NSP
 permit ip 10.100.100.0 0.0.1.255 any
 permit ip 10.10.155.0 0.0.0.255 any
 permit ip 10.10.188.0 0.0.0.255 any
!
route-map OTHER-OUT permit 10
 match as-path 2
 set community 64666:7890
!
!
!
no call rsvp-sync
!
!
dial-peer cor custom
!
!
!
!
gatekeeper
 shutdown
!
!
line con 0
 stopbits 1
line aux 0
 exec-timeout 0 0
 password 7
 login
 modem DTR-active
 no exec
 transport input all
 stopbits 1
line vty 0 4
 password 7
 login
line vty 5 9
 no login
!
ntp update-calendar
ntp server 1.2.3.4
ntp server 1.2.3.5
!
!
end


Thanks for all the help



Bryan

On Sun, 25 Apr 2004, Gert Doering wrote:

> Date: Sun, 25 Apr 2004 22:02:34 +0200
> From: Gert Doering <gert at greenie.muc.de>
> To: James Edwards <hackerwacker at cybermesa.com>
> Cc: cisco-nsp at puck.nether.net
> Subject: Re: [nsp] High CPU question
>
> Hi,
>
> On Sun, Apr 25, 2004 at 12:59:36PM -0600, James Edwards wrote:
> > Does the high CPU% cycle at regular intervals or is it random ?
> >
> > Try a "sho process cpu | exclude 0.00" when the CPU is maxed
> > to see what process is eating up the most CPU %.
>
> The example line shown was 66%/66% - this is IRQ load, and won't show up
> in "show proc cpu".
>
> gert
>