[c-nsp] High CPU - 7206VXR

Tue Nov 20 21:53:19 EST 2007

Hi,

  a.. Interrupt switching is disabled on an interface (or interfaces) that has (have) a lot of traffic

  Interrupt switching refers to the use of switching algorithms other than process switching. Examples include fast switching, optimum switching, Cisco Express Forwarding switching, and so on (refer to Performance Tuning Basics for details). Examine the output of the show interfaces switching command to see which interface is burdened with traffic. You can check the show ip interface command to see which switching method(s) are used on each interface. Re-enable interrupt switching on that interface. Remember that regular fast switching is configured on output interfaces: if fast switching is configured on an interface, packets that go out of that interface are fast-switched. Cisco Express Forwarding switching is configured on input interfaces. To create Forwarding Information Base (FIB) and adjacency table entries on a particular interface, configure Cisco Express Forwarding switching on all interfaces that route to that interface.

  b.. Fast switching on the same interface is disabled

  If an interface has a lot of secondary addresses or subinterfaces and there is a lot of traffic sourced from the interface and destined for an address on that same interface, then all of those packets are process-switched. In this situation, you should enable ip route-cache same-interface on the interface. When Cisco Express Forwarding switching is used, you do not need to enable Cisco Express Forwarding switching on the same interface separately.

  c.. Fast switching on an interface providing policy routing is disabled

  If a route-map has been configured on an interface, and a lot of traffic is handled by the route-map, then the router process-switches this traffic. In this situation, you should enable ip route-cache policy on the interface. Check the restrictions mentioned in the "Enabling Fast-Switched Policy-Based Routing" section of .

  d.. Traffic that cannot be interrupt-switched arrives

  This can be any of the listed types of traffic. Click on linked items for more information.

    a.. Packets for which there is no entry yet in the switching cache

    Even if fast, optimum, or Cisco Express Forwarding switching (CEF) is configured, a packet for which there is no match in the fast-switching cache or FIB and adjacency tables is processed. An entry is then created in the appropriate cache or table, and all subsequent packets that match the same criteria are fast, optimum, or CEF-switched. In normal circumstances, these processed packets do not cause high CPU utilization. However, if there is a device in the network which 1) generates packets at an extremely high rate for devices reachable through the router, and 2) uses different source or destination IP addresses, there is not a match for these packets in the switching cache or table, so they are processed by the IP Input process (if NetFlow switching is configured, source and destination TCP ports are checked against entries in the NetFlow cache as well). This source device can be a non-functional device or, more likely, a device attempting an attack.

    (*) Only with glean adjacencies. Refer to Cisco Express Forwarding documentation for more information about Cisco Express Forwarding adjacencies.

    b.. Packets destined for the router

    These are examples of packets destined for the router:

      a.. Routing updates that arrive at an extremely high rate. If the router receives an enormous amount of routing updates that have to be processed, this task might overload the CPU. Normally, this cannot happen in a stable network. The way you can gather more information depends on the routing protocol you have configured. However, you can start to check the output of the show ip route summary command periodically. Values that change rapidly are a sign of an unstable network. Frequent routing table changes mean increased routing protocol processing, which results in increased CPU utilization. For further information on how to troubleshoot this issue, refer to the Troubleshooting TCP/IP section of the Internetwork Troubleshooting Guide.

      b.. Any other kind of traffic destined for the router. Check who is logged on to the router and user actions. If someone is logged on and issues commands that produce long output, the high CPU utilization by the "IP input" process is followed by a much higher CPU utilization by the Virtual Exec process.

      c.. Spoof attack. To identify the problem, issue the show ip traffic command to check the amount of IP traffic. If there is a problem, the number of received packets with a local destination is significant. Next, examine the output of the show interfaces and show interfaces switching commands to check which interface the packets are coming in. Once you have identified the receiving interface, turn on ip accounting on the outgoing interface and see if there is a pattern. If there is an attack, the source address is almost always different, but the destination address is the same. An access list can be configured to solve the issue temporarily (preferably on the device closest to the source of the packets), but the real solution is to track down the source device and stop the attack.

    c.. Broadcast traffic

    Check the number of broadcast packets in the show interfaces command output. If you compare the amount of broadcasts to the total amount of packets that were received on the interface, you can gain an idea of whether there is an overhead of broadcasts. If there is a LAN with several switches connected to the router, then this can indicate a problem with Spanning Tree.

    d.. IP packets with options

    e.. Packets that require protocol translation

    f.. Multilink Point-to-Point Protocol (supported in Cisco Express Forwarding switching)

    g.. Compressed traffic

    If there is no Compression Service Adapter (CSA) in the router, compressed packets must be process-switched.

    h.. Encrypted traffic

    If there is no Encryption Service Adapter (ESA) in the router, encrypted packets must be process-switched.

    i.. Packets that go through serial interfaces with X.25 encapsulation

    In the X.25 protocol suite, flow control is implemented on the second Open System Interconnection (OSI) layer.

  e.. A lot of packets, that arrive at an extremely high rate, for a destination in a directly attached subnet, for which there is no entry in the Address Resolution Protocol (ARP) table. This should not happen with TCP traffic because of the windowing mechanism, but can happen with User Datagram Protocol (UDP) traffic. To identify the problem, repeat the actions suggested in order to track down a spoof attack.

  f.. A lot of multicast traffic goes through the router. Unfortunately, there is no easy way to examine the amount of multicast traffic. The show ip traffic command only shows summary information. However, if you have configured multicast routing on the router, you can enable fast-switching of multicast packets with the ip mroute-cache interface configuration command (fast-switching of multicast packets is off by default).

  g.. Router is oversubscribed. If the router is over-used and cannot handle this amount of traffic, try to distribute the load among other routers or purchase a high-end router.

  h.. IP Network Address Translation (NAT) is configured on the router, and lots of Domain Name System (DNS) packets go through the router. UDP or TCP packets with source or destination port 53 (DNS) are always punted to process level by NAT.

  i.. There are other packet types that are punted to processing.

rgs
a. rahman isnaini r. sutan

----- Original Message ----- 
From: "Łukasz Bromirski" <lukasz at bromirski.net>
To: "Paul Stewart" <paul at paulstewart.org>
Cc: <cisco-nsp at puck.nether.net>
Sent: Tuesday, November 20, 2007 10:39 PM
Subject: Re: [c-nsp] High CPU - 7206VXR

Paul Stewart wrote:

> Our CPU on this box is very high hitting over 80% at peak times with only
> 100Mb/s of PPPOE traffic.  This should be in the 35-40% range so I'm trying
> to figure out why it's so high.  Below is a CPU snapshot with only 69Mbps of
> traffic:
> CPU utilization for five seconds: 55%/39%; one minute: 56%; five minutes: 56%
>  PID Runtime(ms)    Invoked      uSecs   5Sec   1Min   5Min TTY Process
>   71   458570796 1703950621          0 11.87% 11.27% 11.11%   0 IP Input

[...]

> Any input is most appreciated...

It seems in that case something is taking 11% of CPU time because
it can't be CEF-switched. Try to identify the feature using with
'sh cef drop', 'sh cef not-cef-switched', I'm not sure 12.4(17) would
be the best one for PPPoE aggregation - propably 12.2SB would do better
given it supports all the required features (we didn't see your config).

-- 
"Don't expect me to cry for all the     |               Łukasz Bromirski
  reasons you had to die" -- Kurt Cobain |    http://lukasz.bromirski.net
_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/