[c-nsp] 3750 Dropping into software forwarding?

[Peter Kranz]

Had one of our customer's 3750's jump in CPU utilization from its normal 18%
or so to 80%, and the amount of traffic it would forward drop to around 100
Mb/s via one of its BGP peers. I tried dropping a few routes from the peer
and suddenly CPU utilization dropped back to 18% and all's well. This box
has about ~3500 routes on it. No errors showed up in the logs. Running Cisco
IOS Software, C3750 Software (C3750-I5-M), Version 12.2(25)SEA, RELEASE
SOFTWARE (fc)

I noticed this in the CEF memory, and I'm guessing I've hit a memory limit
in CEF causing it to dump back into software forwarding. Will another SDM
profile help this situation, I'm currently in "desktop default".

sho cef memory
  Memory                          in use/allocated         Count
  ------------------------------------------------------------------
  ADJ: DROP adjacency       :        344/400        ( 86%) [1]
  ADJ: Discard adjacency    :        344/400        ( 86%) [1]
  ADJ: Glean adjacency      :        344/400        ( 86%) [1]
  ADJ: NULL adjacency       :        344/400        ( 86%) [1]
  ADJ: PUNT adjacency       :        344/400        ( 86%) [1]
  ADJ: adj purge context    :       2880/2936       ( 98%) [1]
  ADJ: adjacency            :     132512/132736     ( 99%) [4]
  ADJ: sevs                 :      10064/10120      ( 99%) [1]
  CEF: Brkr Consumer        :        288/624        ( 46%) [6]
  CEF: Brkr Mcast grp       :        240/576        ( 41%) [6]
  CEF: Brkr Update Rec      :      74992/75104      ( 99%) [2]
  CEF: Brkr VRF group       :        144/480        ( 30%) [6]
  CEF: Brkr zone            :        320/992        ( 32%) [12]
  CEF: Broker               :        912/1248       ( 73%) [6]
  CEF: Brokers Array        :         56/112        ( 50%) [1]
  CEF: EVENT msg chunks     :       5000/5056       ( 98%) [1]
  CEF: FIB LC               :       6800/6856       ( 99%) [1]
  CEF: FIB LC array         :        168/224        ( 75%) [1]
  CEF: FIB LC stats array   :       6688/6744       ( 99%) [1]
  CEF: FIBHWIDB             :      40848/44712      ( 91%) [69]
  CEF: FIBIDB               :      13864/17728      ( 78%) [69]
  CEF: FIBSWSB control      :        792/1296       ( 61%) [9]
  CEF: HWIDB MAP TABLE      :      25200/25256      ( 99%) [1]
  CEF: IPv4 ARP throttle    :       1024/1080       ( 94%) [1]
  CEF: IPv4 not-sw cnt      :         72/128        ( 56%) [1]
  CEF: IPv4 not-sw si       :         72/128        ( 56%) [1]
  CEF: IPv4 process         :        408/744        ( 54%) [6]
  CEF: TABLE msg chunks     :      20000/20056      ( 99%) [1]
  CEF: Table LC info        :        344/400        ( 86%) [1]
  CEF: Table active slot in :         48/104        ( 46%) [1]
  CEF: arp throttle chunk   :      14096/14152      ( 99%) [1]
  CEF: dQ elems             :      78736/78848      ( 99%) [2]
  CEF: dQ walks             :        920/1032       ( 89%) [2]
  CEF: fib                  :     327680/327960     ( 99%) [5]
  CEF: fib GSB              :      65536/65592      ( 99%) [1]
  CEF: fib deps             :       3520/3632       ( 96%) [2]
  CEF: fib its              :      10648/10760      ( 98%) [2]
  CEF: fib_fib msg chunks   :      65536/65592      ( 99%) [1]
  CEF: fib_fib_s            :       6000/6112       ( 98%) [2]
  CEF: fib_fib_src_adj_sb   :       3000/3056       ( 98%) [1]
  CEF: fib_fib_src_special_ :       3000/3056       ( 98%) [1]
  CEF: fibidb table         :      25200/25256      ( 99%) [1]
  CEF: hash table           :     262144/262256     ( 99%) [2]
  CEF: hwapi swobj walks    :       1656/1768       ( 93%) [2]
  CEF: loadinfos            :      33488/33600      ( 99%) [2]
  CEF: mpls long path exts  :      69728/69840      ( 99%) [2]
  CEF: mpls path exts       :      73824/73936      ( 99%) [2]
  CEF: nh entry context     :      10808/10920      ( 98%) [2]
  CEF: non_ip entry context :      10720/10832      ( 98%) [2]
  CEF: oce dependent walks  :        912/1024       ( 89%) [2]
  CEF: path list dependents :      10648/10760      ( 98%) [2]
  CEF: pathl                :      31944/32280      ( 98%) [6]
  CEF: pathl ifs            :       5648/5760       ( 98%) [2]
  CEF: pathl its            :      10648/10760      ( 98%) [2]
  CEF: paths                :      41984/42432      ( 98%) [8]
  CEF: prefix query msg chu :      20000/20056      ( 99%) [1]
  CEF: table                :       1672/1784       ( 93%) [2]
  CEF: table index          :      25200/25256      ( 99%) [1]
  CEF: table walks          :        488/600        ( 81%) [2]
  CEF: terminal fibs list   :        144/480        ( 30%) [6]
  CEF: up event chunk       :      20000/20056      ( 99%) [1]
  Memory                          in use/allocated         Count
  ------------------------------------------------------------------

  CEF: xdr msg chunks       :      32768/32824      ( 99%) [1]
  CEF:test fib entry sbs    :       3000/3056       ( 98%) [1]
  COLL: coll rec            :      72184/72296      ( 99%) [2]
  TAL: MTRIE n08            :     197640/197976     ( 99%) [6]
  TAL: control block        :         40/96         ( 41%) [1]
  TAL: item list elem       :      65536/65592      ( 99%) [1]
  TAL: mtrie control block  :       1656/1712       ( 96%) [1]
  TAL: rtree aux            :         56/112        ( 50%) [1]
  TAL: rtree control block  :         40/96         ( 41%) [1]
  TAL: rtree nodes          :      21760/21872      ( 99%) [2]
  TAL: tree control         :         80/192        ( 41%) [2]
  ------------------------------------------------------------------
  Totals                    :    1975744/1992712    ( 99%) [303]


Peter Kranz
Founder/CEO - Unwired Ltd
www.UnwiredLtd.com
Desk: 510-868-1614 x100
Mobile: 510-207-0000
pkranz at unwiredltd.com