[c-nsp] Cisco 2811 performance issue - dual(new) isp
Chuck Church
chuckchurch at gmail.com
Sat Dec 24 10:25:54 EST 2011
Silly question maybe, but do you have any logging in your ACLs? If not,
that first bug sounds possible. I've got a 2821 running 12.4(25f), doing
NAT overload with heavy QOS and policy routing, get about 99% route-cache in
both directions. Which is similar to your config when inspection is off.
IOS issue seems plausible.
Chuck
From: Jmail Clist [mailto:jmlist80 at gmail.com]
Sent: Friday, December 23, 2011 4:41 PM
To: Reuben Farrelly
Cc: Chuck Church; cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] Cisco 2811 performance issue - dual(new) isp
After running for most of the days, things are back to getting mainly
process switched. ?? Strange.
rtr2811#sh int fa0/1 stats
FastEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 3366529 213364344 66121 21868973
Route cache 57045 40344237 50866 11970836
Total 3423574 253708581 116987 33839809
On Fri, Dec 23, 2011 at 9:45 AM, Jmail Clist <jmlist80 at gmail.com> wrote:
That cef command was pretty useful. Before you scroll down to the
output/stats, here are the only two
bugs that look like they might be related to my issue. With test #1,
(everything disabled), it was ALL
process switched. Test #2 looks slightly better with only IP
virtual-reassembly enabled. Something is
going on here and I'm more puzzled than ever. Test #3 caused lots of process
switching when doing the speed tests(???). Test #4 is even more surprising
because things seem better under "normal" traffic loads. Thoughts?
I'd like to find a FTP server to test against instead of using speedguide,
speakeasy, etc.
CSCsa67785 Bug Details
crypto-map/NAT/IPS wont work properly in CEF path
Symptoms: Packets may be dropped on the interface when NAT/IPSEC/IPS is
configured on the same interface.
Conditions: If IPSec/NAT and CBAC or IPS/IDS is configured on the same
interface and the packet gets punted by any of the features, then the packet
may be dropped.
Workaround: Remove from the configuration the feature which punts the packet
to process path.
CSCtd25213 Bug Details
NAT not working for locally generated packets
Symptoms: NAT is not working for locally-generated packets.
Conditions: This symptom is observed when NAT is configured for inside and
outside addresses, and when a self-generated packet is sent to OL.
Workaround: Instead of using dynamic NAT, use static NAT for self-generated
packets.
1) disabled cbac/acl and ip virtual-reassembly
interface FastEthernet0/1
ip address x.x.x.x 255.255.255.0
no ip redirects
ip nat outside
no ip virtual-reassembly
duplex auto
speed auto
end
rtr2811#sh int fa0/1 stats
FastEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 12212 757602 133 16723
Route cache 173 20535 270 35125
Total 12385 778137 403 51848
rtr2811#sh ip cef switching statistics feature
IPv4 CEF input features:
Feature Drop Consume Punt Punt2Host Gave
route
NAT Outside 0 0 0 25
0
Total 0 0 0 25
0
IPv4 CEF output features:
Feature Drop Consume Punt Punt2Host New
i/f
Post-routing NAT 0 0 0 68
0
Total 0 0 0 68
0
IPv4 CEF post-encap features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF for us features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF punt features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF local features:
Feature Drop Consume Punt Punt2Host Gave
route
Total 0 0 0 0
0
rtr2811#
2) enabled ip virtual-reassembly ONLY
interface FastEthernet0/1
ip address x.x.x.x 255.255.255.0
no ip redirects
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
end
rtr2811#sh int fa0/1 stats
FastEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 1277 78657 16 1589
Route cache 14 3851 32 4087
Total 1291 82508 48 5676
rtr2811#sh ip cef switching statistics feature
IPv4 CEF input features:
Feature Drop Consume Punt Punt2Host Gave
route
NAT Outside 0 0 0 1
0
Total 0 0 0 1
0
IPv4 CEF output features:
Feature Drop Consume Punt Punt2Host New
i/f
Post-routing NAT 0 0 0 12
0
Total 0 0 0 12
0
IPv4 CEF post-encap features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF for us features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF punt features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF local features:
Feature Drop Consume Punt Punt2Host Gave
route
Total 0 0 0 0
0
rtr2811#
NOTE: After this I enabled CBAC-int & Ext_ACL-inbound again. Performance was
almost good as #2 still. I
also cleared counters once more and waited 10 minutes. Here are the results
again. Any ideas????
3) I ran a speedtest on www.speakeasy.net <http://www.speakeasy.net/> and
process switching went through the roo
rtr2811#sh int fa0/1 stats
FastEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 17858 1157573 467 143934
Route cache 1072 964530 837 98966
Total 18930 2122103 1304 242900
rtr2811#
rtr2811#running speedtest now
^
% Invalid input detected at '^' marker.
rtr2811#sh int fa0/1 stats
FastEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 21414 1379133 507 159277
Route cache 10317 10944391 8426 7415536
Total 31731 12323524 8933 7574813
rtr2811#sh int fa0/1 stats
FastEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 21490 1384753 513 162841
Route cache 10322 10946281 8426 7415536
Total 31812 12331034 8939 7578377
rtr2811#
4) cleared counters one last time and let it from midnight to 9:39am
rtr2811#sh int fa0/1 stats
FastEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 2091010 132620733 42136 13987400
Route cache 42156 32749186 36559 10473996
Total 2133166 165369919 78695 24461396
rtr2811#sh ip cef switching statistics feature
IPv4 CEF input features:
Feature Drop Consume Punt Punt2Host Gave
route
Access List 11840 0 0 13286
0
NAT Outside 0 0 0 3389
0
Total 11840 0 0 16675
0
IPv4 CEF output features:
Feature Drop Consume Punt Punt2Host New
i/f
Post-routing NAT 0 0 0 28310
0
Firewall (inspec 57 0 0 13
0
Total 57 0 0 28323
0
IPv4 CEF post-encap features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF for us features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF punt features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF local features:
Feature Drop Consume Punt Punt2Host Gave
route
Total 0 0 0 0
0
rtr2811#
On Thu, Dec 22, 2011 at 4:24 PM, Reuben Farrelly <reuben-cisco-nsp at reub.net>
wrote:
The command:
router#show ip cef switching statistics feature
Will show you which feature is causing traffic to be punted to CPU.
Reuben
On 23/12/2011 7:42 AM, Chuck Church wrote:
You're on the right path. The more important number is the packets in/out,
as opposed to the characters. Look at the ratio of packets in/out for
processor vs. Route-cache for the two interfaces. Fa0/1 is process
switching about 80% of them inbound. That's pretty bad. The output looks
better. Compare that to VLAN 10, where in both directions, only about 10%
are process switched. The stats for the switchports are meaningless, so you
can ignore those as the switch ASICs deal with those, until they hit the
VLAN int. Figure out what feature (or IOS bug??) is causing so much process
switching, and I think it'll get better.
More information about the cisco-nsp
mailing list