[c-nsp] Cisco 2811 performance issue - dual(new) isp
Chuck Church
chuckchurch at gmail.com
Sat Dec 24 20:12:45 EST 2011
Just a thought, has the router ever been rebooted since adding the second
link? I've seen issues over the years where a whole bunch of changes over a
period of time has resulted in weird behavior. (Kind of like when you
delete a subinterface, it'll say it will remain until a reboot). That might
explain the NTP issue. Maybe even the performance issue.
Chuck
From: Jmail Clist [mailto:jmlist80 at gmail.com]
Sent: Saturday, December 24, 2011 5:10 PM
To: Chuck Church
Cc: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] Cisco 2811 performance issue - dual(new) isp
Yea, I'll give the upgrade ago. I gotta schedule it out. In the meantime,
I'm parshing through "debug ip packet" data to see what is being process
switched. I set the debug condition to interface fa0/1 before I started.
It looks like tons of stuff is being process switched for no apparent
reason. Also, I have the router making calls to the defined NTP server but
sourcing the addresses with the old isp interface's ip address of fa0/0)
but going out the new ISP connection (fa0/1) for ntp updates when clearly
the default 0.0.0.0 route is out the new isp connection (fa0/1). I don't
understand why he's sourcing it like that.
IP: s=orig_isp (local), d=129.7.1.66 (FastEthernet0/1
It also likes like main culprit on the new_ISP interface is "Post routing
NAT". I may not be looking at the data correctly but it seems like my NAT
traffic is not being switched in hardware and I'm not using any route-maps.
Just the standard overload statement.
sample debug ip packet outputt
4.249 (FastEthernet0/1), len 78, sending full packet
036360: Dec 24 16:30:29.120: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
52, stop process pak for forus packet
036361: Dec 24 16:30:29.120: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
52, enqueue feature, Firewall(3), rtype 0, forus FALSE,
sendself FALSE, mtu 0, fwdchk FALSE
036362: Dec 24 16:30:29.124: IP: s=172.18.1.1 (local), d=172.18.1.23, len
52, local feature, NAT(2), rtype 0, forus FALSE,
sendself FALSE, mtu 0, fwdchk FALSE
036363: Dec 24 16:30:29.304: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
40, stop process pak for forus packet
036364: Dec 24 16:30:29.304: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
40, enqueue feature, Firewall(3), rtype 0, forus FALSE,
sendself FALSE, mtu 0, fwdchk FALSE
036365: Dec 24 16:30:30.324: IP: s=192.168.1.30 (local), d=192.168.3.1, len
76, local feature, NAT(2), rtype 0, forus FALSE,
sendself FALSE, mtu 0, fwdchk FALSE
036366: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66, len 76,
local feature, NAT(2), rtype 0, forus FALSE, sendself
FALSE, mtu 0, fwdchk FALSE
036367: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66
(FastEthernet0/1), len 76, sending
036368: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66
(FastEthernet0/1), len 76, output feature, CCE Output
Classification(5), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len 40, enqueue feature,
Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
036704: Dec 24 16:30:50.904: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len
40, stop process pak for forus packet
036705: Dec 24 16:30:50.904: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len
40, enqueue feature, Firewall(3), rtype 0, forus
FALSE, sendself FALSE, mtu 0, fwdchk FALSE
036706: Dec 24 16:30:51.032: IP: s=192.168.1.80 (Vlan10), d=192.168.1.30,
len 28, stop process pak for forus packetpe 1, forus
FALSE, sendself FALSE, mtu 0, fwdchk FALSE
036372: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66
(FastEthernet0/1), len 76, output feature, Firewall (inspect)
(38), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
036373: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66
(FastEthernet0/1), len 76, output feature, Post-Ingress-NetFlow
(52), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
036374: Dec 24 16:30:30.328: IP: s=orig_isp (local), d=129.7.1.66
(FastEthernet0/1), len 76, sending full packet
036375: Dec 24 16:30:30.404: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30,
len 28, stop process pak for forus packet
036376: Dec 24 16:30:30.404: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30,
len 28, enqueue feature, Firewall(3), rtype 0, forus
FALSE, sendself FALSE, mtu 0, fwdchk FALSE
--More-- rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
036713: Dec 24 16:30:51.220: IP: s=192.168.1.30 (local), d=192.168.2.3, len
576, local feature, NAT(2), rtype 0, forus FALSE,
sendself FALSE, mtu 0, fwdchk FALSE
More..
041793: Dec 24 16:46:59.141: IP: s=172.18.1.1 (local), d=172.18.1.23, len
52, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
041794: Dec 24 16:46:59.233: IP: s=192.168.1.80 (Vlan10), d=192.168.1.30,
len 28, stop process pak for forus packet
041795: Dec 24 16:46:59.233: IP: s=192.168.1.80 (Vlan10), d=192.168.1.30,
len 28, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
mtu 0, fwdchk FALSE
041796: Dec 24 16:46:59.233: IP: s=192.168.1.30 (local), d=192.168.1.80, len
28, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
041797: Dec 24 16:46:59.321: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
40, stop process pak for forus packet
041798: Dec 24 16:46:59.321: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu
0, fwdchk FALSE
041799: Dec 24 16:46:59.893: IP: s=192.168.1.30 (local), d=192.168.1.13, len
56, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
041800: Dec 24 16:47:00.005: IP: s=192.168.1.146 (Vlan10), d=x-email-svc-x
(FastEthernet0/1), len 68, output feature, CCE Output Classification(5),
rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
041801: Dec 24 16:47:00.005: IP: s=192.168.1.30 (local), d=192.168.1.146,
len 56, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
042126: Dec 24 16:47:17.013: IP: s=192.168.1.30 (local), d=192.168.1.79, len
28, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
042127: Dec 24 16:47:17.017: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len
40, stop process pak for forus packet
042128: Dec 24 16:47:17.017: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len
40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu
0, fwdchk FALSE
042129: Dec 24 16:47:17.021: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len
40, stop process pak for forus packete 0, forus FALSE, sendself FALSE, mtu
0, fwdchk FALSE
041806: Dec 24 16:47:00.033: IP: s=172.18.1.147 (Vlan1), d=172.18.1.1, len
40, stop process pak for forus packet
041807: Dec 24 16:47:00.033: IP: s=172.18.1.147 (Vlan1), d=172.18.1.1, len
40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu
0, fwdchk FALSE
041808: Dec 24 16:47:00.845: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30,
len 28, stop process pak for forus packet
041809: Dec 24 16:47:00.845: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30,
len 28, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
mtu 0, fwdchk FALSE
041810: Dec 24 16:47:00.845: IP: s=192.168.1.30 (local), d=192.168.1.79, len
28, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
92.168.1.30 (local), d=192.168.2.3, len 576, local feature, NAT(2), rtype 0,
forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
042136: Dec 24 16:47:17.217: IP: s=192.168.1.30 (local), d=192.168.2.3, len
576, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
042137: Dec 24 16:47:17.221: IP: s=192.168.1.30 (local), d=192.168.2.3, len
44, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
042138: Dec 24 16:47:17.221: IP: s=192.168.1.30 (local), d=192.168.2.3, len
444, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
042139: Dec 24 16:47:17.225: IP: s=192.168.1.31 (Vlan10), d=192.168.1.30,
len 84, stop process pak for forus packet
042140: Dec 24 16:47:17.225: IP: s=192.168.1.31 (Vlan10), d=192.168.1.30,
len 84, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
mtu 0, fwdchk FALSE
042141: Dec 24 16:47:17.225: IP: s=192.168.1.30 (local), d=192.168.1.31, len
84, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
042142: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len
40, stop process pak for forus packet
042143: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len
40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu
0, fwdchk FALSE
042144: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len
40, stop process pak for forus packet
042145: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len
40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu
0, fwdchk FALSE
On Sat, Dec 24, 2011 at 9:25 AM, Chuck Church <chuckchurch at gmail.com> wrote:
Silly question maybe, but do you have any logging in your ACLs? If not,
that first bug sounds possible. I've got a 2821 running 12.4(25f), doing
NAT overload with heavy QOS and policy routing, get about 99% route-cache in
both directions. Which is similar to your config when inspection is off.
IOS issue seems plausible.
Chuck
From: Jmail Clist [mailto:jmlist80 at gmail.com]
Sent: Friday, December 23, 2011 4:41 PM
To: Reuben Farrelly
Cc: Chuck Church; cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] Cisco 2811 performance issue - dual(new) isp
After running for most of the days, things are back to getting mainly
process switched. ?? Strange.
rtr2811#sh int fa0/1 stats
FastEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 3366529 213364344 66121 21868973
Route cache 57045 40344237 50866 11970836
Total 3423574 253708581 116987 33839809
On Fri, Dec 23, 2011 at 9:45 AM, Jmail Clist <jmlist80 at gmail.com> wrote:
That cef command was pretty useful. Before you scroll down to the
output/stats, here are the only two
bugs that look like they might be related to my issue. With test #1,
(everything disabled), it was ALL
process switched. Test #2 looks slightly better with only IP
virtual-reassembly enabled. Something is
going on here and I'm more puzzled than ever. Test #3 caused lots of process
switching when doing the speed tests(???). Test #4 is even more surprising
because things seem better under "normal" traffic loads. Thoughts?
I'd like to find a FTP server to test against instead of using speedguide,
speakeasy, etc.
CSCsa67785 Bug Details
crypto-map/NAT/IPS wont work properly in CEF path
Symptoms: Packets may be dropped on the interface when NAT/IPSEC/IPS is
configured on the same interface.
Conditions: If IPSec/NAT and CBAC or IPS/IDS is configured on the same
interface and the packet gets punted by any of the features, then the packet
may be dropped.
Workaround: Remove from the configuration the feature which punts the packet
to process path.
CSCtd25213 Bug Details
NAT not working for locally generated packets
Symptoms: NAT is not working for locally-generated packets.
Conditions: This symptom is observed when NAT is configured for inside and
outside addresses, and when a self-generated packet is sent to OL.
Workaround: Instead of using dynamic NAT, use static NAT for self-generated
packets.
1) disabled cbac/acl and ip virtual-reassembly
interface FastEthernet0/1
ip address x.x.x.x 255.255.255.0
no ip redirects
ip nat outside
no ip virtual-reassembly
duplex auto
speed auto
end
rtr2811#sh int fa0/1 stats
FastEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 12212 757602 133 16723
Route cache 173 20535 270 35125
Total 12385 778137 403 51848
rtr2811#sh ip cef switching statistics feature
IPv4 CEF input features:
Feature Drop Consume Punt Punt2Host Gave
route
NAT Outside 0 0 0 25
0
Total 0 0 0 25
0
IPv4 CEF output features:
Feature Drop Consume Punt Punt2Host New
i/f
Post-routing NAT 0 0 0 68
0
Total 0 0 0 68
0
IPv4 CEF post-encap features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF for us features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF punt features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF local features:
Feature Drop Consume Punt Punt2Host Gave
route
Total 0 0 0 0
0
rtr2811#
2) enabled ip virtual-reassembly ONLY
interface FastEthernet0/1
ip address x.x.x.x 255.255.255.0
no ip redirects
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
end
rtr2811#sh int fa0/1 stats
FastEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 1277 78657 16 1589
Route cache 14 3851 32 4087
Total 1291 82508 48 5676
rtr2811#sh ip cef switching statistics feature
IPv4 CEF input features:
Feature Drop Consume Punt Punt2Host Gave
route
NAT Outside 0 0 0 1
0
Total 0 0 0 1
0
IPv4 CEF output features:
Feature Drop Consume Punt Punt2Host New
i/f
Post-routing NAT 0 0 0 12
0
Total 0 0 0 12
0
IPv4 CEF post-encap features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF for us features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF punt features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF local features:
Feature Drop Consume Punt Punt2Host Gave
route
Total 0 0 0 0
0
rtr2811#
NOTE: After this I enabled CBAC-int & Ext_ACL-inbound again. Performance was
almost good as #2 still. I
also cleared counters once more and waited 10 minutes. Here are the results
again. Any ideas????
3) I ran a speedtest on www.speakeasy.net <http://www.speakeasy.net/> and
process switching went through the roo
rtr2811#sh int fa0/1 stats
FastEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 17858 1157573 467 143934
Route cache 1072 964530 837 98966
Total 18930 2122103 1304 242900
rtr2811#
rtr2811#running speedtest now
^
% Invalid input detected at '^' marker.
rtr2811#sh int fa0/1 stats
FastEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 21414 1379133 507 159277
Route cache 10317 10944391 8426 7415536
Total 31731 12323524 8933 7574813
rtr2811#sh int fa0/1 stats
FastEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 21490 1384753 513 162841
Route cache 10322 10946281 8426 7415536
Total 31812 12331034 8939 7578377
rtr2811#
4) cleared counters one last time and let it from midnight to 9:39am
rtr2811#sh int fa0/1 stats
FastEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 2091010 132620733 42136 13987400
Route cache 42156 32749186 36559 10473996
Total 2133166 165369919 78695 24461396
rtr2811#sh ip cef switching statistics feature
IPv4 CEF input features:
Feature Drop Consume Punt Punt2Host Gave
route
Access List 11840 0 0 13286
0
NAT Outside 0 0 0 3389
0
Total 11840 0 0 16675
0
IPv4 CEF output features:
Feature Drop Consume Punt Punt2Host New
i/f
Post-routing NAT 0 0 0 28310
0
Firewall (inspec 57 0 0 13
0
Total 57 0 0 28323
0
IPv4 CEF post-encap features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF for us features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF punt features:
Feature Drop Consume Punt Punt2Host New
i/f
Total 0 0 0 0
0
IPv4 CEF local features:
Feature Drop Consume Punt Punt2Host Gave
route
Total 0 0 0 0
0
rtr2811#
On Thu, Dec 22, 2011 at 4:24 PM, Reuben Farrelly <reuben-cisco-nsp at reub.net>
wrote:
The command:
router#show ip cef switching statistics feature
Will show you which feature is causing traffic to be punted to CPU.
Reuben
On 23/12/2011 7:42 AM, Chuck Church wrote:
You're on the right path. The more important number is the packets in/out,
as opposed to the characters. Look at the ratio of packets in/out for
processor vs. Route-cache for the two interfaces. Fa0/1 is process
switching about 80% of them inbound. That's pretty bad. The output looks
better. Compare that to VLAN 10, where in both directions, only about 10%
are process switched. The stats for the switchports are meaningless, so you
can ignore those as the switch ASICs deal with those, until they hit the
VLAN int. Figure out what feature (or IOS bug??) is causing so much process
switching, and I think it'll get better.
More information about the cisco-nsp
mailing list