[c-nsp] Cisco 2811 performance issue - dual(new) isp
Jmail Clist
jmlist80 at gmail.com
Sat Dec 24 17:09:46 EST 2011
Yea, I'll give the upgrade ago. I gotta schedule it out. In the meantime,
I'm parshing through "debug ip packet" data to see what is being process
switched. I set the debug condition to interface fa0/1 before I started.
It looks like tons of stuff is being process switched for no apparent
reason. Also, I have the router making calls to the defined NTP server but
sourcing the addresses with the old isp interface's ip address of fa0/0)
but going out the new ISP connection (fa0/1) for ntp updates when clearly
the default 0.0.0.0 route is out the new isp connection (fa0/1). I don't
understand why he's sourcing it like that.
IP: s=orig_isp (local), d=129.7.1.66 (FastEthernet0/1
It also likes like main culprit on the new_ISP interface is "Post routing
NAT". I may not be looking at the data correctly but it seems like my NAT
traffic is not being switched in hardware and I'm not using any route-maps.
Just the standard overload statement.
sample debug ip packet outputt
4.249 (FastEthernet0/1), len 78, sending full packet
036360: Dec 24 16:30:29.120: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
52, stop process pak for forus packet
036361: Dec 24 16:30:29.120: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
52, enqueue feature, Firewall(3), rtype 0, forus FALSE,
sendself FALSE, mtu 0, fwdchk FALSE
036362: Dec 24 16:30:29.124: IP: s=172.18.1.1 (local), d=172.18.1.23, len
52, local feature, NAT(2), rtype 0, forus FALSE,
sendself FALSE, mtu 0, fwdchk FALSE
036363: Dec 24 16:30:29.304: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
40, stop process pak for forus packet
036364: Dec 24 16:30:29.304: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
40, enqueue feature, Firewall(3), rtype 0, forus FALSE,
sendself FALSE, mtu 0, fwdchk FALSE
036365: Dec 24 16:30:30.324: IP: s=192.168.1.30 (local), d=192.168.3.1, len
76, local feature, NAT(2), rtype 0, forus FALSE,
sendself FALSE, mtu 0, fwdchk FALSE
036366: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66, len 76,
local feature, NAT(2), rtype 0, forus FALSE, sendself
FALSE, mtu 0, fwdchk FALSE
036367: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66
(FastEthernet0/1), len 76, sending
036368: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66
(FastEthernet0/1), len 76, output feature, CCE Output
Classification(5), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len 40, enqueue feature,
Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
036704: Dec 24 16:30:50.904: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
len 40, stop process pak for forus packet
036705: Dec 24 16:30:50.904: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
len 40, enqueue feature, Firewall(3), rtype 0, forus
FALSE, sendself FALSE, mtu 0, fwdchk FALSE
036706: Dec 24 16:30:51.032: IP: s=192.168.1.80 (Vlan10), d=192.168.1.30,
len 28, stop process pak for forus packetpe 1, forus
FALSE, sendself FALSE, mtu 0, fwdchk FALSE
036372: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66
(FastEthernet0/1), len 76, output feature, Firewall (inspect)
(38), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
036373: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66
(FastEthernet0/1), len 76, output feature, Post-Ingress-NetFlow
(52), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
036374: Dec 24 16:30:30.328: IP: s=orig_isp (local), d=129.7.1.66
(FastEthernet0/1), len 76, sending full packet
036375: Dec 24 16:30:30.404: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30,
len 28, stop process pak for forus packet
036376: Dec 24 16:30:30.404: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30,
len 28, enqueue feature, Firewall(3), rtype 0, forus
FALSE, sendself FALSE, mtu 0, fwdchk FALSE
--More-- rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
036713: Dec 24 16:30:51.220: IP: s=192.168.1.30 (local), d=192.168.2.3, len
576, local feature, NAT(2), rtype 0, forus FALSE,
sendself FALSE, mtu 0, fwdchk FALSE
More..
041793: Dec 24 16:46:59.141: IP: s=172.18.1.1 (local), d=172.18.1.23, len
52, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
041794: Dec 24 16:46:59.233: IP: s=192.168.1.80 (Vlan10), d=192.168.1.30,
len 28, stop process pak for forus packet
041795: Dec 24 16:46:59.233: IP: s=192.168.1.80 (Vlan10), d=192.168.1.30,
len 28, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
mtu 0, fwdchk FALSE
041796: Dec 24 16:46:59.233: IP: s=192.168.1.30 (local), d=192.168.1.80,
len 28, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
041797: Dec 24 16:46:59.321: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
40, stop process pak for forus packet
041798: Dec 24 16:46:59.321: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu
0, fwdchk FALSE
041799: Dec 24 16:46:59.893: IP: s=192.168.1.30 (local), d=192.168.1.13,
len 56, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
041800: Dec 24 16:47:00.005: IP: s=192.168.1.146 (Vlan10), d=x-email-svc-x
(FastEthernet0/1), len 68, output feature, CCE Output Classification(5),
rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
041801: Dec 24 16:47:00.005: IP: s=192.168.1.30 (local), d=192.168.1.146,
len 56, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
042126: Dec 24 16:47:17.013: IP: s=192.168.1.30 (local), d=192.168.1.79,
len 28, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
042127: Dec 24 16:47:17.017: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
len 40, stop process pak for forus packet
042128: Dec 24 16:47:17.017: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
len 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
mtu 0, fwdchk FALSE
042129: Dec 24 16:47:17.021: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
len 40, stop process pak for forus packete 0, forus FALSE, sendself FALSE,
mtu 0, fwdchk FALSE
041806: Dec 24 16:47:00.033: IP: s=172.18.1.147 (Vlan1), d=172.18.1.1, len
40, stop process pak for forus packet
041807: Dec 24 16:47:00.033: IP: s=172.18.1.147 (Vlan1), d=172.18.1.1, len
40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu
0, fwdchk FALSE
041808: Dec 24 16:47:00.845: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30,
len 28, stop process pak for forus packet
041809: Dec 24 16:47:00.845: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30,
len 28, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
mtu 0, fwdchk FALSE
041810: Dec 24 16:47:00.845: IP: s=192.168.1.30 (local), d=192.168.1.79,
len 28, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
92.168.1.30 (local), d=192.168.2.3, len 576, local feature, NAT(2), rtype
0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
042136: Dec 24 16:47:17.217: IP: s=192.168.1.30 (local), d=192.168.2.3, len
576, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
042137: Dec 24 16:47:17.221: IP: s=192.168.1.30 (local), d=192.168.2.3, len
44, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
042138: Dec 24 16:47:17.221: IP: s=192.168.1.30 (local), d=192.168.2.3, len
444, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
042139: Dec 24 16:47:17.225: IP: s=192.168.1.31 (Vlan10), d=192.168.1.30,
len 84, stop process pak for forus packet
042140: Dec 24 16:47:17.225: IP: s=192.168.1.31 (Vlan10), d=192.168.1.30,
len 84, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
mtu 0, fwdchk FALSE
042141: Dec 24 16:47:17.225: IP: s=192.168.1.30 (local), d=192.168.1.31,
len 84, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
fwdchk FALSE
042142: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
len 40, stop process pak for forus packet
042143: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
len 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
mtu 0, fwdchk FALSE
042144: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
len 40, stop process pak for forus packet
042145: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
len 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
mtu 0, fwdchk FALSE
On Sat, Dec 24, 2011 at 9:25 AM, Chuck Church <chuckchurch at gmail.com> wrote:
> Silly question maybe, but do you have any logging in your ACLs? If not,
> that first bug sounds possible. I’ve got a 2821 running 12.4(25f), doing
> NAT overload with heavy QOS and policy routing, get about 99% route-cache
> in both directions. Which is similar to your config when inspection is
> off. IOS issue seems plausible.****
>
> ** **
>
> Chuck****
>
> ** **
>
> *From:* Jmail Clist [mailto:jmlist80 at gmail.com]
> *Sent:* Friday, December 23, 2011 4:41 PM
> *To:* Reuben Farrelly
> *Cc:* Chuck Church; cisco-nsp at puck.nether.net
>
> *Subject:* Re: [c-nsp] Cisco 2811 performance issue - dual(new) isp****
>
> ** **
>
> After running for most of the days, things are back to getting mainly
> process switched. ?? Strange.****
>
> ****
>
> rtr2811#sh int fa0/1 stats
> FastEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 3366529 213364344 66121 21868973
> Route cache 57045 40344237 50866 11970836
> Total 3423574 253708581 116987 33839809
>
> ****
>
> On Fri, Dec 23, 2011 at 9:45 AM, Jmail Clist <jmlist80 at gmail.com> wrote:**
> **
>
>
>
> That cef command was pretty useful. Before you scroll down to the
> output/stats, here are the only two ****
>
> bugs that look like they might be related to my issue. With test #1,
> (everything disabled), it was ALL ****
>
> process switched. Test #2 looks slightly better with only IP
> virtual-reassembly enabled. Something is ****
>
> going on here and I'm more puzzled than ever. Test #3 caused lots of
> process switching when doing the speed tests(???). Test #4 is even more
> surprising because things seem better under "normal" traffic loads.
> Thoughts?****
>
> ****
>
> I'd like to find a FTP server to test against instead of using speedguide,
> speakeasy, etc.****
>
>
> CSCsa67785 Bug Details
> crypto-map/NAT/IPS wont work properly in CEF path
> Symptoms: Packets may be dropped on the interface when NAT/IPSEC/IPS is
> configured on the same interface.
> Conditions: If IPSec/NAT and CBAC or IPS/IDS is configured on the same
> interface and the packet gets punted by any of the features, then the
> packet
> may be dropped.
> Workaround: Remove from the configuration the feature which punts the
> packet
> to process path.****
>
> CSCtd25213 Bug Details
> NAT not working for locally generated packets
> Symptoms: NAT is not working for locally-generated packets.
> Conditions: This symptom is observed when NAT is configured for inside and
> outside addresses, and when a self-generated packet is sent to OL.
> Workaround: Instead of using dynamic NAT, use static NAT for
> self-generated
> packets. ****
>
>
> 1) disabled cbac/acl and ip virtual-reassembly****
>
> interface FastEthernet0/1
> ip address x.x.x.x 255.255.255.0****
>
> no ip redirects
> ip nat outside****
>
> no ip virtual-reassembly
> duplex auto
> speed auto
> end ****
>
> rtr2811#sh int fa0/1 stats
> FastEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out****
>
> Processor 12212 757602 133 16723
> Route cache 173 20535 270 35125
> Total 12385 778137 403 51848
> rtr2811#sh ip cef switching statistics feature
> IPv4 CEF input features:
> Feature Drop Consume Punt Punt2Host Gave
> route
> NAT Outside 0 0 0
> 25 0
> Total 0 0 0
> 25 0 ****
>
> IPv4 CEF output features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Post-routing NAT 0 0 0
> 68 0
> Total 0 0 0
> 68 0****
>
> IPv4 CEF post-encap features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF for us features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF punt features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF local features:
> Feature Drop Consume Punt Punt2Host Gave
> route
> Total 0 0 0
> 0 0
> rtr2811#****
>
>
> 2) enabled ip virtual-reassembly ONLY ****
>
>
> interface FastEthernet0/1
> ip address x.x.x.x 255.255.255.0****
>
> no ip redirects
> ip nat outside****
>
> ip virtual-reassembly
> duplex auto
> speed auto****
>
> end ****
>
> rtr2811#sh int fa0/1 stats
> FastEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out****
>
> Processor 1277 78657 16 1589
> Route cache 14 3851 32 4087
> Total 1291 82508 48 5676
> rtr2811#sh ip cef switching statistics feature
> IPv4 CEF input features:
> Feature Drop Consume Punt Punt2Host Gave
> route
> NAT Outside 0 0 0
> 1 0
> Total 0 0 0
> 1 0 ****
>
> IPv4 CEF output features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Post-routing NAT 0 0 0
> 12 0
> Total 0 0 0
> 12 0****
>
> IPv4 CEF post-encap features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF for us features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF punt features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF local features:
> Feature Drop Consume Punt Punt2Host Gave
> route
> Total 0 0 0
> 0 0
> rtr2811#****
>
>
> NOTE: After this I enabled CBAC-int & Ext_ACL-inbound again. Performance
> was almost good as #2 still. I ****
>
> also cleared counters once more and waited 10 minutes. Here are the
> results again. Any ideas????****
>
>
> 3) I ran a speedtest on www.speakeasy.net and process switching went
> through the roo****
>
> rtr2811#sh int fa0/1 stats
> FastEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out****
>
> Processor 17858 1157573 467 143934
> Route cache 1072 964530 837 98966
> Total 18930 2122103 1304 242900
> rtr2811#
> rtr2811#running speedtest now
> ^
> % Invalid input detected at '^' marker. ****
>
> rtr2811#sh int fa0/1 stats
> FastEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out****
>
> Processor 21414 1379133 507 159277
> Route cache 10317 10944391 8426 7415536
> Total 31731 12323524 8933 7574813 ****
>
>
> rtr2811#sh int fa0/1 stats
> FastEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out****
>
> Processor 21490 1384753 513 162841
> Route cache 10322 10946281 8426 7415536
> Total 31812 12331034 8939 7578377
> rtr2811# ****
>
> 4) cleared counters one last time and let it from midnight to 9:39am****
>
> rtr2811#sh int fa0/1 stats
> FastEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out****
>
> Processor 2091010 132620733 42136 13987400
> Route cache 42156 32749186 36559 10473996
> Total 2133166 165369919 78695 24461396
> rtr2811#sh ip cef switching statistics feature
> IPv4 CEF input features:
> Feature Drop Consume Punt Punt2Host Gave
> route
> Access List 11840 0 0
> 13286 0
> NAT Outside 0 0 0
> 3389 0
> Total 11840 0 0
> 16675 0 ****
>
> IPv4 CEF output features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Post-routing NAT 0 0 0
> 28310 0
> Firewall (inspec 57 0 0
> 13 0
> Total 57 0 0
> 28323 0****
>
> IPv4 CEF post-encap features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF for us features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF punt features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF local features:
> Feature Drop Consume Punt Punt2Host Gave
> route
> Total 0 0 0
> 0 0
> rtr2811#****
>
> On Thu, Dec 22, 2011 at 4:24 PM, Reuben Farrelly <
> reuben-cisco-nsp at reub.net> wrote:****
>
> The command:
>
> router#show ip cef switching statistics feature
>
> Will show you which feature is causing traffic to be punted to CPU.
>
> Reuben ****
>
>
>
>
> On 23/12/2011 7:42 AM, Chuck Church wrote:****
>
> You're on the right path. The more important number is the packets in/out,
> as opposed to the characters. Look at the ratio of packets in/out for
> processor vs. Route-cache for the two interfaces. Fa0/1 is process
> switching about 80% of them inbound. That's pretty bad. The output
> looks
> better. Compare that to VLAN 10, where in both directions, only about 10%
> are process switched. The stats for the switchports are meaningless, so
> you
> can ignore those as the switch ASICs deal with those, until they hit the
> VLAN int. Figure out what feature (or IOS bug??) is causing so much
> process
> switching, and I think it'll get better.****
>
> ** **
>
> ** **
>
More information about the cisco-nsp
mailing list