[c-nsp] Cisco 2811 performance issue - dual(new) isp
Jmail Clist
jmlist80 at gmail.com
Sat Dec 24 22:51:18 EST 2011
Actually that is at the top of my list. I'm hoping that is going to
clear up a lot of things. I've seen a lot of things as well just act screwy
over time until a reboot is performed. I just need to get permission to do
so. I'll keep you informed. And thanks again for all the help. This has
been a weird one.
On Sat, Dec 24, 2011 at 7:12 PM, Chuck Church <chuckchurch at gmail.com> wrote:
> Just a thought, has the router ever been rebooted since adding the
> second link? I’ve seen issues over the years where a whole bunch of
> changes over a period of time has resulted in weird behavior. (Kind of
> like when you delete a subinterface, it’ll say it will remain until a
> reboot). That might explain the NTP issue. Maybe even the performance
> issue.****
>
> ** **
>
> Chuck****
>
> ** **
>
> *From:* Jmail Clist [mailto:jmlist80 at gmail.com]
> *Sent:* Saturday, December 24, 2011 5:10 PM
> *To:* Chuck Church
> *Cc:* cisco-nsp at puck.nether.net
>
> *Subject:* Re: [c-nsp] Cisco 2811 performance issue - dual(new) isp****
>
> ** **
>
> Yea, I'll give the upgrade ago. I gotta schedule it out. In the meantime,
> I'm parshing through "debug ip packet" data to see what is being process
> switched. I set the debug condition to interface fa0/1 before I started.
> It looks like tons of stuff is being process switched for no apparent
> reason. Also, I have the router making calls to the defined NTP server but
> sourcing the addresses with the old isp interface's ip address of fa0/0)
> but going out the new ISP connection (fa0/1) for ntp updates when clearly
> the default 0.0.0.0 route is out the new isp connection (fa0/1). I don't
> understand why he's sourcing it like that. ****
>
> IP: s=orig_isp (local), d=129.7.1.66 (FastEthernet0/1****
>
> It also likes like main culprit on the new_ISP interface is "Post routing
> NAT". I may not be looking at the data correctly but it seems like my NAT
> traffic is not being switched in hardware and I'm not using any route-maps.
> Just the standard overload statement.****
>
> ****
>
> sample debug ip packet outputt****
>
> 4.249 (FastEthernet0/1), len 78, sending full packet
> 036360: Dec 24 16:30:29.120: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
> 52, stop process pak for forus packet
> 036361: Dec 24 16:30:29.120: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
> 52, enqueue feature, Firewall(3), rtype 0, forus FALSE, ****
>
> sendself FALSE, mtu 0, fwdchk FALSE
> 036362: Dec 24 16:30:29.124: IP: s=172.18.1.1 (local), d=172.18.1.23, len
> 52, local feature, NAT(2), rtype 0, forus FALSE, ****
>
> sendself FALSE, mtu 0, fwdchk FALSE
> 036363: Dec 24 16:30:29.304: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
> 40, stop process pak for forus packet
> 036364: Dec 24 16:30:29.304: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
> 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, ****
>
> sendself FALSE, mtu 0, fwdchk FALSE
> 036365: Dec 24 16:30:30.324: IP: s=192.168.1.30 (local), d=192.168.3.1,
> len 76, local feature, NAT(2), rtype 0, forus FALSE, ****
>
> sendself FALSE, mtu 0, fwdchk FALSE
> 036366: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66, len 76,
> local feature, NAT(2), rtype 0, forus FALSE, sendself ****
>
> FALSE, mtu 0, fwdchk FALSE
> 036367: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66
> (FastEthernet0/1), len 76, sending
> 036368: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66
> (FastEthernet0/1), len 76, output feature, CCE Output ****
>
> Classification(5), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk
> FALSE
> IP: s=192.168.2.3 (Vlan10), d=192.168.1.30, len 40, enqueue feature,
> Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, ****
>
> fwdchk FALSE
> 036704: Dec 24 16:30:50.904: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
> len 40, stop process pak for forus packet
> 036705: Dec 24 16:30:50.904: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
> len 40, enqueue feature, Firewall(3), rtype 0, forus ****
>
> FALSE, sendself FALSE, mtu 0, fwdchk FALSE
> 036706: Dec 24 16:30:51.032: IP: s=192.168.1.80 (Vlan10), d=192.168.1.30,
> len 28, stop process pak for forus packetpe 1, forus ****
>
> FALSE, sendself FALSE, mtu 0, fwdchk FALSE
> 036372: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66
> (FastEthernet0/1), len 76, output feature, Firewall (inspect)****
>
> (38), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
> 036373: Dec 24 16:30:30.324: IP: s=orig_isp (local), d=129.7.1.66
> (FastEthernet0/1), len 76, output feature, Post-Ingress-NetFlow****
>
> (52), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
> 036374: Dec 24 16:30:30.328: IP: s=orig_isp (local), d=129.7.1.66
> (FastEthernet0/1), len 76, sending full packet
> 036375: Dec 24 16:30:30.404: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30,
> len 28, stop process pak for forus packet
> 036376: Dec 24 16:30:30.404: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30,
> len 28, enqueue feature, Firewall(3), rtype 0, forus ****
>
> FALSE, sendself FALSE, mtu 0, fwdchk FALSE
> --More-- rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk
> FALSE
> 036713: Dec 24 16:30:51.220: IP: s=192.168.1.30 (local), d=192.168.2.3,
> len 576, local feature, NAT(2), rtype 0, forus FALSE, ****
>
> sendself FALSE, mtu 0, fwdchk FALSE****
>
> ****
>
> More..****
>
> ****
>
> 041793: Dec 24 16:46:59.141: IP: s=172.18.1.1 (local), d=172.18.1.23, len
> 52, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
> fwdchk FALSE
> 041794: Dec 24 16:46:59.233: IP: s=192.168.1.80 (Vlan10), d=192.168.1.30,
> len 28, stop process pak for forus packet
> 041795: Dec 24 16:46:59.233: IP: s=192.168.1.80 (Vlan10), d=192.168.1.30,
> len 28, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
> mtu 0, fwdchk FALSE
> 041796: Dec 24 16:46:59.233: IP: s=192.168.1.30 (local), d=192.168.1.80,
> len 28, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
> fwdchk FALSE
> 041797: Dec 24 16:46:59.321: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
> 40, stop process pak for forus packet
> 041798: Dec 24 16:46:59.321: IP: s=172.18.1.23 (Vlan1), d=172.18.1.1, len
> 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu
> 0, fwdchk FALSE
> 041799: Dec 24 16:46:59.893: IP: s=192.168.1.30 (local), d=192.168.1.13,
> len 56, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
> fwdchk FALSE
> 041800: Dec 24 16:47:00.005: IP: s=192.168.1.146 (Vlan10), d=x-email-svc-x
> (FastEthernet0/1), len 68, output feature, CCE Output Classification(5),
> rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
> 041801: Dec 24 16:47:00.005: IP: s=192.168.1.30 (local), d=192.168.1.146,
> len 56, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
> fwdchk FALSE
> 042126: Dec 24 16:47:17.013: IP: s=192.168.1.30 (local), d=192.168.1.79,
> len 28, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
> fwdchk FALSE
> 042127: Dec 24 16:47:17.017: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
> len 40, stop process pak for forus packet
> 042128: Dec 24 16:47:17.017: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
> len 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
> mtu 0, fwdchk FALSE
> 042129: Dec 24 16:47:17.021: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
> len 40, stop process pak for forus packete 0, forus FALSE, sendself FALSE,
> mtu 0, fwdchk FALSE
> 041806: Dec 24 16:47:00.033: IP: s=172.18.1.147 (Vlan1), d=172.18.1.1, len
> 40, stop process pak for forus packet
> 041807: Dec 24 16:47:00.033: IP: s=172.18.1.147 (Vlan1), d=172.18.1.1, len
> 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE, mtu
> 0, fwdchk FALSE
> 041808: Dec 24 16:47:00.845: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30,
> len 28, stop process pak for forus packet
> 041809: Dec 24 16:47:00.845: IP: s=192.168.1.79 (Vlan10), d=192.168.1.30,
> len 28, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
> mtu 0, fwdchk FALSE
> 041810: Dec 24 16:47:00.845: IP: s=192.168.1.30 (local), d=192.168.1.79,
> len 28, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
> fwdchk FALSE
> 92.168.1.30 (local), d=192.168.2.3, len 576, local feature, NAT(2), rtype
> 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
> 042136: Dec 24 16:47:17.217: IP: s=192.168.1.30 (local), d=192.168.2.3,
> len 576, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu
> 0, fwdchk FALSE
> 042137: Dec 24 16:47:17.221: IP: s=192.168.1.30 (local), d=192.168.2.3,
> len 44, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
> fwdchk FALSE
> 042138: Dec 24 16:47:17.221: IP: s=192.168.1.30 (local), d=192.168.2.3,
> len 444, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu
> 0, fwdchk FALSE
> 042139: Dec 24 16:47:17.225: IP: s=192.168.1.31 (Vlan10), d=192.168.1.30,
> len 84, stop process pak for forus packet
> 042140: Dec 24 16:47:17.225: IP: s=192.168.1.31 (Vlan10), d=192.168.1.30,
> len 84, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
> mtu 0, fwdchk FALSE
> 042141: Dec 24 16:47:17.225: IP: s=192.168.1.30 (local), d=192.168.1.31,
> len 84, local feature, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0,
> fwdchk FALSE
> 042142: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
> len 40, stop process pak for forus packet
> 042143: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
> len 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
> mtu 0, fwdchk FALSE
> 042144: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
> len 40, stop process pak for forus packet
> 042145: Dec 24 16:47:17.285: IP: s=192.168.2.3 (Vlan10), d=192.168.1.30,
> len 40, enqueue feature, Firewall(3), rtype 0, forus FALSE, sendself FALSE,
> mtu 0, fwdchk FALSE****
>
> ****
>
>
> ****
>
> On Sat, Dec 24, 2011 at 9:25 AM, Chuck Church <chuckchurch at gmail.com>
> wrote:****
>
> Silly question maybe, but do you have any logging in your ACLs? If not,
> that first bug sounds possible. I’ve got a 2821 running 12.4(25f), doing
> NAT overload with heavy QOS and policy routing, get about 99% route-cache
> in both directions. Which is similar to your config when inspection is
> off. IOS issue seems plausible.****
>
> ****
>
> Chuck****
>
> ****
>
> *From:* Jmail Clist [mailto:jmlist80 at gmail.com]
> *Sent:* Friday, December 23, 2011 4:41 PM
> *To:* Reuben Farrelly
> *Cc:* Chuck Church; cisco-nsp at puck.nether.net ****
>
>
> *Subject:* Re: [c-nsp] Cisco 2811 performance issue - dual(new) isp****
>
> ****
>
> After running for most of the days, things are back to getting mainly
> process switched. ?? Strange.****
>
> ****
>
> rtr2811#sh int fa0/1 stats
> FastEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 3366529 213364344 66121 21868973
> Route cache 57045 40344237 50866 11970836
> Total 3423574 253708581 116987 33839809****
>
> On Fri, Dec 23, 2011 at 9:45 AM, Jmail Clist <jmlist80 at gmail.com> wrote:**
> **
>
>
>
> That cef command was pretty useful. Before you scroll down to the
> output/stats, here are the only two ****
>
> bugs that look like they might be related to my issue. With test #1,
> (everything disabled), it was ALL ****
>
> process switched. Test #2 looks slightly better with only IP
> virtual-reassembly enabled. Something is ****
>
> going on here and I'm more puzzled than ever. Test #3 caused lots of
> process switching when doing the speed tests(???). Test #4 is even more
> surprising because things seem better under "normal" traffic loads.
> Thoughts?****
>
> ****
>
> I'd like to find a FTP server to test against instead of using speedguide,
> speakeasy, etc.****
>
>
> CSCsa67785 Bug Details
> crypto-map/NAT/IPS wont work properly in CEF path
> Symptoms: Packets may be dropped on the interface when NAT/IPSEC/IPS is
> configured on the same interface.
> Conditions: If IPSec/NAT and CBAC or IPS/IDS is configured on the same
> interface and the packet gets punted by any of the features, then the
> packet
> may be dropped.
> Workaround: Remove from the configuration the feature which punts the
> packet
> to process path.****
>
> CSCtd25213 Bug Details
> NAT not working for locally generated packets
> Symptoms: NAT is not working for locally-generated packets.
> Conditions: This symptom is observed when NAT is configured for inside and
> outside addresses, and when a self-generated packet is sent to OL.
> Workaround: Instead of using dynamic NAT, use static NAT for
> self-generated
> packets. ****
>
>
> 1) disabled cbac/acl and ip virtual-reassembly****
>
> interface FastEthernet0/1
> ip address x.x.x.x 255.255.255.0****
>
> no ip redirects
> ip nat outside****
>
> no ip virtual-reassembly
> duplex auto
> speed auto
> end ****
>
> rtr2811#sh int fa0/1 stats
> FastEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out****
>
> Processor 12212 757602 133 16723
> Route cache 173 20535 270 35125
> Total 12385 778137 403 51848
> rtr2811#sh ip cef switching statistics feature
> IPv4 CEF input features:
> Feature Drop Consume Punt Punt2Host Gave
> route
> NAT Outside 0 0 0
> 25 0
> Total 0 0 0
> 25 0 ****
>
> IPv4 CEF output features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Post-routing NAT 0 0 0
> 68 0
> Total 0 0 0
> 68 0****
>
> IPv4 CEF post-encap features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF for us features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF punt features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF local features:
> Feature Drop Consume Punt Punt2Host Gave
> route
> Total 0 0 0
> 0 0
> rtr2811#****
>
>
> 2) enabled ip virtual-reassembly ONLY ****
>
>
> interface FastEthernet0/1
> ip address x.x.x.x 255.255.255.0****
>
> no ip redirects
> ip nat outside****
>
> ip virtual-reassembly
> duplex auto
> speed auto****
>
> end ****
>
> rtr2811#sh int fa0/1 stats
> FastEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out****
>
> Processor 1277 78657 16 1589
> Route cache 14 3851 32 4087
> Total 1291 82508 48 5676
> rtr2811#sh ip cef switching statistics feature
> IPv4 CEF input features:
> Feature Drop Consume Punt Punt2Host Gave
> route
> NAT Outside 0 0 0
> 1 0
> Total 0 0 0
> 1 0 ****
>
> IPv4 CEF output features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Post-routing NAT 0 0 0
> 12 0
> Total 0 0 0
> 12 0****
>
> IPv4 CEF post-encap features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF for us features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF punt features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF local features:
> Feature Drop Consume Punt Punt2Host Gave
> route
> Total 0 0 0
> 0 0
> rtr2811#****
>
>
> NOTE: After this I enabled CBAC-int & Ext_ACL-inbound again. Performance
> was almost good as #2 still. I ****
>
> also cleared counters once more and waited 10 minutes. Here are the
> results again. Any ideas????****
>
>
> 3) I ran a speedtest on www.speakeasy.net and process switching went
> through the roo****
>
> rtr2811#sh int fa0/1 stats
> FastEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out****
>
> Processor 17858 1157573 467 143934
> Route cache 1072 964530 837 98966
> Total 18930 2122103 1304 242900
> rtr2811#
> rtr2811#running speedtest now
> ^
> % Invalid input detected at '^' marker. ****
>
> rtr2811#sh int fa0/1 stats
> FastEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out****
>
> Processor 21414 1379133 507 159277
> Route cache 10317 10944391 8426 7415536
> Total 31731 12323524 8933 7574813 ****
>
>
> rtr2811#sh int fa0/1 stats
> FastEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out****
>
> Processor 21490 1384753 513 162841
> Route cache 10322 10946281 8426 7415536
> Total 31812 12331034 8939 7578377
> rtr2811# ****
>
> 4) cleared counters one last time and let it from midnight to 9:39am****
>
> rtr2811#sh int fa0/1 stats
> FastEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out****
>
> Processor 2091010 132620733 42136 13987400
> Route cache 42156 32749186 36559 10473996
> Total 2133166 165369919 78695 24461396
> rtr2811#sh ip cef switching statistics feature
> IPv4 CEF input features:
> Feature Drop Consume Punt Punt2Host Gave
> route
> Access List 11840 0 0
> 13286 0
> NAT Outside 0 0 0
> 3389 0
> Total 11840 0 0
> 16675 0 ****
>
> IPv4 CEF output features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Post-routing NAT 0 0 0
> 28310 0
> Firewall (inspec 57 0 0
> 13 0
> Total 57 0 0
> 28323 0****
>
> IPv4 CEF post-encap features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF for us features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF punt features:
> Feature Drop Consume Punt Punt2Host New
> i/f
> Total 0 0 0
> 0 0****
>
> IPv4 CEF local features:
> Feature Drop Consume Punt Punt2Host Gave
> route
> Total 0 0 0
> 0 0
> rtr2811#****
>
> On Thu, Dec 22, 2011 at 4:24 PM, Reuben Farrelly <
> reuben-cisco-nsp at reub.net> wrote:****
>
> The command:
>
> router#show ip cef switching statistics feature
>
> Will show you which feature is causing traffic to be punted to CPU.
>
> Reuben ****
>
>
>
>
> On 23/12/2011 7:42 AM, Chuck Church wrote:****
>
> You're on the right path. The more important number is the packets in/out,
> as opposed to the characters. Look at the ratio of packets in/out for
> processor vs. Route-cache for the two interfaces. Fa0/1 is process
> switching about 80% of them inbound. That's pretty bad. The output
> looks
> better. Compare that to VLAN 10, where in both directions, only about 10%
> are process switched. The stats for the switchports are meaningless, so
> you
> can ignore those as the switch ASICs deal with those, until they hit the
> VLAN int. Figure out what feature (or IOS bug??) is causing so much
> process
> switching, and I think it'll get better.****
>
> ****
>
> ****
>
> ** **
>
More information about the cisco-nsp
mailing list