[c-nsp] Better way of finding out the source of process switched
traffic?
Rodney Dunn
rodunn at cisco.com
Tue Feb 1 19:31:05 EST 2005
On Tue, Feb 01, 2005 at 06:47:49PM -0500, Dave Temkin wrote:
> Update - This was caused by Stateful NAT (snat). Once I disabled NAT,
> the punted counter stopped incrementing.
>
> CEF Translated packets: 6961, CEF Punted packets: 195089
I'm confused. What command was this from?
Did you upgrade?
Are you saying you were still punting after the upgrade?
If so that doesn't seem right to me even if you
were doing SNAT.
I've not worked on a SNAT setup yet either.
Rodney
>
>
> Mind you, that was just after a reboot (no more than 5 minutes ago) so
> obviously it was punting almost everything.
>
> -Dave
>
> On Thu, 27 Jan 2005, Rodney Dunn wrote:
>
> > 12.3(4)T or later...
> >
> > Pick the latest 12.3(8)T(x)
> > 12.3(11)T(x) image on CCO.
> >
> >
> > Rodney
> >
> > On Thu, Jan 27, 2005 at 09:28:36AM -0500, Dave Temkin wrote:
> > > Were these rolled into 12.3 mainline at any point or do I need to stick to
> > > T?
> > >
> > > Thanks,
> > > -Dave
> > >
> > > On Thu, 27 Jan 2005, Rodney Dunn wrote:
> > >
> > > > I think that is still the problem.
> > > >
> > > > You punt those control packets even
> > > > though it's no for a NAT'ed address
> > > > because you don't know if it supposed
> > > > to be NAT'ed at that stage.
> > > >
> > > > That was all changed with the NAT improvements
> > > > in 12.3(4)T and later.
> > > >
> > > > fyi..here is the packet decode. Notice the FIN
> > > > bit is set.
> > > >
> > > > Frame 1 (70 bytes on wire, 70 bytes captured)
> > > > Arrival Time: Dec 31, 1969 16:00:00.000000000
> > > > Time delta from previous packet: 0.000000000 seconds
> > > > Time since reference or first frame: 0.000000000 seconds
> > > > Frame Number: 1
> > > > Packet Length: 70 bytes
> > > > Capture Length: 70 bytes
> > > > Ethernet II, Src: 00:02:16:30:a5:81, Dst: 00:00:0c:07:ac:66
> > > > Destination: 00:00:0c:07:ac:66 (00:00:0c:07:ac:66)
> > > > Source: 00:02:16:30:a5:81 (00:02:16:30:a5:81)
> > > > Type: IP (0x0800)
> > > > Trailer: 000000000000050000000000
> > > > Frame check sequence: 0x00000000 (incorrect, should be 0x13e8ba13)
> > > > Internet Protocol, Src Addr: 166.90.137.22 (166.90.137.22), Dst Addr: Y.Y.Y.Y
> > > > Version: 4
> > > > Header length: 20 bytes
> > > > Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
> > > > 0000 00.. = Differentiated Services Codepoint: Default (0x00)
> > > > .... ..0. = ECN-Capable Transport (ECT): 0
> > > > .... ...0 = ECN-CE: 0
> > > > Total Length: 40
> > > > Identification: 0x1c71 (7281)
> > > > Flags: 0x04 (Don't Fragment)
> > > > 0... = Reserved bit: Not set
> > > > .1.. = Don't fragment: Set
> > > > ..0. = More fragments: Not set
> > > > Fragment offset: 0
> > > > Time to live: 56
> > > > Protocol: TCP (0x06)
> > > > Header checksum: 0x03b6 (correct)
> > > > Source: 166.90.137.22 (166.90.137.22)
> > > > Destination: 141.162.101.150 (141.162.101.150)
> > > > Transmission Control Protocol, Src Port: 80 (80), Dst Port: 47965 (47965), Seq: 0, Ack: 0, Len: 0
> > > > Source port: 80 (80)
> > > > Destination port: 47965 (47965)
> > > > Sequence number: 0 (relative sequence number)
> > > > Acknowledgement number: 0 (relative ack number)
> > > > Header length: 20 bytes
> > > > Flags: 0x0011 (FIN, ACK)
> > > > 0... .... = Congestion Window Reduced (CWR): Not set
> > > > .0.. .... = ECN-Echo: Not set
> > > > ..0. .... = Urgent: Not set
> > > > ...1 .... = Acknowledgment: Set
> > > > .... 0... = Push: Not set
> > > > .... .0.. = Reset: Not set
> > > > .... ..0. = Syn: Not set
> > > > .... ...1 = Fin: Set
> > > > Window size: 6984
> > > > Checksum: 0xdf5d (correct)
> > > >
> > > > On Thu, Jan 27, 2005 at 09:14:42AM -0500, Dave Temkin wrote:
> > > > > Oddly enough - it's all HTTP traffic - and it seems perfectly normal..
> > > > > And it's not going to one of the NAT'ed addresses.
> > > > >
> > > > > bala-choke-1#show buff input-interface f2/0 packet
> > > > >
> > > > > Public particle pools:
> > > > >
> > > > > Private particle pools:
> > > > >
> > > > > bala-choke-1#show buff input-interface f2/0 packet
> > > > >
> > > > > Buffer information for Small buffer at 0x63FCBB14
> > > > > data_area 0xF25E824, refcount 1, next 0x63FC7374, flags 0x280
> > > > > linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
> > > > > if_input 0x638875AC (FastEthernet2/0), if_output 0x0 (None)
> > > > > inputtime 31w5d (elapsed 00:00:04.648)
> > > > > outputtime 00:00:00.000 (elapsed never), oqnumber 65535
> > > > > datagramstart 0xF25E86A, datagramsize 60, maximum size 260
> > > > > mac_start 0xF25E86A, addr_start 0xF25E86A, info_start 0x0
> > > > > network_start 0xF25E878, transport_start 0xF25E88C, caller_pc 0x608957C4
> > > > >
> > > > > source: 166.90.137.22, destination: y.y.y.y, id: 0x1C71, ttl:
> > > > > 56,
> > > > > TOS: 0 prot: 6, source port 80, destination port 47965
> > > > >
> > > > > 0F25E860: 0000 0C07AC66 ....,f
> > > > > 0F25E870: 00021630 A5810800 45000028 1C714000 ...0%...E..(.q at .
> > > > > 0F25E880: 380603B6 A65A8916 8DA26596 0050BB5D 8..6&Z..."e..P;]
> > > > > 0F25E890: 0EDE0396 BA2C0A36 50111B48 DF5D0000 .^..:,.6P..H_]..
> > > > > 0F25E8A0: 00000000 000005 .......
> > > > >
> > > > >
> > > > > bala-choke-1#show buff input-interface f2/0 packet
> > > > >
> > > > > Buffer information for Big buffer at 0x6420A0CC
> > > > > data_area 0xF292BC4, refcount 1, next 0x0, flags 0x280
> > > > > linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
> > > > > if_input 0x638875AC (FastEthernet2/0), if_output 0x0 (None)
> > > > > inputtime 00:00:00.000 (elapsed never)
> > > > > outputtime 00:00:00.000 (elapsed never), oqnumber 65535
> > > > > datagramstart 0xF292C0A, datagramsize 1253, maximum size 1692
> > > > > mac_start 0xF292C0A, addr_start 0xF292C0A, info_start 0x0
> > > > > network_start 0xF292C18, transport_start 0xF292C2C, caller_pc 0x608957C4
> > > > >
> > > > > source: 63.211.153.3, destination: y.y.y.x, id: 0x66D2, ttl:
> > > > > 119,
> > > > > TOS: 0 prot: 6, source port 80, destination port 61477
> > > > >
> > > > > 0F292C00: 0000 0C07AC66 ....,f
> > > > > 0F292C10: 00021630 A5810800 450004D7 66D24000 ...0%...E..WfR at .
> > > > >
> > > > >
> > > > >
> > > > > -Dave
> > > > >
> > > > > On Thu, 27 Jan 2005, Rodney Dunn wrote:
> > > > >
> > > > > > Tidbit: When you are collecting command outputs to
> > > > > > help troubleshoot enable this for your terminal
> > > > > > session:
> > > > > >
> > > > > > terminal exec prompt timestamp
> > > > > >
> > > > > > Then do:
> > > > > >
> > > > > > clear counters
> > > > > > sh int stat
> > > > > > sh int | incl protocol|bits/sec
> > > > > >
> > > > > > wait 30 seconds
> > > > > >
> > > > > > clear counters
> > > > > > sh int stat
> > > > > > sh int | incl protocol|bits/sec
> > > > > >
> > > > > >
> > > > > > what percentage of the traffic is being
> > > > > > punted to process level?
> > > > > >
> > > > > > Run the 'sh buff' commands like I gave you
> > > > > > and let's see a few of the packets.
> > > > > >
> > > > > > Rodney
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > tOn Thu, Jan 27, 2005 at 08:58:05AM -0500, Dave Temkin wrote:
> > > > > > > Thanks Rodney.
> > > > > > >
> > > > > > > The one thing I'm hesitant to blame it on is the fact that the actual
> > > > > > > NAT'ed traffic is very very little (it's AIM conversations, that's it).
> > > > > > > So I'm wondering why on a box that's as big as this one (NPE-400) it'd
> > > > > > > choke on that...
> > > > > > >
> > > > > > > -Dave
> > > > > > >
> > > > > > > --
> > > > > > > David Temkin
> > > > > > >
> > > > > > > On Thu, 27 Jan 2005, Rodney Dunn wrote:
> > > > > > >
> > > > > > > > Your problem is almost surely that the packets
> > > > > > > > being punted are TCP control packets where
> > > > > > > > we punt to create/tear down the translations.
> > > > > > > > SYN, FIN, RST.
> > > > > > > >
> > > > > > > > If you want to see the packets at process
> > > > > > > > level you can either turn on:
> > > > > > > > debug ip packet <acl> to limit the granularity of
> > > > > > > > the debug since that only prints packets at process
> > > > > > > > level. /*not true for 12.2S with the right commands*/
> > > > > > > >
> > > > > > > > You can also do "sh buffers input-interface <name> packet"
> > > > > > > > a few times and catch some packets in the buffer and
> > > > > > > > manually decode the TCP header to see if the flags are set
> > > > > > > > in the header.
> > > > > > > >
> > > > > > > > Now, in 12.3(4)T and later we made some NAT enhancements
> > > > > > > > where we create the flows in the CEF path without punting
> > > > > > > > traffic. That is the suggested way to go if you are seeing
> > > > > > > > a high number of process switched traffic with NAT enabled.
> > > > > > > >
> > > > > > > > Rodney
> > > > > > > >
> > > > > > > > On Thu, Jan 27, 2005 at 07:42:33AM -0500, Dave Temkin wrote:
> > > > > > > > > I've got an internet-facing router that's seeing a very high rate of
> > > > > > > > > process switched traffic. Nothing too crazy is configured on this router
> > > > > > > > > - a little bit of NAT, a couple of route maps, BGP. That's about it.
> > > > > > > > > Aside from doing a debug ip packet and killing the router (it's passing
> > > > > > > > > about 30-40mbit of traffic), are there any other options for tracking down
> > > > > > > > > what's in the process queue? Router is running 12.3.6a
> > > > > > > > >
> > > > > > > > > FastEthernet0/0
> > > > > > > > > Throttle count 4
> > > > > > > > > Drops RP 5 SP 0
> > > > > > > > > SPD Flushes Fast 3103 SSE 0
> > > > > > > > > SPD Aggress Fast 0
> > > > > > > > > SPD Priority Inputs 83215964 Drops 0
> > > > > > > > >
> > > > > > > > > Protocol IP
> > > > > > > > > Switching path Pkts In Chars In Pkts Out Chars Out
> > > > > > > > > Process 1803602701 4025634609 1661069368 456573125
> > > > > > > > > Cache misses 0 - - -
> > > > > > > > > Fast 2713542052 1802705001 3837108389 304620460
> > > > > > > > > Auton/SSE 0 0 0 0
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > FastEthernet1/0 Outside
> > > > > > > > > Throttle count 0
> > > > > > > > > Drops RP 0 SP 0
> > > > > > > > > SPD Flushes Fast 1796 SSE 0
> > > > > > > > > SPD Aggress Fast 0
> > > > > > > > > SPD Priority Inputs 6927146 Drops 0
> > > > > > > > >
> > > > > > > > > Protocol IP
> > > > > > > > > Switching path Pkts In Chars In Pkts Out Chars Out
> > > > > > > > > Process 543622379 2397426796 317919218 1743487367
> > > > > > > > > Cache misses 0 - - -
> > > > > > > > > Fast 3071349692 1923264716 1211037578 2505497398
> > > > > > > > > Auton/SSE 0 0 0 0
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > FastEthernet2/0 Outside 2
> > > > > > > > > Throttle count 0
> > > > > > > > > Drops RP 0 SP 0
> > > > > > > > > SPD Flushes Fast 1480 SSE 0
> > > > > > > > > SPD Aggress Fast 0
> > > > > > > > > SPD Priority Inputs 42435822 Drops 0
> > > > > > > > >
> > > > > > > > > Protocol IP
> > > > > > > > > Switching path Pkts In Chars In Pkts Out Chars Out
> > > > > > > > > Process 1056561152 1934756549 1414955036 1939153311
> > > > > > > > > Cache misses 0 - - -
> > > > > > > > > Fast 819752907 318162363 1502399074 1302221329
> > > > > > > > > Auton/SSE 0 0 0 0
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > !
> > > > > > > > > interface FastEthernet0/0.101
> > > > > > > > > encapsulation dot1Q 101
> > > > > > > > > ip address x.x.x.x x.x.x.x.x
> > > > > > > > > no ip redirects
> > > > > > > > > no ip proxy-arp
> > > > > > > > > ip nat inside
> > > > > > > > > ip policy route-map RM101
> > > > > > > > > no cdp enable
> > > > > > > > > standby 101 ip x.x.x.y
> > > > > > > > > standby 101 timers 1 3
> > > > > > > > > standby 101 priority 250
> > > > > > > > > standby 101 preempt
> > > > > > > > > standby 101 name HSRP101
> > > > > > > > > !
> > > > > > > > >
> > > > > > > > > !
> > > > > > > > > interface FastEthernet1/0
> > > > > > > > > description Outside 1
> > > > > > > > > ip address x.x.x.x x.x.x.x
> > > > > > > > > ip access-group Yipes-Outside in
> > > > > > > > > ip nat outside
> > > > > > > > > load-interval 30
> > > > > > > > > duplex full
> > > > > > > > > ntp disable
> > > > > > > > > hold-queue 300 in
> > > > > > > > > hold-queue 300 out
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -Dave
> > > > > > > > > _______________________________________________
> > > > > > > > > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > > > > > > > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > > > > > > > archive at http://puck.nether.net/pipermail/cisco-nsp/
> > > > > > > >
> > > > > >
> > > >
> >
More information about the cisco-nsp
mailing list