[c-nsp] Better way of finding out the source of process switched
traffic?
Rodney Dunn
rodunn at cisco.com
Thu Jan 27 09:33:13 EST 2005
12.3(4)T or later...
Pick the latest 12.3(8)T(x)
12.3(11)T(x) image on CCO.
Rodney
On Thu, Jan 27, 2005 at 09:28:36AM -0500, Dave Temkin wrote:
> Were these rolled into 12.3 mainline at any point or do I need to stick to
> T?
>
> Thanks,
> -Dave
>
> On Thu, 27 Jan 2005, Rodney Dunn wrote:
>
> > I think that is still the problem.
> >
> > You punt those control packets even
> > though it's no for a NAT'ed address
> > because you don't know if it supposed
> > to be NAT'ed at that stage.
> >
> > That was all changed with the NAT improvements
> > in 12.3(4)T and later.
> >
> > fyi..here is the packet decode. Notice the FIN
> > bit is set.
> >
> > Frame 1 (70 bytes on wire, 70 bytes captured)
> > Arrival Time: Dec 31, 1969 16:00:00.000000000
> > Time delta from previous packet: 0.000000000 seconds
> > Time since reference or first frame: 0.000000000 seconds
> > Frame Number: 1
> > Packet Length: 70 bytes
> > Capture Length: 70 bytes
> > Ethernet II, Src: 00:02:16:30:a5:81, Dst: 00:00:0c:07:ac:66
> > Destination: 00:00:0c:07:ac:66 (00:00:0c:07:ac:66)
> > Source: 00:02:16:30:a5:81 (00:02:16:30:a5:81)
> > Type: IP (0x0800)
> > Trailer: 000000000000050000000000
> > Frame check sequence: 0x00000000 (incorrect, should be 0x13e8ba13)
> > Internet Protocol, Src Addr: 166.90.137.22 (166.90.137.22), Dst Addr: Y.Y.Y.Y
> > Version: 4
> > Header length: 20 bytes
> > Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
> > 0000 00.. = Differentiated Services Codepoint: Default (0x00)
> > .... ..0. = ECN-Capable Transport (ECT): 0
> > .... ...0 = ECN-CE: 0
> > Total Length: 40
> > Identification: 0x1c71 (7281)
> > Flags: 0x04 (Don't Fragment)
> > 0... = Reserved bit: Not set
> > .1.. = Don't fragment: Set
> > ..0. = More fragments: Not set
> > Fragment offset: 0
> > Time to live: 56
> > Protocol: TCP (0x06)
> > Header checksum: 0x03b6 (correct)
> > Source: 166.90.137.22 (166.90.137.22)
> > Destination: 141.162.101.150 (141.162.101.150)
> > Transmission Control Protocol, Src Port: 80 (80), Dst Port: 47965 (47965), Seq: 0, Ack: 0, Len: 0
> > Source port: 80 (80)
> > Destination port: 47965 (47965)
> > Sequence number: 0 (relative sequence number)
> > Acknowledgement number: 0 (relative ack number)
> > Header length: 20 bytes
> > Flags: 0x0011 (FIN, ACK)
> > 0... .... = Congestion Window Reduced (CWR): Not set
> > .0.. .... = ECN-Echo: Not set
> > ..0. .... = Urgent: Not set
> > ...1 .... = Acknowledgment: Set
> > .... 0... = Push: Not set
> > .... .0.. = Reset: Not set
> > .... ..0. = Syn: Not set
> > .... ...1 = Fin: Set
> > Window size: 6984
> > Checksum: 0xdf5d (correct)
> >
> > On Thu, Jan 27, 2005 at 09:14:42AM -0500, Dave Temkin wrote:
> > > Oddly enough - it's all HTTP traffic - and it seems perfectly normal..
> > > And it's not going to one of the NAT'ed addresses.
> > >
> > > bala-choke-1#show buff input-interface f2/0 packet
> > >
> > > Public particle pools:
> > >
> > > Private particle pools:
> > >
> > > bala-choke-1#show buff input-interface f2/0 packet
> > >
> > > Buffer information for Small buffer at 0x63FCBB14
> > > data_area 0xF25E824, refcount 1, next 0x63FC7374, flags 0x280
> > > linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
> > > if_input 0x638875AC (FastEthernet2/0), if_output 0x0 (None)
> > > inputtime 31w5d (elapsed 00:00:04.648)
> > > outputtime 00:00:00.000 (elapsed never), oqnumber 65535
> > > datagramstart 0xF25E86A, datagramsize 60, maximum size 260
> > > mac_start 0xF25E86A, addr_start 0xF25E86A, info_start 0x0
> > > network_start 0xF25E878, transport_start 0xF25E88C, caller_pc 0x608957C4
> > >
> > > source: 166.90.137.22, destination: y.y.y.y, id: 0x1C71, ttl:
> > > 56,
> > > TOS: 0 prot: 6, source port 80, destination port 47965
> > >
> > > 0F25E860: 0000 0C07AC66 ....,f
> > > 0F25E870: 00021630 A5810800 45000028 1C714000 ...0%...E..(.q at .
> > > 0F25E880: 380603B6 A65A8916 8DA26596 0050BB5D 8..6&Z..."e..P;]
> > > 0F25E890: 0EDE0396 BA2C0A36 50111B48 DF5D0000 .^..:,.6P..H_]..
> > > 0F25E8A0: 00000000 000005 .......
> > >
> > >
> > > bala-choke-1#show buff input-interface f2/0 packet
> > >
> > > Buffer information for Big buffer at 0x6420A0CC
> > > data_area 0xF292BC4, refcount 1, next 0x0, flags 0x280
> > > linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
> > > if_input 0x638875AC (FastEthernet2/0), if_output 0x0 (None)
> > > inputtime 00:00:00.000 (elapsed never)
> > > outputtime 00:00:00.000 (elapsed never), oqnumber 65535
> > > datagramstart 0xF292C0A, datagramsize 1253, maximum size 1692
> > > mac_start 0xF292C0A, addr_start 0xF292C0A, info_start 0x0
> > > network_start 0xF292C18, transport_start 0xF292C2C, caller_pc 0x608957C4
> > >
> > > source: 63.211.153.3, destination: y.y.y.x, id: 0x66D2, ttl:
> > > 119,
> > > TOS: 0 prot: 6, source port 80, destination port 61477
> > >
> > > 0F292C00: 0000 0C07AC66 ....,f
> > > 0F292C10: 00021630 A5810800 450004D7 66D24000 ...0%...E..WfR at .
> > >
> > >
> > >
> > > -Dave
> > >
> > > On Thu, 27 Jan 2005, Rodney Dunn wrote:
> > >
> > > > Tidbit: When you are collecting command outputs to
> > > > help troubleshoot enable this for your terminal
> > > > session:
> > > >
> > > > terminal exec prompt timestamp
> > > >
> > > > Then do:
> > > >
> > > > clear counters
> > > > sh int stat
> > > > sh int | incl protocol|bits/sec
> > > >
> > > > wait 30 seconds
> > > >
> > > > clear counters
> > > > sh int stat
> > > > sh int | incl protocol|bits/sec
> > > >
> > > >
> > > > what percentage of the traffic is being
> > > > punted to process level?
> > > >
> > > > Run the 'sh buff' commands like I gave you
> > > > and let's see a few of the packets.
> > > >
> > > > Rodney
> > > >
> > > >
> > > >
> > > >
> > > > tOn Thu, Jan 27, 2005 at 08:58:05AM -0500, Dave Temkin wrote:
> > > > > Thanks Rodney.
> > > > >
> > > > > The one thing I'm hesitant to blame it on is the fact that the actual
> > > > > NAT'ed traffic is very very little (it's AIM conversations, that's it).
> > > > > So I'm wondering why on a box that's as big as this one (NPE-400) it'd
> > > > > choke on that...
> > > > >
> > > > > -Dave
> > > > >
> > > > > --
> > > > > David Temkin
> > > > >
> > > > > On Thu, 27 Jan 2005, Rodney Dunn wrote:
> > > > >
> > > > > > Your problem is almost surely that the packets
> > > > > > being punted are TCP control packets where
> > > > > > we punt to create/tear down the translations.
> > > > > > SYN, FIN, RST.
> > > > > >
> > > > > > If you want to see the packets at process
> > > > > > level you can either turn on:
> > > > > > debug ip packet <acl> to limit the granularity of
> > > > > > the debug since that only prints packets at process
> > > > > > level. /*not true for 12.2S with the right commands*/
> > > > > >
> > > > > > You can also do "sh buffers input-interface <name> packet"
> > > > > > a few times and catch some packets in the buffer and
> > > > > > manually decode the TCP header to see if the flags are set
> > > > > > in the header.
> > > > > >
> > > > > > Now, in 12.3(4)T and later we made some NAT enhancements
> > > > > > where we create the flows in the CEF path without punting
> > > > > > traffic. That is the suggested way to go if you are seeing
> > > > > > a high number of process switched traffic with NAT enabled.
> > > > > >
> > > > > > Rodney
> > > > > >
> > > > > > On Thu, Jan 27, 2005 at 07:42:33AM -0500, Dave Temkin wrote:
> > > > > > > I've got an internet-facing router that's seeing a very high rate of
> > > > > > > process switched traffic. Nothing too crazy is configured on this router
> > > > > > > - a little bit of NAT, a couple of route maps, BGP. That's about it.
> > > > > > > Aside from doing a debug ip packet and killing the router (it's passing
> > > > > > > about 30-40mbit of traffic), are there any other options for tracking down
> > > > > > > what's in the process queue? Router is running 12.3.6a
> > > > > > >
> > > > > > > FastEthernet0/0
> > > > > > > Throttle count 4
> > > > > > > Drops RP 5 SP 0
> > > > > > > SPD Flushes Fast 3103 SSE 0
> > > > > > > SPD Aggress Fast 0
> > > > > > > SPD Priority Inputs 83215964 Drops 0
> > > > > > >
> > > > > > > Protocol IP
> > > > > > > Switching path Pkts In Chars In Pkts Out Chars Out
> > > > > > > Process 1803602701 4025634609 1661069368 456573125
> > > > > > > Cache misses 0 - - -
> > > > > > > Fast 2713542052 1802705001 3837108389 304620460
> > > > > > > Auton/SSE 0 0 0 0
> > > > > > >
> > > > > > >
> > > > > > > FastEthernet1/0 Outside
> > > > > > > Throttle count 0
> > > > > > > Drops RP 0 SP 0
> > > > > > > SPD Flushes Fast 1796 SSE 0
> > > > > > > SPD Aggress Fast 0
> > > > > > > SPD Priority Inputs 6927146 Drops 0
> > > > > > >
> > > > > > > Protocol IP
> > > > > > > Switching path Pkts In Chars In Pkts Out Chars Out
> > > > > > > Process 543622379 2397426796 317919218 1743487367
> > > > > > > Cache misses 0 - - -
> > > > > > > Fast 3071349692 1923264716 1211037578 2505497398
> > > > > > > Auton/SSE 0 0 0 0
> > > > > > >
> > > > > > >
> > > > > > > FastEthernet2/0 Outside 2
> > > > > > > Throttle count 0
> > > > > > > Drops RP 0 SP 0
> > > > > > > SPD Flushes Fast 1480 SSE 0
> > > > > > > SPD Aggress Fast 0
> > > > > > > SPD Priority Inputs 42435822 Drops 0
> > > > > > >
> > > > > > > Protocol IP
> > > > > > > Switching path Pkts In Chars In Pkts Out Chars Out
> > > > > > > Process 1056561152 1934756549 1414955036 1939153311
> > > > > > > Cache misses 0 - - -
> > > > > > > Fast 819752907 318162363 1502399074 1302221329
> > > > > > > Auton/SSE 0 0 0 0
> > > > > > >
> > > > > > >
> > > > > > > !
> > > > > > > interface FastEthernet0/0.101
> > > > > > > encapsulation dot1Q 101
> > > > > > > ip address x.x.x.x x.x.x.x.x
> > > > > > > no ip redirects
> > > > > > > no ip proxy-arp
> > > > > > > ip nat inside
> > > > > > > ip policy route-map RM101
> > > > > > > no cdp enable
> > > > > > > standby 101 ip x.x.x.y
> > > > > > > standby 101 timers 1 3
> > > > > > > standby 101 priority 250
> > > > > > > standby 101 preempt
> > > > > > > standby 101 name HSRP101
> > > > > > > !
> > > > > > >
> > > > > > > !
> > > > > > > interface FastEthernet1/0
> > > > > > > description Outside 1
> > > > > > > ip address x.x.x.x x.x.x.x
> > > > > > > ip access-group Yipes-Outside in
> > > > > > > ip nat outside
> > > > > > > load-interval 30
> > > > > > > duplex full
> > > > > > > ntp disable
> > > > > > > hold-queue 300 in
> > > > > > > hold-queue 300 out
> > > > > > >
> > > > > > >
> > > > > > > -Dave
> > > > > > > _______________________________________________
> > > > > > > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > > > > > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > > > > > archive at http://puck.nether.net/pipermail/cisco-nsp/
> > > > > >
> > > >
> >
More information about the cisco-nsp
mailing list