[c-nsp] Better way of finding out the source of process switched traffic?

Dave Temkin dave at ordinaryworld.com
Tue Feb 1 18:47:49 EST 2005


Update - This was caused by Stateful NAT (snat).  Once I disabled NAT,
the punted counter stopped incrementing.

CEF Translated packets: 6961, CEF Punted packets: 195089


Mind you, that was just after a reboot (no more than 5 minutes ago) so
obviously it was punting almost everything.

-Dave

On Thu, 27 Jan 2005, Rodney Dunn wrote:

> 12.3(4)T or later...
>
> Pick the latest 12.3(8)T(x)
> 12.3(11)T(x) image on CCO.
>
>
> Rodney
>
> On Thu, Jan 27, 2005 at 09:28:36AM -0500, Dave Temkin wrote:
> > Were these rolled into 12.3 mainline at any point or do I need to stick to
> > T?
> >
> > Thanks,
> > -Dave
> >
> > On Thu, 27 Jan 2005, Rodney Dunn wrote:
> >
> > > I think that is still the problem.
> > >
> > > You punt those control packets even
> > > though it's no for a NAT'ed address
> > > because you don't know if it supposed
> > > to be NAT'ed at that stage.
> > >
> > > That was all changed with the NAT improvements
> > > in 12.3(4)T and later.
> > >
> > > fyi..here is the packet decode.  Notice the FIN
> > > bit is set.
> > >
> > > Frame 1 (70 bytes on wire, 70 bytes captured)
> > >     Arrival Time: Dec 31, 1969 16:00:00.000000000
> > >     Time delta from previous packet: 0.000000000 seconds
> > >     Time since reference or first frame: 0.000000000 seconds
> > >     Frame Number: 1
> > >     Packet Length: 70 bytes
> > >     Capture Length: 70 bytes
> > > Ethernet II, Src: 00:02:16:30:a5:81, Dst: 00:00:0c:07:ac:66
> > >     Destination: 00:00:0c:07:ac:66 (00:00:0c:07:ac:66)
> > >     Source: 00:02:16:30:a5:81 (00:02:16:30:a5:81)
> > >     Type: IP (0x0800)
> > >     Trailer: 000000000000050000000000
> > >     Frame check sequence: 0x00000000 (incorrect, should be 0x13e8ba13)
> > > Internet Protocol, Src Addr: 166.90.137.22 (166.90.137.22), Dst Addr:  Y.Y.Y.Y
> > >     Version: 4
> > >     Header length: 20 bytes
> > >     Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
> > >         0000 00.. = Differentiated Services Codepoint: Default (0x00)
> > >         .... ..0. = ECN-Capable Transport (ECT): 0
> > >         .... ...0 = ECN-CE: 0
> > >     Total Length: 40
> > >     Identification: 0x1c71 (7281)
> > >     Flags: 0x04 (Don't Fragment)
> > >         0... = Reserved bit: Not set
> > >         .1.. = Don't fragment: Set
> > >         ..0. = More fragments: Not set
> > >     Fragment offset: 0
> > >     Time to live: 56
> > >     Protocol: TCP (0x06)
> > >     Header checksum: 0x03b6 (correct)
> > >     Source: 166.90.137.22 (166.90.137.22)
> > >     Destination: 141.162.101.150 (141.162.101.150)
> > > Transmission Control Protocol, Src Port: 80 (80), Dst Port: 47965 (47965), Seq: 0, Ack: 0, Len: 0
> > >     Source port: 80 (80)
> > >     Destination port: 47965 (47965)
> > >     Sequence number: 0    (relative sequence number)
> > >     Acknowledgement number: 0    (relative ack number)
> > >     Header length: 20 bytes
> > >     Flags: 0x0011 (FIN, ACK)
> > >         0... .... = Congestion Window Reduced (CWR): Not set
> > >         .0.. .... = ECN-Echo: Not set
> > >         ..0. .... = Urgent: Not set
> > >         ...1 .... = Acknowledgment: Set
> > >         .... 0... = Push: Not set
> > >         .... .0.. = Reset: Not set
> > >         .... ..0. = Syn: Not set
> > >         .... ...1 = Fin: Set
> > >     Window size: 6984
> > >     Checksum: 0xdf5d (correct)
> > >
> > > On Thu, Jan 27, 2005 at 09:14:42AM -0500, Dave Temkin wrote:
> > > > Oddly enough - it's all HTTP traffic - and it seems perfectly normal..
> > > > And it's not going to one of the NAT'ed addresses.
> > > >
> > > > bala-choke-1#show buff input-interface f2/0 packet
> > > >
> > > > Public particle pools:
> > > >
> > > > Private particle pools:
> > > >
> > > > bala-choke-1#show buff input-interface f2/0 packet
> > > >
> > > > Buffer information for Small buffer at 0x63FCBB14
> > > >   data_area 0xF25E824, refcount 1, next 0x63FC7374, flags 0x280
> > > >   linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
> > > >   if_input 0x638875AC (FastEthernet2/0), if_output 0x0 (None)
> > > >   inputtime 31w5d (elapsed 00:00:04.648)
> > > >   outputtime 00:00:00.000 (elapsed never), oqnumber 65535
> > > >   datagramstart 0xF25E86A, datagramsize 60, maximum size 260
> > > >   mac_start 0xF25E86A, addr_start 0xF25E86A, info_start 0x0
> > > >   network_start 0xF25E878, transport_start 0xF25E88C, caller_pc 0x608957C4
> > > >
> > > >   source: 166.90.137.22, destination: y.y.y.y, id: 0x1C71, ttl:
> > > > 56,
> > > >   TOS: 0 prot: 6, source port 80, destination port 47965
> > > >
> > > > 0F25E860:                       0000 0C07AC66            ....,f
> > > > 0F25E870: 00021630 A5810800 45000028 1C714000  ...0%...E..(.q at .
> > > > 0F25E880: 380603B6 A65A8916 8DA26596 0050BB5D  8..6&Z..."e..P;]
> > > > 0F25E890: 0EDE0396 BA2C0A36 50111B48 DF5D0000  .^..:,.6P..H_]..
> > > > 0F25E8A0: 00000000 000005                      .......
> > > >
> > > >
> > > > bala-choke-1#show buff input-interface f2/0 packet
> > > >
> > > > Buffer information for Big buffer at 0x6420A0CC
> > > >   data_area 0xF292BC4, refcount 1, next 0x0, flags 0x280
> > > >   linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
> > > >   if_input 0x638875AC (FastEthernet2/0), if_output 0x0 (None)
> > > >   inputtime 00:00:00.000 (elapsed never)
> > > >   outputtime 00:00:00.000 (elapsed never), oqnumber 65535
> > > >   datagramstart 0xF292C0A, datagramsize 1253, maximum size 1692
> > > >   mac_start 0xF292C0A, addr_start 0xF292C0A, info_start 0x0
> > > >   network_start 0xF292C18, transport_start 0xF292C2C, caller_pc 0x608957C4
> > > >
> > > >   source: 63.211.153.3, destination: y.y.y.x, id: 0x66D2, ttl:
> > > > 119,
> > > >   TOS: 0 prot: 6, source port 80, destination port 61477
> > > >
> > > > 0F292C00:                       0000 0C07AC66            ....,f
> > > > 0F292C10: 00021630 A5810800 450004D7 66D24000  ...0%...E..WfR at .
> > > >
> > > >
> > > >
> > > > -Dave
> > > >
> > > > On Thu, 27 Jan 2005, Rodney Dunn wrote:
> > > >
> > > > > Tidbit: When you are collecting command outputs to
> > > > > help troubleshoot enable this for your terminal
> > > > > session:
> > > > >
> > > > > terminal exec prompt timestamp
> > > > >
> > > > > Then do:
> > > > >
> > > > > clear counters
> > > > > sh int stat
> > > > > sh int | incl protocol|bits/sec
> > > > >
> > > > > wait 30 seconds
> > > > >
> > > > > clear counters
> > > > > sh int stat
> > > > > sh int | incl protocol|bits/sec
> > > > >
> > > > >
> > > > > what percentage of the traffic is being
> > > > > punted to process level?
> > > > >
> > > > > Run the 'sh buff' commands like I gave you
> > > > > and let's see a few of the packets.
> > > > >
> > > > > Rodney
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > tOn Thu, Jan 27, 2005 at 08:58:05AM -0500, Dave Temkin wrote:
> > > > > > Thanks Rodney.
> > > > > >
> > > > > > The one thing I'm hesitant to blame it on is the fact that the actual
> > > > > > NAT'ed traffic is very very little (it's AIM conversations, that's it).
> > > > > > So I'm wondering why on a box that's as big as this one (NPE-400) it'd
> > > > > > choke on that...
> > > > > >
> > > > > > -Dave
> > > > > >
> > > > > > --
> > > > > > David Temkin
> > > > > >
> > > > > > On Thu, 27 Jan 2005, Rodney Dunn wrote:
> > > > > >
> > > > > > > Your problem is almost surely that the packets
> > > > > > > being punted are TCP control packets where
> > > > > > > we punt to create/tear down the translations.
> > > > > > > SYN, FIN, RST.
> > > > > > >
> > > > > > > If you want to see the packets at process
> > > > > > > level you can either turn on:
> > > > > > > debug ip packet <acl> to limit the granularity of
> > > > > > > the debug since that only prints packets at process
> > > > > > > level. /*not true for 12.2S with the right commands*/
> > > > > > >
> > > > > > > You can also do "sh buffers input-interface <name> packet"
> > > > > > > a few times and catch some packets in the buffer and
> > > > > > > manually decode the TCP header to see if the flags are set
> > > > > > > in the header.
> > > > > > >
> > > > > > > Now, in 12.3(4)T and later we made some NAT enhancements
> > > > > > > where we create the flows in the CEF path without punting
> > > > > > > traffic.  That is the suggested way to go if you are seeing
> > > > > > > a high number of process switched traffic with NAT enabled.
> > > > > > >
> > > > > > > Rodney
> > > > > > >
> > > > > > > On Thu, Jan 27, 2005 at 07:42:33AM -0500, Dave Temkin wrote:
> > > > > > > > I've got an internet-facing router that's seeing a very high rate of
> > > > > > > > process switched traffic.  Nothing too crazy is configured on this router
> > > > > > > > - a little bit of NAT, a couple of route maps, BGP.  That's about it.
> > > > > > > > Aside from doing a debug ip packet and killing the router (it's passing
> > > > > > > > about 30-40mbit of traffic), are there any other options for tracking down
> > > > > > > > what's in the process queue?  Router is running 12.3.6a
> > > > > > > >
> > > > > > > > FastEthernet0/0
> > > > > > > >           Throttle count          4
> > > > > > > >                    Drops         RP          5         SP          0
> > > > > > > >              SPD Flushes       Fast       3103        SSE          0
> > > > > > > >              SPD Aggress       Fast          0
> > > > > > > >             SPD Priority     Inputs   83215964      Drops          0
> > > > > > > >
> > > > > > > >     Protocol  IP
> > > > > > > >           Switching path    Pkts In   Chars In   Pkts Out  Chars Out
> > > > > > > >                  Process 1803602701 4025634609 1661069368  456573125
> > > > > > > >             Cache misses          0          -          -          -
> > > > > > > >                     Fast 2713542052 1802705001 3837108389  304620460
> > > > > > > >                Auton/SSE          0          0          0          0
> > > > > > > >
> > > > > > > >
> > > > > > > > FastEthernet1/0 Outside
> > > > > > > >           Throttle count          0
> > > > > > > >                    Drops         RP          0         SP          0
> > > > > > > >              SPD Flushes       Fast       1796        SSE          0
> > > > > > > >              SPD Aggress       Fast          0
> > > > > > > >             SPD Priority     Inputs    6927146      Drops          0
> > > > > > > >
> > > > > > > >     Protocol  IP
> > > > > > > >           Switching path    Pkts In   Chars In   Pkts Out  Chars Out
> > > > > > > >                  Process  543622379 2397426796  317919218 1743487367
> > > > > > > >             Cache misses          0          -          -          -
> > > > > > > >                     Fast 3071349692 1923264716 1211037578 2505497398
> > > > > > > >                Auton/SSE          0          0          0          0
> > > > > > > >
> > > > > > > >
> > > > > > > > FastEthernet2/0 Outside 2
> > > > > > > >           Throttle count          0
> > > > > > > >                    Drops         RP          0         SP          0
> > > > > > > >              SPD Flushes       Fast       1480        SSE          0
> > > > > > > >              SPD Aggress       Fast          0
> > > > > > > >             SPD Priority     Inputs   42435822      Drops          0
> > > > > > > >
> > > > > > > >     Protocol  IP
> > > > > > > >           Switching path    Pkts In   Chars In   Pkts Out  Chars Out
> > > > > > > >                  Process 1056561152 1934756549 1414955036 1939153311
> > > > > > > >             Cache misses          0          -          -          -
> > > > > > > >                     Fast  819752907  318162363 1502399074 1302221329
> > > > > > > >                Auton/SSE          0          0          0          0
> > > > > > > >
> > > > > > > >
> > > > > > > > !
> > > > > > > > interface FastEthernet0/0.101
> > > > > > > >  encapsulation dot1Q 101
> > > > > > > >  ip address x.x.x.x x.x.x.x.x
> > > > > > > >  no ip redirects
> > > > > > > >  no ip proxy-arp
> > > > > > > >  ip nat inside
> > > > > > > >  ip policy route-map RM101
> > > > > > > >  no cdp enable
> > > > > > > >  standby 101 ip x.x.x.y
> > > > > > > >  standby 101 timers 1 3
> > > > > > > >  standby 101 priority 250
> > > > > > > >  standby 101 preempt
> > > > > > > >  standby 101 name HSRP101
> > > > > > > > !
> > > > > > > >
> > > > > > > > !
> > > > > > > > interface FastEthernet1/0
> > > > > > > >  description Outside 1
> > > > > > > >  ip address x.x.x.x x.x.x.x
> > > > > > > >  ip access-group Yipes-Outside in
> > > > > > > >  ip nat outside
> > > > > > > >  load-interval 30
> > > > > > > >  duplex full
> > > > > > > >  ntp disable
> > > > > > > >  hold-queue 300 in
> > > > > > > >  hold-queue 300 out
> > > > > > > >
> > > > > > > >
> > > > > > > > -Dave
> > > > > > > > _______________________________________________
> > > > > > > > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > > > > > > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > > > > > > archive at http://puck.nether.net/pipermail/cisco-nsp/
> > > > > > >
> > > > >
> > >
>


More information about the cisco-nsp mailing list