[c-nsp] Better way of finding out the source of process switched
traffic?
Dave Temkin
dave at ordinaryworld.com
Thu Jan 27 09:18:05 EST 2005
As far as what's getting punted:
bala-choke-1#sh int stat
Load for five secs: 21%/12%; one minute: 18%; five minutes: 18%
Time source is NTP, 09:16:33.362 EST Thu Jan 27 2005
FastEthernet0/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 11 713 15 976
Route cache 13 6882 13 11586
Total 24 7595 28 12562
FastEthernet1/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 0 0 2 122
Route cache 9 8603 13 6830
Total 9 8603 15 6952
FastEthernet2/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 0 0 0 0
Route cache 2 1651 0 0
Total 2 1651 0 0
bala-choke-1#sh int | incl protocol|bits/sec
FastEthernet0/0 is up, line protocol is up
30 second input rate 2785000 bits/sec, 1379 packets/sec
30 second output rate 7066000 bits/sec, 1396 packets/sec
FastEthernet0/0.101 is up, line protocol is up
FastEthernet0/0.103 is up, line protocol is up
FastEthernet0/0.110 is up, line protocol is up
FastEthernet0/0.111 is up, line protocol is up
FastEthernet0/0.800 is up, line protocol is up
FastEthernet0/0.801 is up, line protocol is up
FastEthernet0/0.802 is up, line protocol is up
FastEthernet1/0 is up, line protocol is up
30 second input rate 4767000 bits/sec, 977 packets/sec
30 second output rate 2730000 bits/sec, 1362 packets/sec
FastEthernet2/0 is up, line protocol is up
5 minute input rate 2096000 bits/sec, 359 packets/sec
5 minute output rate 2000 bits/sec, 4 packets/sec
bala-choke-1#sh int | incl protocol|bits/sec
FastEthernet0/0 is up, line protocol is up
30 second input rate 2491000 bits/sec, 1353 packets/sec
30 second output rate 7686000 bits/sec, 1407 packets/sec
FastEthernet0/0.101 is up, line protocol is up
FastEthernet0/0.103 is up, line protocol is up
FastEthernet0/0.110 is up, line protocol is up
FastEthernet0/0.111 is up, line protocol is up
FastEthernet0/0.800 is up, line protocol is up
FastEthernet0/0.801 is up, line protocol is up
FastEthernet0/0.802 is up, line protocol is up
FastEthernet1/0 is up, line protocol is up
30 second input rate 5413000 bits/sec, 1029 packets/sec
30 second output rate 2440000 bits/sec, 1340 packets/sec
FastEthernet2/0 is up, line protocol is up
5 minute input rate 2250000 bits/sec, 379 packets/sec
5 minute output rate 4000 bits/sec, 7 packets/sec
bala-choke-1#sh int stat
Load for five secs: 10%/6%; one minute: 17%; five minutes: 18%
Time source is NTP, 09:17:54.241 EST Thu Jan 27 2005
FastEthernet0/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 19640 1331558 19330 3660910
Route cache 90014 22901439 96587 75927787
Total 109654 24232997 115917 79588697
FastEthernet1/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 12372 2158585 18889 1202885
Route cache 68798 49543259 89661 22469401
Total 81170 51701844 108550 23672286
FastEthernet2/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 6499 1398433 341 23855
Route cache 27813 25999911 353 71982
Total 34312 27398344 694 95837
On Thu, 27 Jan 2005, Dave Temkin wrote:
> Oddly enough - it's all HTTP traffic - and it seems perfectly normal..
> And it's not going to one of the NAT'ed addresses.
>
> bala-choke-1#show buff input-interface f2/0 packet
>
> Public particle pools:
>
> Private particle pools:
>
> bala-choke-1#show buff input-interface f2/0 packet
>
> Buffer information for Small buffer at 0x63FCBB14
> data_area 0xF25E824, refcount 1, next 0x63FC7374, flags 0x280
> linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
> if_input 0x638875AC (FastEthernet2/0), if_output 0x0 (None)
> inputtime 31w5d (elapsed 00:00:04.648)
> outputtime 00:00:00.000 (elapsed never), oqnumber 65535
> datagramstart 0xF25E86A, datagramsize 60, maximum size 260
> mac_start 0xF25E86A, addr_start 0xF25E86A, info_start 0x0
> network_start 0xF25E878, transport_start 0xF25E88C, caller_pc 0x608957C4
>
> source: 166.90.137.22, destination: y.y.y.y, id: 0x1C71, ttl:
> 56,
> TOS: 0 prot: 6, source port 80, destination port 47965
>
> 0F25E860: 0000 0C07AC66 ....,f
> 0F25E870: 00021630 A5810800 45000028 1C714000 ...0%...E..(.q at .
> 0F25E880: 380603B6 A65A8916 8DA26596 0050BB5D 8..6&Z..."e..P;]
> 0F25E890: 0EDE0396 BA2C0A36 50111B48 DF5D0000 .^..:,.6P..H_]..
> 0F25E8A0: 00000000 000005 .......
>
>
> bala-choke-1#show buff input-interface f2/0 packet
>
> Buffer information for Big buffer at 0x6420A0CC
> data_area 0xF292BC4, refcount 1, next 0x0, flags 0x280
> linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
> if_input 0x638875AC (FastEthernet2/0), if_output 0x0 (None)
> inputtime 00:00:00.000 (elapsed never)
> outputtime 00:00:00.000 (elapsed never), oqnumber 65535
> datagramstart 0xF292C0A, datagramsize 1253, maximum size 1692
> mac_start 0xF292C0A, addr_start 0xF292C0A, info_start 0x0
> network_start 0xF292C18, transport_start 0xF292C2C, caller_pc 0x608957C4
>
> source: 63.211.153.3, destination: y.y.y.x, id: 0x66D2, ttl:
> 119,
> TOS: 0 prot: 6, source port 80, destination port 61477
>
> 0F292C00: 0000 0C07AC66 ....,f
> 0F292C10: 00021630 A5810800 450004D7 66D24000 ...0%...E..WfR at .
>
>
>
> -Dave
>
> On Thu, 27 Jan 2005, Rodney Dunn wrote:
>
> > Tidbit: When you are collecting command outputs to
> > help troubleshoot enable this for your terminal
> > session:
> >
> > terminal exec prompt timestamp
> >
> > Then do:
> >
> > clear counters
> > sh int stat
> > sh int | incl protocol|bits/sec
> >
> > wait 30 seconds
> >
> > clear counters
> > sh int stat
> > sh int | incl protocol|bits/sec
> >
> >
> > what percentage of the traffic is being
> > punted to process level?
> >
> > Run the 'sh buff' commands like I gave you
> > and let's see a few of the packets.
> >
> > Rodney
> >
> >
> >
> >
> > tOn Thu, Jan 27, 2005 at 08:58:05AM -0500, Dave Temkin wrote:
> > > Thanks Rodney.
> > >
> > > The one thing I'm hesitant to blame it on is the fact that the actual
> > > NAT'ed traffic is very very little (it's AIM conversations, that's it).
> > > So I'm wondering why on a box that's as big as this one (NPE-400) it'd
> > > choke on that...
> > >
> > > -Dave
> > >
> > > --
> > > David Temkin
> > >
> > > On Thu, 27 Jan 2005, Rodney Dunn wrote:
> > >
> > > > Your problem is almost surely that the packets
> > > > being punted are TCP control packets where
> > > > we punt to create/tear down the translations.
> > > > SYN, FIN, RST.
> > > >
> > > > If you want to see the packets at process
> > > > level you can either turn on:
> > > > debug ip packet <acl> to limit the granularity of
> > > > the debug since that only prints packets at process
> > > > level. /*not true for 12.2S with the right commands*/
> > > >
> > > > You can also do "sh buffers input-interface <name> packet"
> > > > a few times and catch some packets in the buffer and
> > > > manually decode the TCP header to see if the flags are set
> > > > in the header.
> > > >
> > > > Now, in 12.3(4)T and later we made some NAT enhancements
> > > > where we create the flows in the CEF path without punting
> > > > traffic. That is the suggested way to go if you are seeing
> > > > a high number of process switched traffic with NAT enabled.
> > > >
> > > > Rodney
> > > >
> > > > On Thu, Jan 27, 2005 at 07:42:33AM -0500, Dave Temkin wrote:
> > > > > I've got an internet-facing router that's seeing a very high rate of
> > > > > process switched traffic. Nothing too crazy is configured on this router
> > > > > - a little bit of NAT, a couple of route maps, BGP. That's about it.
> > > > > Aside from doing a debug ip packet and killing the router (it's passing
> > > > > about 30-40mbit of traffic), are there any other options for tracking down
> > > > > what's in the process queue? Router is running 12.3.6a
> > > > >
> > > > > FastEthernet0/0
> > > > > Throttle count 4
> > > > > Drops RP 5 SP 0
> > > > > SPD Flushes Fast 3103 SSE 0
> > > > > SPD Aggress Fast 0
> > > > > SPD Priority Inputs 83215964 Drops 0
> > > > >
> > > > > Protocol IP
> > > > > Switching path Pkts In Chars In Pkts Out Chars Out
> > > > > Process 1803602701 4025634609 1661069368 456573125
> > > > > Cache misses 0 - - -
> > > > > Fast 2713542052 1802705001 3837108389 304620460
> > > > > Auton/SSE 0 0 0 0
> > > > >
> > > > >
> > > > > FastEthernet1/0 Outside
> > > > > Throttle count 0
> > > > > Drops RP 0 SP 0
> > > > > SPD Flushes Fast 1796 SSE 0
> > > > > SPD Aggress Fast 0
> > > > > SPD Priority Inputs 6927146 Drops 0
> > > > >
> > > > > Protocol IP
> > > > > Switching path Pkts In Chars In Pkts Out Chars Out
> > > > > Process 543622379 2397426796 317919218 1743487367
> > > > > Cache misses 0 - - -
> > > > > Fast 3071349692 1923264716 1211037578 2505497398
> > > > > Auton/SSE 0 0 0 0
> > > > >
> > > > >
> > > > > FastEthernet2/0 Outside 2
> > > > > Throttle count 0
> > > > > Drops RP 0 SP 0
> > > > > SPD Flushes Fast 1480 SSE 0
> > > > > SPD Aggress Fast 0
> > > > > SPD Priority Inputs 42435822 Drops 0
> > > > >
> > > > > Protocol IP
> > > > > Switching path Pkts In Chars In Pkts Out Chars Out
> > > > > Process 1056561152 1934756549 1414955036 1939153311
> > > > > Cache misses 0 - - -
> > > > > Fast 819752907 318162363 1502399074 1302221329
> > > > > Auton/SSE 0 0 0 0
> > > > >
> > > > >
> > > > > !
> > > > > interface FastEthernet0/0.101
> > > > > encapsulation dot1Q 101
> > > > > ip address x.x.x.x x.x.x.x.x
> > > > > no ip redirects
> > > > > no ip proxy-arp
> > > > > ip nat inside
> > > > > ip policy route-map RM101
> > > > > no cdp enable
> > > > > standby 101 ip x.x.x.y
> > > > > standby 101 timers 1 3
> > > > > standby 101 priority 250
> > > > > standby 101 preempt
> > > > > standby 101 name HSRP101
> > > > > !
> > > > >
> > > > > !
> > > > > interface FastEthernet1/0
> > > > > description Outside 1
> > > > > ip address x.x.x.x x.x.x.x
> > > > > ip access-group Yipes-Outside in
> > > > > ip nat outside
> > > > > load-interval 30
> > > > > duplex full
> > > > > ntp disable
> > > > > hold-queue 300 in
> > > > > hold-queue 300 out
> > > > >
> > > > >
> > > > > -Dave
> > > > > _______________________________________________
> > > > > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > > > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > > > archive at http://puck.nether.net/pipermail/cisco-nsp/
> > > >
> >
>
More information about the cisco-nsp
mailing list