[c-nsp] High interrupt load on 7507
Rodney Dunn
rodunn at cisco.com
Wed Jun 21 13:48:41 EDT 2006
>
> > Post a 'sh cef line', 'sh int stat' after you clear the counters
> > and let it run for 2 minutes.
>
> Slot 1 has cDS-3s on it, which I excluded from this e-mail since I'm not
> concerned with them right now.
>
> The issue this morning was traffic from BVI212 into Fa6/0/0. I took out
> the BVI and just went with a straight sub-if, but the load was
> identical. I know BVIs are not dCEF switched, although I'm not seeing a
> performance difference between using straight sub-ifs or BVIs. One
> curious thing is that Fa 6/0/0 used to be Gi4/0/0.100, but the CPU was
> running 70% just for interrupting when it wasn't doing a whole lot.
> Moving it to Fa 6/0/0 dropped it to about 25% (e.g. from one 5min
> interval to the next, moving from the sub-if to Fa 6/0/0 and maintaining
> the same traffic level the CPU load was very different). That's why I'm
> thinking the sub-if stuff is not being dCEF switched.
I'm 99% positive dot1q will be dCEF switched.
BVI performance is horrible and surely not dCEF switched. I strongly
don't recommend it on a 75xx at all.
>
> core1#show cef line
> Slot MsgSent XDRSent Window LowQ MedQ HighQ Flags
> 6 2783055 7810106 221 0 0 0 up
> 0 2783052 7810103 221 0 0 0 up
> 4 2783057 7810057 221 0 0 0 up
> 1 2783051 7809965 221 0 0 0 up
>
> VRF Default-table, version 1784573, 116855 routes
> Slot Version CEF-XDR I/Fs State Flags
> 6 1784573 2284901 24 Active sync, table-up
> 0 1784573 2284901 45 Active sync, table-up
> 4 1784573 2284901 45 Active sync, table-up
> 1 1784573 2284901 48 Active sync, table-up
> core1#
Ok..that tells you dCEF is up to all the slots which is good.
>
> core1#sh int st
> GigabitEthernet0/0/0
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 4676 405918 307 21084
> Route cache 302379 328418662 4957 692518
> Distributed cache 0 0 0 0
> Total 307055 328824580 5264 713602
So we know you are not dCEF switching any traffic in or out of that
slot so features must be punting.
Looking at it ALL of the subints have bridge-group so all traffic
in and out of that slot will be punted to the RSP.
> GigabitEthernet4/0/0
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 42395 28657559 3264 249259
> Route cache 530584 466662433 388103 161035767
> Distributed cache 219315 167903831 697578 165047429
So some traffic coming in and out of 4/0/0 has a lot that is dCEF switched.
If you look at all those subints you have some that don't have a bridge-group
so therefore that traffic to/from those subints are dCEF switched.
The ones with bridge-group are punted.
> Total 792294 663223823 1088945 326332455
> FastEthernet6/0/0
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 2716 331015 1911 265564
> Route cache 121846 12634388 150584 152442701
> Distributed cache 1006816 229643636 761018 222849022
> Total 1131378 242609039 913513 375557287
You are dCEF switching some to/from this one.
> interface FastEthernet6/0/0
> ip address 207.166.193.2 255.255.255.0 secondary
> ip address 207.166.219.161 255.255.255.248 secondary
> ip address 207.166.192.2 255.255.255.0
> no ip redirects
> no ip proxy-arp
> ip route-cache same-interface
> ip ospf cost 20
> ip ospf priority 60
> ip policy route-map Filterz
> full-duplex
> no cdp enable
> standby 10 ip 207.166.192.1
> standby 10 priority 150
> standby 10 preempt
> standby 100 ip 207.166.193.1
> standby 100 priority 110
> standby 100 preempt
there is nothing there to prevent dCEF switching. However, if the interface
the traffic is going to or coming from doesn't support dCEF (ie: a bridge-group
interface you have to punt).
> Serial6/1/0
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 707 103637 594 35798
> Route cache 263761 122891945 153301 132731683
> Distributed cache 947156 235143260 989048 375224824
> Total 1211624 358138842 1142943 507992305
Same for this interface:
> interface Serial6/1/0
> description DS3 to Cogent (Order #1-33ZTH)
> ip address 216.28.126.134 255.255.255.252
> ip access-group as2828-inbound in
> no ip redirects
> dsu bandwidth 44210
> scramble
> framing c-bit
> cablelength 100
> crc 32
> serial restart-delay 0
only thing here is the named ACL. I *think* we support that in the dCEF
path on this code.
What you could do is turn on netflow "ip route-cache flow" and look
at the flows on the RSP "sh ip cache flow". Those will only be the flows
on the RSP that are punted up to it so you can figure out the ingress
and egress packet paths.
Then if-con to the vip or do execute-on slot X sh ip cahce flow and see
what flows show up on the VIP (dCEF switched).
I don't know your main packet path but if it's through those BVI's
you will punt and the switching performance will be pretty bad.
I don't recommend that.
> Interface Serial6/1/1 is disabled
>
> BVI103
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 2 303 2 120
> Route cache 12842 17236619 7470 570007
> Distributed cache 0 0 0 0
> Total 12844 17236922 7472 570127
> BVI120
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 92 6220 66 4092
> Route cache 3942 3748701 3019 354752
> Distributed cache 0 0 0 0
> Total 4034 3754921 3085 358844
> BVI121
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 79 4870 69 4272
> Route cache 0 0 0 0
> Distributed cache 0 0 0 0
> Total 79 4870 69 4272
> BVI122
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 76 4690 70 4332
> Route cache 3996 2848363 2010 373504
> Distributed cache 0 0 0 0
> Total 4072 2853053 2080 377836
> BVI123
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 76 4690 70 4332
> Route cache 2531 2815804 1209 240225
> Distributed cache 0 0 0 0
> Total 2607 2820494 1279 244557
> BVI124
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 77 4750 71 4392
> Route cache 94526 125027678 39592 4213275
> Distributed cache 0 0 0 0
> Total 94603 125032428 39663 4217667
> BVI126
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 79 6352 72 4512
> Route cache 30288 3110109 22601 1790170
> Distributed cache 0 0 0 0
> Total 30367 3116461 22673 1794682
> BVI127
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 88 5946 82 5212
> Route cache 31143 30945333 12882 2132586
> Distributed cache 0 0 0 0
> Total 31231 30951279 12964 2137798
> BVI128
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 74 4570 68 4212
> Route cache 150501 156326432 122319 11926416
> Distributed cache 0 0 0 0
> Total 150575 156331002 122387 11930628
> BVI201
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 369 22302 86 5332
> Route cache 36505 17024380 30064 7530296
> Distributed cache 0 0 0 0
> Total 36874 17046682 30150 7535628
> BVI203
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 15 942 34 2670
> Route cache 5920 3627125 2863 418317
> Distributed cache 0 0 0 0
> Total 5935 3628067 2897 420987
> BVI204
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 0 0 20 1800
> Route cache 5 490 5 490
> Distributed cache 0 0 0 0
> Total 5 490 25 2290
> BVI205
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 0 0 24 2040
> Route cache 5131 671374 6900 6189797
> Distributed cache 0 0 0 0
> Total 5131 671374 6924 6191837
> BVI206
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 3 437 21 1860
> Route cache 15786 6699537 23922 25091706
> Distributed cache 0 0 0 0
> Total 15789 6699974 23943 25093566
> BVI208
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 3 180 23 1980
> Route cache 3626 679404 2962 1430542
> Distributed cache 0 0 0 0
> Total 3629 679584 2985 1432522
> BVI209
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 1 60 24 2040
> Route cache 1587 201217 1709 1118831
> Distributed cache 0 0 0 0
> Total 1588 201277 1733 1120871
> BVI211
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 3 270 23 2070
> Route cache 409 29671 396 32401
> Distributed cache 0 0 0 0
> Total 412 29941 419 34471
> BVI212
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 77 4750 91 6192
> Route cache 14258 12254718 4949 648485
> Distributed cache 0 0 0 0
> Total 14335 12259468 5040 654677
> Loopback0
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 0 0 0 0
> Route cache 0 0 0 0
> Distributed cache 0 0 0 0
> Total 0 0 0 0
> Loopback1
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 0 0 0 0
> Route cache 0 0 0 0
> Distributed cache 0 0 0 0
>
> > 'sh run' so we can see the features you have enabled on the ingress
> > egress interfaces.
>
>
>
> bridge irb
> !
> !
> interface Loopback0
> ip address 207.166.219.1 255.255.255.255
> !
> interface Loopback1
> ip address 207.166.219.5 255.255.255.255
> !
> interface GigabitEthernet0/0/0
> no ip address
> load-interval 30
> negotiation auto
> hold-queue 2500 in
> !
> interface GigabitEthernet0/0/0.100
> encapsulation dot1Q 100
> no ip redirects
> no ip proxy-arp
> no cdp enable
> bridge-group 100
> !
> interface GigabitEthernet0/0/0.101
> encapsulation dot1Q 101
> no cdp enable
> !
> interface GigabitEthernet0/0/0.103
> encapsulation dot1Q 103
> no cdp enable
> bridge-group 103
> !
> interface GigabitEthernet0/0/0.120
> encapsulation dot1Q 120
> no cdp enable
> bridge-group 120
> !
> interface GigabitEthernet0/0/0.121
> encapsulation dot1Q 121
> no cdp enable
> bridge-group 121
> !
> interface GigabitEthernet0/0/0.122
> encapsulation dot1Q 122
> no cdp enable
> bridge-group 122
> !
> interface GigabitEthernet0/0/0.123
> encapsulation dot1Q 123
> no cdp enable
> bridge-group 123
> !
> interface GigabitEthernet0/0/0.124
> encapsulation dot1Q 124
> no cdp enable
> bridge-group 124
> !
> interface GigabitEthernet0/0/0.126
> encapsulation dot1Q 126
> no cdp enable
> bridge-group 126
> !
> interface GigabitEthernet0/0/0.127
> encapsulation dot1Q 127
> no cdp enable
> bridge-group 127
> !
> interface GigabitEthernet0/0/0.128
> encapsulation dot1Q 128
> no cdp enable
> bridge-group 128
> !
> interface GigabitEthernet0/0/0.129
> encapsulation dot1Q 129
> no ip redirects
> shutdown
> no cdp enable
> bridge-group 129
> !
> interface GigabitEthernet0/0/0.131
> encapsulation dot1Q 131
> no cdp enable
> bridge-group 131
> bridge-group 131 spanning-disabled
> !
> interface GigabitEthernet0/0/0.132
> encapsulation dot1Q 132
> no cdp enable
> bridge-group 132
> bridge-group 132 spanning-disabled
> !
> interface GigabitEthernet0/0/0.201
> encapsulation dot1Q 201
> no cdp enable
> bridge-group 201
> !
> interface GigabitEthernet0/0/0.203
> encapsulation dot1Q 203
> no cdp enable
> bridge-group 203
> !
> interface GigabitEthernet0/0/0.204
> encapsulation dot1Q 204
> no cdp enable
> bridge-group 204
> !
> interface GigabitEthernet0/0/0.205
> encapsulation dot1Q 205
> no cdp enable
> bridge-group 205
> !
> interface GigabitEthernet0/0/0.206
> encapsulation dot1Q 206
> no cdp enable
> bridge-group 206
> !
> interface GigabitEthernet0/0/0.208
> encapsulation dot1Q 208
> no cdp enable
> bridge-group 208
> !
> interface GigabitEthernet0/0/0.209
> encapsulation dot1Q 209
> no cdp enable
> bridge-group 209
> !
> interface GigabitEthernet0/0/0.211
> encapsulation dot1Q 211
> no cdp enable
> bridge-group 211
> !
> interface GigabitEthernet0/0/0.212
> encapsulation dot1Q 212
> no cdp enable
> bridge-group 212
> !
> interface GigabitEthernet4/0/0
> no ip address
> load-interval 30
> negotiation auto
> hold-queue 2500 in
> !
> interface GigabitEthernet4/0/0.101
> encapsulation dot1Q 101
> !
> interface GigabitEthernet4/0/0.103
> encapsulation dot1Q 103
> bridge-group 103
> bridge-group 103 priority 64
> !
> interface GigabitEthernet4/0/0.120
> encapsulation dot1Q 120
> bridge-group 120
> bridge-group 120 priority 64
> !
> interface GigabitEthernet4/0/0.121
> encapsulation dot1Q 121
> bridge-group 121
> bridge-group 121 priority 64
> !
> interface GigabitEthernet4/0/0.122
> encapsulation dot1Q 122
> bridge-group 122
> bridge-group 122 priority 64
> !
> interface GigabitEthernet4/0/0.123
> encapsulation dot1Q 123
> bridge-group 123
> bridge-group 123 priority 64
> !
> interface GigabitEthernet4/0/0.124
> encapsulation dot1Q 124
> bridge-group 124
> bridge-group 124 priority 64
> !
> interface GigabitEthernet4/0/0.126
> encapsulation dot1Q 126
> bridge-group 126
> bridge-group 126 priority 64
> !
> interface GigabitEthernet4/0/0.127
> encapsulation dot1Q 127
> bridge-group 127
> bridge-group 127 priority 64
> !
> interface GigabitEthernet4/0/0.128
> encapsulation dot1Q 128
> bridge-group 128
> bridge-group 128 priority 64
> !
> interface GigabitEthernet4/0/0.129
> encapsulation dot1Q 129
> ip address 207.166.219.90 255.255.255.248
> no ip redirects
> no cdp enable
> standby 129 ip 207.166.219.89
> standby 129 priority 110
> standby 129 preempt
> !
> interface GigabitEthernet4/0/0.131
> encapsulation dot1Q 131
> bridge-group 131
> bridge-group 131 priority 64
> bridge-group 131 spanning-disabled
> !
> interface GigabitEthernet4/0/0.132
> encapsulation dot1Q 132
> bridge-group 132
> bridge-group 132 priority 64
> bridge-group 132 spanning-disabled
> !
> interface GigabitEthernet4/0/0.201
> encapsulation dot1Q 201
> bridge-group 201
> bridge-group 201 priority 64
> !
> interface GigabitEthernet4/0/0.203
> encapsulation dot1Q 203
> bridge-group 203
> bridge-group 203 priority 64
> !
> interface GigabitEthernet4/0/0.204
> encapsulation dot1Q 204
> bridge-group 204
> bridge-group 204 priority 64
> !
> interface GigabitEthernet4/0/0.205
> encapsulation dot1Q 205
> no cdp enable
> bridge-group 205
> bridge-group 205 priority 64
> !
> interface GigabitEthernet4/0/0.206
> encapsulation dot1Q 206
> bridge-group 206
> bridge-group 206 priority 64
> !
> interface GigabitEthernet4/0/0.208
> encapsulation dot1Q 208
> bridge-group 208
> bridge-group 208 priority 64
> !
> interface GigabitEthernet4/0/0.209
> encapsulation dot1Q 209
> bridge-group 209
> bridge-group 209 priority 64
> !
> interface GigabitEthernet4/0/0.211
> encapsulation dot1Q 211
> bridge-group 211
> bridge-group 211 priority 64
> !
> interface GigabitEthernet4/0/0.212
> encapsulation dot1Q 212
> bridge-group 212
> bridge-group 212 priority 64
> !
> interface GigabitEthernet4/0/0.400
> encapsulation dot1Q 400
> ip address 207.166.219.9 255.255.255.252
> ip ospf cost 1
> no cdp enable
> !
> interface FastEthernet6/0/0
> ip address 207.166.193.2 255.255.255.0 secondary
> ip address 207.166.219.161 255.255.255.248 secondary
> ip address 207.166.192.2 255.255.255.0
> no ip redirects
> no ip proxy-arp
> ip route-cache same-interface
> ip ospf cost 20
> ip ospf priority 60
> ip policy route-map Filterz
> full-duplex
> no cdp enable
> standby 10 ip 207.166.192.1
> standby 10 priority 150
> standby 10 preempt
> standby 100 ip 207.166.193.1
> standby 100 priority 110
> standby 100 preempt
> !
> interface Serial6/1/0
> description DS3 to Cogent (Order #1-33ZTH)
> ip address 216.28.126.134 255.255.255.252
> ip access-group as2828-inbound in
> no ip redirects
> dsu bandwidth 44210
> scramble
> framing c-bit
> cablelength 100
> crc 32
> serial restart-delay 0
> !
> interface Serial6/1/1
> no ip address
> shutdown
> dsu bandwidth 44210
> framing c-bit
> cablelength 10
> serial restart-delay 0
> !
> interface BVI103
> description VLAN 103 - N2Net Colocation 2
> ip address 207.166.222.129 255.255.255.192
> no ip redirects
> rate-limit input access-group 17 384000 65536 65536 conform-action
> transmit exceed-action drop
> rate-limit input access-group 18 1048000 65536 65536 conform-action
> transmit exceed-action drop
> rate-limit input access-group 22 1048000 65536 65536 conform-action
> transmit exceed-action drop
> rate-limit output access-group 17 384000 65536 65536 conform-action
> transmit exceed-action drop
> rate-limit output access-group 18 1048000 65536 65536 conform-action
> transmit exceed-action drop
> rate-limit output access-group 22 1048000 65536 65536 conform-action
> transmit exceed-action drop
> !
> interface BVI120
> description VLAN interface to Health Journeys transit
> ip address 207.166.219.53 255.255.255.252 secondary
> ip address 207.166.219.146 255.255.255.248
> no ip redirects
> standby 120 ip 207.166.219.145
> standby 120 priority 110
> standby 120 preempt
> !
> interface BVI121
> description VLAN 121 to Euro Adpot Agency transit
> ip address 207.166.219.154 255.255.255.248
> no ip redirects
> rate-limit input 10480000 131072 131072 conform-action transmit
> exceed-action drop
> rate-limit output 10480000 131072 131072 conform-action transmit
> exceed-action drop
> standby 121 ip 207.166.219.153
> standby 121 priority 110
> standby 121 preempt
> !
> interface BVI122
> description VLAN 122 to ICG Solutions transit
> ip address 207.166.219.98 255.255.255.248
> no ip redirects
> rate-limit input 1048000 65536 65536 conform-action transmit
> exceed-action drop
> rate-limit output 1048000 65536 65536 conform-action transmit
> exceed-action drop
> standby 122 ip 207.166.219.97
> standby 122 priority 110
> standby 122 preempt
> !
> interface BVI123
> description VLAN 123 to Rainbow Babies transit
> ip address 207.166.219.130 255.255.255.248
> no ip redirects
> rate-limit input 1048000 65536 65536 conform-action transmit
> exceed-action drop
> rate-limit output 1048000 65536 65536 conform-action transmit
> exceed-action drop
> standby 123 ip 207.166.219.129
> standby 123 priority 110
> standby 123 preempt
> !
> interface BVI124
> description VLAN 124 - Transit to Cox Consulting
> ip address 207.166.219.61 255.255.255.252 secondary
> ip address 207.166.219.138 255.255.255.248
> no ip redirects
> standby 124 ip 207.166.219.137
> standby 124 priority 110
> standby 124 preempt
> !
> interface BVI126
> description VLAN 126 - Corona Colo
> ip address 207.166.219.34 255.255.255.248
> no ip redirects
> ip route-cache same-interface
> ip route-cache flow
> standby 126 ip 207.166.219.33
> standby 126 priority 110
> standby 126 preempt
> !
> interface BVI127
> description VLAN 127 - Fathom IT Colo
> ip address 207.166.219.42 255.255.255.248
> ip access-group vlan127egress in
> no ip redirects
> standby 127 ip 207.166.219.41
> standby 127 priority 110
> standby 127 preempt
> !
> interface BVI128
> ip address 207.166.219.82 255.255.255.248
> no ip redirects
> standby 128 ip 207.166.219.81
> standby 128 priority 110
> standby 128 preempt
> !
> interface BVI201
> description VLAN 201 - Mat Kovach
> ip address 207.166.219.114 255.255.255.248 secondary
> ip address 207.166.200.193 255.255.255.192
> no ip redirects
> standby 201 ip 207.166.219.113
> standby 201 priority 110
> standby 201 preempt
> !
> interface BVI203
> description VLAN 203 - Playbook
> ip address 207.166.200.1 255.255.255.224 secondary
> ip address 207.166.200.5 255.255.255.224
> no ip redirects
> !
> interface BVI204
> description VLAN 204 - Allwork
> ip address 207.166.197.17 255.255.255.252
> no ip redirects
> !
> interface BVI205
> ip address 207.166.196.193 255.255.255.240
> !
> interface BVI206
> description VLAN 206 - Cornerstone
> ip address 207.166.218.193 255.255.255.192
> no ip redirects
> rate-limit input 1048000 65536 65536 conform-action transmit
> exceed-action drop
> rate-limit output 1048000 65536 65536 conform-action transmit
> exceed-action drop
> !
> interface BVI208
> description VLAN 208 - Fast Ethernet to CoDFA 13th fl
> ip address 207.166.219.73 255.255.255.252
> no ip redirects
> !
> interface BVI209
> description VLAN 209 - Buckley King
> ip address 207.166.196.65 255.255.255.248
> no ip redirects
> !
> interface BVI211
> ip address 207.166.198.41 255.255.255.248
> no ip redirects
> !
> interface BVI212
> ip address 207.166.197.170 255.255.255.248
> no ip redirects
> standby 212 ip 207.166.197.169
> standby 212 priority 110
> standby 212 preempt
>
>
> David
More information about the cisco-nsp
mailing list