[c-nsp] troubleshooting SVI input drops on MSFC3
Sukumar Subburayan
sukumars at cisco.com
Wed May 9 19:18:39 EDT 2007
Also, I am not sure how the original poster got into this situation.
When user configures secondaries, we automatically disable 'ip redirects'
on the interface. Eg:
2-2-mid-720(config-if)#ip add 7.7.7.7 255.255.255.0
2-2-mid-720(config-if)#ip add 8.8.8.8 255.255.255.0 sec
ICMP redirect is disabled on interfaces with secondary IP address
2-2-mid-720#show run int vlan 2
Building configuration...
Current configuration : 118 bytes
!
interface Vlan2
ip address 8.8.8.8 255.255.255.0 secondary
ip address 7.7.7.7 255.255.255.0
no ip redirects
end
Was this working before? Maybe, someone went and manually enabled 'ip
redirects' again and that caused the sudden CPU spike?
sukumar
On Wed, 9 May 2007, Sukumar Subburayan wrote:
> The decision to redirect the packet for ICMP redirect is done using a bit in
> the PFC HW and it is a per-vlan bit, and not per flow. This bit can either be
> on or off and is controlled by having '<no> ip redirects'. That is why you
> are seeing all traffic in the vlan getting punted to software, if we don't
> have 'no ip redirects' configured.
>
> sukumar
>
>
>
> On Wed, 9 May 2007, Church, Charles wrote:
>
>> I don't think the PFC has any logic in it to provide ICMP responses for
>> any of the message types. Someone from Cisco would know for sure
>> though. Without the ability for the PFC to respond, those packets would
>> get forwarded to the MSFC. It seems that it should only do it once per
>> flow, but it sounds like it was doing it for all the traffic, since your
>> CPU went down so dramatically.
>>
>>
>> Chuck Church
>> Multimax Principal Network Engineer, CCIE #8776
>> EDS Contractor, Multimax - Navy Marine Corps Intranet (NMCI)
>> 1210 N. Parker Rd. | Greenville, SC 29609
>> Office: 864-335-9473 | Cell: 864-266-3978
>> cchurch at multimax.com
>>
>> -----Original Message-----
>> From: cisco-nsp-bounces at puck.nether.net
>> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of barney gumbo
>> Sent: Wednesday, May 09, 2007 5:39 PM
>> To: cisco-nsp at puck.nether.net
>> Subject: Re: [c-nsp] troubleshooting SVI input drops on MSFC3
>>
>> I disabled IP redirects on the SVI and traffic is no longer seen on the
>> SVI interface, no longer seeing SVI input drops, and CPU returned to
>> normal.
>>
>> Why does the router sending an IP redirect cause traffic to hit the
>> MSFC?
>>
>> Or, does the traffic hit the MSFC because the CEF adjacency is the same
>> interface for all prefixes, i.e. router-on-a-stick?
>>
>> I've read references to both as being the reason one would see high CPU
>> on an MSFC along with SVI input drops, just trying to understand the
>> hardware better. It seems the magic number is around 40-50 Mbps which
>> resulted in 75% CPU usage and input drops. That just seems low for such
>> powerful hardware...
>>
>>
>> On 5/9/07, barney gumbo <barney.gumbo at gmail.com> wrote:
>>>
>>> I see ICMP redirects (which is not disabled on that SVI) and they are
>>> incrementing. I can understand high CPU as a result of the router
>>> being forced to trx alot of ICMP redirects, however that (the MSFC
>>> sending ICMP
>>> redirects) doesnt cause the traffic to be process-switched, does it?
>>>
>>> ICMP statistics:
>>> Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 318
>> unreachable
>>> 10817442 echo, 3142088 echo reply, 0 mask requests, 0 mask
>>> replies, 0 quench
>>> 0 parameter, 0 timestamp, 0 info request, 0 other
>>> 0 irdp solicitations, 0 irdp advertisements
>>> Sent: 115887 redirects, 2262 unreachable, 3144305 echo, 10817442
>>> echo reply
>>> 0 mask requests, 0 mask replies, 0 quench, 0 timestamp
>>> 0 info reply, 1740292 time exceeded, 0 parameter problem
>>> 0 irdp solicitations, 0 irdp advertisements
>>>
>>>
>>> ICMP statistics:
>>> Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 318
>> unreachable
>>> 10820192 echo, 3142088 echo reply, 0 mask requests, 0 mask
>>> replies, 0 quench
>>> 0 parameter, 0 timestamp, 0 info request, 0 other
>>> 0 irdp solicitations, 0 irdp advertisements
>>> Sent: 115939 redirects, 2262 unreachable, 3144305 echo, 10820192
>>> echo reply
>>> 0 mask requests, 0 mask replies, 0 quench, 0 timestamp
>>> 0 info reply, 1740292 time exceeded, 0 parameter problem
>>> 0 irdp solicitations, 0 irdp advertisements
>>>
>>>
>>> On 5/9/07, Church, Charles <cchurch at multimax.com> wrote:
>>>>
>>>> Sending ICMP redirects for all those?
>>>>
>>>> Chuck
>>>>
>>>> --- Original Message ---
>>>> From:"barney gumbo" <barney.gumbo at gmail.com> Sent:Wed 5/9/07 12:15
>>>> pm To:"Dale W. Carder" <dwcarder at doit.wisc.edu>
>>>> Cc:"cisco-nsp at puck.nether.net" < cisco-nsp at puck.nether.net>
>>>> Subj:Re: [c-nsp] troubleshooting SVI input drops on MSFC3
>>>>
>>>> I have a little more info this time..
>>>>
>>>> It appears that the traffic is being process switched.
>>>>
>>>> It started when this switch became more of a "router on a stick".
>>>> Previously most traffic flowed from one SVI to the other; this 6503
>>>> is essentially an access or WAN router. Now traffic flows from
>>>> routers on the same VLAN into the SVI and then back out of the SVI
>>>> to get to the next hop, which is a firewall, all on the same VLAN.
>>>>
>>>> It seems like once the flow volume hits around 40 Mbps, the input
>>>> drops begin.
>>>>
>>>> Going away from the router-on-a-stick design, where the bulk of the
>>>> traffic transit's across the switch instead of in/out the same
>>>> interface, is not a trivial change, so I would like to try and get
>>>> some confidence that this is related to the problem before I start
>>>> to make changes. This is all just a guess, but it's the only major
>>>> thing that has changed in the last week.
>>>>
>>>> Any ideas if this could be my cause of process-switchng and input
>> drops?
>>>>
>>>> Any ideas on how I can verify the router-on-a-stick forwarding is
>>>> definitely to blame?
>>>>
>>>> On 5/9/07, Dale W. Carder <dwcarder at doit.wisc.edu> wrote:
>>>>>
>>>>>
>>>>> Here's some commands to get you started:
>>>>>
>>>>> sh buffers input-interface
>>>>> sh int vlan1234 switching
>>>>> sh ip interface
>>>>> sh ip traffic
>>>>> sh cef drop
>>>>> sh ip cache flow
>>>>> sh cef not-cef-switched
>>>>>
>>>>> Some more help can be found here:
>>>>> http://www.cisco.com/warp/public/63/queue_drops.html
>>>>>
>>>>> You also might want to verify that you didn't configure a feature
>>>>> that causes punts.
>>>>>
>>>>> If you really want to get dirty, you can create a span session to
>>>>> monitor traffic destined to the RP. This has been discussed on
>>>>> this list once or twice, but it is a bit messy.
>>>>>
>>>>> Dale
>>>>>
>>>>>
>>>>> On May 9, 2007, at 9:43 AM, barney gumbo wrote:
>>>>>
>>>>>> I am seeing high input interface drops on an SVI interface on an
>>
>>>>>> MSFC3. The
>>>>>> MSFC3 is installed in a 6503 chassis with Sup720. The switch is
>>
>>>>>> running hybird mode.
>>>>>>
>>>>>> The traffic load has increased, and CPU is running high when the
>>
>>>>>> traffic load increases. I don't know why the SVI is showing
>>>>>> increased traffic load because normally I don't see traffic
>>>>>> through the SVI, it all get's
>>>> MLS
>>>>>> switched. Something in the last week has caused traffic to be
>>>>>> switched through the SVI showing the high input drops. The
>>>>>> overal load of traffic which should be routed (MLS switched) via
>>
>>>>>> the interface has not increased or decreased; all of a sudden in
>>
>>>>>> the last week traffic is being
>>>>>> (seemingly)
>>>>>> process switched through this SVI.
>>>>>>
>>>>>> Where do I begin troubleshooting high interface drops on an SVI?
>>>>>
>>>> _______________________________________________
>>>> cisco-nsp mailing list cisco-nsp at puck.nether.net
>>>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>>>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>>>
>>>
>>>
>> _______________________________________________
>> cisco-nsp mailing list cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>
>> _______________________________________________
>> cisco-nsp mailing list cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>
>
More information about the cisco-nsp
mailing list