[c-nsp] ASR1K forwarding failures on 10G SPA's

[Mark Tees]

That sounds like what I experienced in ASR920 land recently with bad
packets filling up interface input queues causing a wedge.

When it happens check the interface input queues and save the output.

The resolution for us so far has been tight CoPP with discards, iACLs, and
the like to only allow things towards the boxes that are as trusted as
possible.

On Tuesday, 4 October 2016, Sascha Pollok <sp at iphh.net> wrote:

> Just to make sure: latest IOS XE version? Its not the NTP processing bug
> filling up interface queues? How does the input queue look on the affected
> interfaces?
>
> Cheers
> Sascha
>
>
> Am 4. Oktober 2016 05:33:39 schrieb Stephen Fulton <sf at lists.esoteric.ca>:
>
> ISIS adjacencies drop as well as BGP sessions on neighboring devices drop.
>>
>> Issue just reoccurred.
>>
>> -- Stephen
>>
>> On 2016-10-03 10:59 PM, Scott Granados wrote:
>>
>>> Anything logged while this happens?
>>>
>>> On Oct 3, 2016, at 10:52 PM, Stephen Fulton <sf at lists.esoteric.ca>
>>>> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I have run into a number of forwarding failure events on ASR1K's with
>>>> 10G SPA's.  These have occurred across a range of IOS-XE versions, using
>>>> various ROMMON versions and across two different ASR1K platforms (1002's
>>>> and 1004's).  Multiple SPA's have been replaced, IOS-XE versions and ROMMON
>>>> versions upgraded and in the case of the ASR1004's, SIP's replaced (both
>>>> SIP10 and SIP40's).  TAC cases have been opened several times.
>>>>
>>>> What occurs is forwarding across an interface fails completely.  The
>>>> easiest way to find it is the lack of ARP entries on the
>>>> interface/sub-interface, due to time-outs, but traffic is still attempting
>>>> to traverse the interface.  When I ping the IP address associated with the
>>>> failed interface, it fails.  ARP resolution of any neighbors fails, and
>>>> neighboring devices on the same broadcast domain cannot reach it - though
>>>> will see its MAC in the ARP table.
>>>>
>>>> In all cases, ISIS and MPLS was configured on the interfaces.  BFD has
>>>> been on some, not on others.
>>>>
>>>> I recently found learned of another organization that saw the same
>>>> behavior on an ASR1006 with 10G SPA's.  SPA's and SIP's were replaced and
>>>> the last advice they received from TAC was that if it occurred again the
>>>> chassis would need to replaced.  It did but they chose not to replace the
>>>> chassis and simply stopped using 10G entirely.
>>>>
>>>> Has anyone else seen this?
>>>>
>>>> -- Stephen
>>>>
>>>> _______________________________________________
>>>> cisco-nsp mailing list  cisco-nsp at puck.nether.net
>>>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>>>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>>>
>>>
>>> _______________________________________________
>> cisco-nsp mailing list  cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>
>>
>
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>


-- 
Regards,

Mark L. Tees