[c-nsp] ASR1K forwarding failures on 10G SPA's

[Sascha Pollok]

Exactly. OP might try to raise hold-queue xx in on those interfaces. If it 
solves the problem temporarily (!) he found it.

If so, show buffers input-interfacw should give a hint.
The NTP bug came up pretty recently (2 months or so?) so it could actually 
be the cause.

-Sascha


Am 4. Oktober 2016 07:45:36 schrieb Mark Tees <marktees at gmail.com>:

> That sounds like what I experienced in ASR920 land recently with bad
> packets filling up interface input queues causing a wedge.
>
> When it happens check the interface input queues and save the output.
>
> The resolution for us so far has been tight CoPP with discards, iACLs, and
> the like to only allow things towards the boxes that are as trusted as
> possible.
>
> On Tuesday, 4 October 2016, Sascha Pollok <sp at iphh.net> wrote:
>
>> Just to make sure: latest IOS XE version? Its not the NTP processing bug
>> filling up interface queues? How does the input queue look on the affected
>> interfaces?
>>
>> Cheers
>> Sascha
>>
>>
>> Am 4. Oktober 2016 05:33:39 schrieb Stephen Fulton <sf at lists.esoteric.ca>:
>>
>> ISIS adjacencies drop as well as BGP sessions on neighboring devices drop.
>>>
>>> Issue just reoccurred.
>>>
>>> -- Stephen
>>>
>>> On 2016-10-03 10:59 PM, Scott Granados wrote:
>>>
>>>> Anything logged while this happens?
>>>>
>>>> On Oct 3, 2016, at 10:52 PM, Stephen Fulton <sf at lists.esoteric.ca>
>>>>> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I have run into a number of forwarding failure events on ASR1K's with
>>>>> 10G SPA's.  These have occurred across a range of IOS-XE versions, using
>>>>> various ROMMON versions and across two different ASR1K platforms (1002's
>>>>> and 1004's).  Multiple SPA's have been replaced, IOS-XE versions and ROMMON
>>>>> versions upgraded and in the case of the ASR1004's, SIP's replaced (both
>>>>> SIP10 and SIP40's).  TAC cases have been opened several times.
>>>>>
>>>>> What occurs is forwarding across an interface fails completely.  The
>>>>> easiest way to find it is the lack of ARP entries on the
>>>>> interface/sub-interface, due to time-outs, but traffic is still attempting
>>>>> to traverse the interface.  When I ping the IP address associated with the
>>>>> failed interface, it fails.  ARP resolution of any neighbors fails, and
>>>>> neighboring devices on the same broadcast domain cannot reach it - though
>>>>> will see its MAC in the ARP table.
>>>>>
>>>>> In all cases, ISIS and MPLS was configured on the interfaces.  BFD has
>>>>> been on some, not on others.
>>>>>
>>>>> I recently found learned of another organization that saw the same
>>>>> behavior on an ASR1006 with 10G SPA's.  SPA's and SIP's were replaced and
>>>>> the last advice they received from TAC was that if it occurred again the
>>>>> chassis would need to replaced.  It did but they chose not to replace the
>>>>> chassis and simply stopped using 10G entirely.
>>>>>
>>>>> Has anyone else seen this?
>>>>>
>>>>> -- Stephen
>>>>>
>>>>> _______________________________________________
>>>>> cisco-nsp mailing list  cisco-nsp at puck.nether.net
>>>>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>>>>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>>>>
>>>>
>>>> _______________________________________________
>>> cisco-nsp mailing list  cisco-nsp at puck.nether.net
>>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>>
>>>
>>
>> _______________________________________________
>> cisco-nsp mailing list  cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>
>
>
> --
> Regards,
>
> Mark L. Tees