[c-nsp] ASR1K forwarding failures on 10G SPA's

[Stephen Fulton]

Gentlemen,

Interesting, I checked this morning and the input drops were very high, 
despite being cleared 12 hours ago on a router no longer in production. 
If anyone has an TAC case they can reference (privately or otherwise) 
I'd appreciate it, as I have a TAC case open now.  I'll wait on updating 
IOS-XE from 3.16.3.S until TAC is ready.

Thanks,

-- Stephen

On 2016-10-04 1:57 AM, Sascha Pollok wrote:
> Exactly. OP might try to raise hold-queue xx in on those interfaces. If
> it solves the problem temporarily (!) he found it.
>
> If so, show buffers input-interfacw should give a hint.
> The NTP bug came up pretty recently (2 months or so?) so it could
> actually be the cause.
>
> -Sascha
>
> Am 4. Oktober 2016 07:45:36 schrieb Mark Tees <marktees at gmail.com>:
>
>> That sounds like what I experienced in ASR920 land recently with bad
>> packets filling up interface input queues causing a wedge.
>>
>> When it happens check the interface input queues and save the output.
>>
>> The resolution for us so far has been tight CoPP with discards, iACLs,
>> and the like to only allow things towards the boxes that are as
>> trusted as possible.
>>
>> On Tuesday, 4 October 2016, Sascha Pollok <sp at iphh.net
>> <mailto:sp at iphh.net>> wrote:
>>
>>     Just to make sure: latest IOS XE version? Its not the NTP
>>     processing bug filling up interface queues? How does the input
>>     queue look on the affected interfaces?
>>
>>     Cheers
>>     Sascha
>>
>>
>>     Am 4. Oktober 2016 05:33:39 schrieb Stephen Fulton
>>     <sf at lists.esoteric.ca>:
>>
>>         ISIS adjacencies drop as well as BGP sessions on neighboring
>>         devices drop.
>>
>>         Issue just reoccurred.
>>
>>         -- Stephen
>>
>>         On 2016-10-03 10:59 PM, Scott Granados wrote:
>>
>>             Anything logged while this happens?
>>
>>                 On Oct 3, 2016, at 10:52 PM, Stephen Fulton
>>                 <sf at lists.esoteric.ca> wrote:
>>
>>                 Hi all,
>>
>>                 I have run into a number of forwarding failure events
>>                 on ASR1K's with 10G SPA's.  These have occurred across
>>                 a range of IOS-XE versions, using various ROMMON
>>                 versions and across two different ASR1K platforms
>>                 (1002's and 1004's).  Multiple SPA's have been
>>                 replaced, IOS-XE versions and ROMMON versions upgraded
>>                 and in the case of the ASR1004's, SIP's replaced (both
>>                 SIP10 and SIP40's).  TAC cases have been opened
>>                 several times.
>>
>>                 What occurs is forwarding across an interface fails
>>                 completely.  The easiest way to find it is the lack of
>>                 ARP entries on the interface/sub-interface, due to
>>                 time-outs, but traffic is still attempting to traverse
>>                 the interface.  When I ping the IP address associated
>>                 with the failed interface, it fails.  ARP resolution
>>                 of any neighbors fails, and neighboring devices on the
>>                 same broadcast domain cannot reach it - though will
>>                 see its MAC in the ARP table.
>>
>>                 In all cases, ISIS and MPLS was configured on the
>>                 interfaces.  BFD has been on some, not on others.
>>
>>                 I recently found learned of another organization that
>>                 saw the same behavior on an ASR1006 with 10G SPA's.
>>                 SPA's and SIP's were replaced and the last advice they
>>                 received from TAC was that if it occurred again the
>>                 chassis would need to replaced.  It did but they chose
>>                 not to replace the chassis and simply stopped using
>>                 10G entirely.
>>
>>                 Has anyone else seen this?
>>
>>                 -- Stephen
>>
>>                 _______________________________________________
>>                 cisco-nsp mailing list  cisco-nsp at puck.nether.net
>>                 https://puck.nether.net/mailman/listinfo/cisco-nsp
>>                 <https://puck.nether.net/mailman/listinfo/cisco-nsp>
>>                 archive at http://puck.nether.net/pipermail/cisco-nsp/
>>                 <http://puck.nether.net/pipermail/cisco-nsp/>
>>
>>
>>         _______________________________________________
>>         cisco-nsp mailing list  cisco-nsp at puck.nether.net
>>         https://puck.nether.net/mailman/listinfo/cisco-nsp
>>         <https://puck.nether.net/mailman/listinfo/cisco-nsp>
>>         archive at http://puck.nether.net/pipermail/cisco-nsp/
>>         <http://puck.nether.net/pipermail/cisco-nsp/>
>>
>>
>>
>>     _______________________________________________
>>     cisco-nsp mailing list  cisco-nsp at puck.nether.net
>>     https://puck.nether.net/mailman/listinfo/cisco-nsp
>>     <https://puck.nether.net/mailman/listinfo/cisco-nsp>
>>     archive at http://puck.nether.net/pipermail/cisco-nsp/
>>     <http://puck.nether.net/pipermail/cisco-nsp/>
>>
>>
>>
>> --
>> Regards,
>>
>> Mark L. Tees
>>