[c-nsp] 6704-10GE huge input drops (flushes)

[Antonio Soares]

The counters were cleared yesterday and no issues were seen but we already see some drops:

cat6k#sh int te1/1 | inc drops|rate|clearing 
  Last clearing of "show interface" counters 16:12:04
  Input queue: 0/2000/899/899 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  30 second input rate 1212044000 bits/sec, 317185 packets/sec
  30 second output rate 2868403000 bits/sec, 368467 packets/sec
cat6k#

Yes, back-to-back L3 interface to a GSR. No MPLS, no sub-interfaces. Only IPv4/IPv6 addressing and ISIS there.

When the last occurrence happened, we saw an increase of 5 million drops.

It's a sporadic thing, it lasts a couple of minutes then everything returns to normal.


Regards,

Antonio Soares, CCIE #18473 (R&S/SP)
amsoares at netcabo.pt
http://www.ccie18473.net



-----Original Message-----
From: Saku Ytti [mailto:saku at ytti.fi] 
Sent: terça-feira, 7 de Maio de 2013 11:42
To: Antonio Soares
Cc: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] 6704-10GE huge input drops (flushes)

It's probably some trash ending up in control-plane, congesting your receive queue, consequently other valid control-plane stuff, like BGP/IGP have to compete for gaining access to processing.

I'm very surprised it's back-to-back to GSR. Are there subinterfaces between them? Or just single untagged core IP/MPLS interface?

I presume you're not seeing constant growth in the drops? That is, what ever issue you have is sporadic, and when it happens, you get huge amount of trash, otherwise nothing?

On 7 May 2013 13:10, Antonio Soares <amsoares at netcabo.pt> wrote:
>
> The ibc values I see now (normal behavior):
>
> cat6k#show ibc | inc packets
>         5 minute rx rate 65000 bits/sec, 94 packets/sec
>         5 minute tx rate 27000 bits/sec, 35 packets/sec
>
> Show ibc from show tech captured two hours after the first occurrence:
>
> Interface information:
>         Interface IBC0/0(idb 0x50ECB7A8)
>         Hardware is Mistral IBC (revision 5)
>         5 minute rx rate 323000 bits/sec, 89 packets/sec
>         5 minute tx rate 37000 bits/sec, 29 packets/sec
>         5868647586 packets input, 801582284787 bytes
>         1350630188 broadcasts received
>         3458894285 packets output, 636391923013 bytes
>         1580035101 broadcasts sent
>         0 Inband input packet drops
>         0 Bridge Packet loopback drops
>         973954452 Packets CEF Switched, 11041138 Packets Fast Switched
>         0 Packets SLB Switched, 0 Packets CWAN Switched
>         IBC resets   = 1; last at 06:30:38.772 UTC Thu Jan 13 2011
>
> I was convinced that the flushes had to do with Hw. What happens with 
> these counters when we have bursts of traffic ?
> Is it possible to a have a burst of traffic impacting the bgp/igp 
> adjacencies ?
>
>
> Regards,
>
> Antonio Soares, CCIE #18473 (R&S/SP)
> amsoares at netcabo.pt
> http://www.ccie18473.net
>
>
>
> -----Original Message-----
> From: cisco-nsp [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf 
> Of Saku Ytti
> Sent: terça-feira, 7 de Maio de 2013 07:38
> To: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] 6704-10GE huge input drops (flushes)
>
> On (2013-05-06 23:59 +0100), Antonio Soares wrote:
>
>>   Input queue: 0/2000/15863293/15863293 (size/max/drops/flushes); 
>> Total
>
>> I was thinking about bursts or micro-bursts but when this happened, 
>> the bgp/isis sessions that go over this link went down.
>
>> But it makes some sense because the 6704 does full-rate and the
>> 6708/6716 are oversubscribed (more buffering capacity needed).
>
> These counters have nothing to do with HW, they are from SW path. 
> You're either SW switching or getting some trash to control-plane.
>
> You could do netdr or pinnacle or RP/SP ERSPAN capture to see what 
> packets are hitting control-plane
>
> First step would be to compare 'show ibc | i packets/sec' between 
> other box which does not suffer from this, to confirm that packet 
> rates are unexpectedly high.
>
> --
>   ++ytti
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net 
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>



--
  ++ytti