[c-nsp] The dreaded microburst - definition and troubleshooting

[Rodney Dunn]

On Fri, Apr 24, 2009 at 11:47:34AM -0500, Frank Bulk wrote:
> I know what you're feeling.  I had a case open with TAC to diagnose why were
> getting input drops, and they wanted me to packet capture all the traffic
> going to the interface to see if I could identify what traffic was
> generating the microburst....except the volume is 40 to 60 Mbps and the
> drops may not show up for an hour at a time.  How does one correlate it?

Just to be accurate..input drops are a bit different beast. On the
new code that has EPC you can do a process level capture to see the punted
traffic. And if you didn't have the fix for:

CSCse05447    7200 ethernet interfaces should not throttle on input queue full drops 

the input drops could cuase us to throttle the ingress interface hence
result in overruns.



> Run "sh controllers $interface | inc rx_no_descriptors|rx_resource_error"
> and "sh int $interface | inc ignored" every second?  The TAC engineer made
> it sound trivial to match things up, but it's not.  So instead the TAC
> engineer did a little more research and in the end attributed it to
> microbursts.  

It's not easy.

> 
> I don't really understand how Cisco can build a product (I'm using the
> NPE-G2) that can't take deal with microbursts, especially if the other
> interface is a Cisco product!

It's like the GEIP days where it was just meant as a transition plan
from the agg to the core. The speeds of the CPU's went up but not
to the speed of the core. Eventually you have to upgrade the agg to
support the rate. Just took a little longer for ASR1000 to come out.

  The RX ring on an NPE-G2 is a 128 -- why
> wasn't it designed with 1024?  Rodney mentions working with the BU, but it
> appears in the end it doesn't matter that input drops due to microbursts are
> a fact with this product. 

I want to be 100% accurate...it's not input drops that people correlate
to the "input drop" on the 'sh int' output. That is totally different and
not a result of a throttle/overrun. We tried to bump up the rx ring depth
to be more tolerate of those burst but after a detailed analysis it simply
wasn't worth the risk to change it.

Real answer is move to something that can do gig linerate...ASR1000, etc..

Rodney

 The Cisco TAC engineer was apologetic, but he
> wasn't going to lobby the PM. 
> 
> Frank
> 
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net
> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Dale Shaw
> Sent: Thursday, April 23, 2009 11:38 PM
> To: cisco-nsp at puck.nether.net
> Subject: [c-nsp] The dreaded microburst - definition and troubleshooting
> 
> Hi all,
> 
> Is there a universally agreed upon definition for a 'microburst'?
> 
> Is there a defined time measurement - i.e. 5ms, 10ms, 50ms, 100ms,
> 1000ms - during which a certain bps or pps threshold must be
> met/exceeded?
> 
> Does anyone have any tips for troubleshooting microbursts,
> particularly in relation to the c7200 platform exhibiting "no buff"
> drops? We're going to capture some data (w/SPAN on an adjacent switch)
> but it would be nice to be able to look at the data and somehow marry
> it up with incrementing drop counters on the affected c7200 interface.
> 
> It would be nice to be able to explain such drops like "within the
> measurement window, we saw traffic at bps/pps rate x, and we know that
> anything beyond bps/pps rate y will result in drops".
> 
> I suppose it's platform-specific, but how does one come up with an
> accurate benchmark? Is such precision just wishful thinking in the
> murky world of microbursts? :-)
> 
> cheers,
> Dale
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
> 
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/