[c-nsp] Weird Multicast microburst amplification issue

[Matthew Huff]

Yes. The problem only occurs when connected to any other switch than the source switch. We have over 300 servers, it isn't anything with a specific server, but rather the nature of not being on the same switch as the source. The data is a high packet count, bursty traffic. However, the drops only occur on the 6748 module  via the layer 2 output on a server subscribing to that multicast traffic. This occurs if we turn layer 2 flowcontrol on or off. No pause packets are generated from the server, rather this is 100% related to the output ASIC queues on the 6748 module.



----
Matthew Huff             | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039
aim: matthewbhuff        | Fax:   914-460-4139


> -----Original Message-----
> From: Chuck Church [mailto:chuckchurch at gmail.com]
> Sent: Friday, December 09, 2011 2:43 PM
> To: Matthew Huff; 'cisco-nsp'
> Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
> 
> Can you move the source server over to switch B to see if the problem
> still exists on switch B then, or moves to switch A?  Anything showing
> up in the logs?
> 
> Chuck
> 
> 
> -----Original Message-----
> From: Matthew Huff [mailto:mhuff at ox.com]
> Sent: Friday, December 09, 2011 2:25 PM
> To: 'Chuck Church'; 'cisco-nsp'
> Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
> 
> Unfortunately, it isn't something simple like that. The output drops
> are continuously happening. The network is very stable. There are not
> other issues during this time. It's something amplifying the burst of
> the stream, either the multicast replication or passing through the
> layer 3 interface.
> 
> IF we run a test with a server on switch a, a client on switch a, and a
> client on switch b, only the client on switch b is seeing the problem.
> The problem isn't passing the data from switch-a to b, but rather
> something during the transmission that changes the shape of the data to
> be a heavier burst.
> 
> 
> 
> ----
> Matthew Huff             | 1 Manhattanville Rd
> Director of Operations   | Purchase, NY 10577
> OTA Management LLC       | Phone: 914-460-4039
> aim: matthewbhuff        | Fax:   914-460-4139
> 
> 
> > -----Original Message-----
> > From: Chuck Church [mailto:chuckchurch at gmail.com]
> > Sent: Friday, December 09, 2011 2:11 PM
> > To: Matthew Huff; 'cisco-nsp'
> > Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
> >
> > Are there multiple streams passing through the switch?  A spanning
> > tree recalculation will cause IGMP to flush associations, and flood
> > all streams out all ports until they're relearned.  Portfast will fix
> > it, as will a multicast-specific interface command, would need to
> look
> > it up.
> >
> > Chuck
> >
> > -----Original Message-----
> > From: cisco-nsp-bounces at puck.nether.net
> > [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Matthew Huff
> > Sent: Friday, December 09, 2011 1:49 PM
> > To: 'cisco-nsp (cisco-nsp at puck.nether.net)'
> > Subject: [c-nsp] Weird Multicast microburst amplification issue
> >
> > We have a multicast data stream (real-time ticker data) that by its
> > nature is very bursty.
> >
> > When we connect a source server via gigabit Ethernet to our
> > 6500/sup720 switch via a 6748 module and a destination server via
> > gigabit to  the same or different module in the same switch,
> > everything works fine. If the destination server is on a different
> > switch connected by a layer3 10GB connection then we have significant
> > output drops on the Ethernet connected to the destination server.
> >
> > All switches are 6509/sup720 with 6748 line cards. QoS is disabled
> > globally.
> > The servers are identical. The output drops only occur on the
> Ethernet
> > drop connected to the server.
> >
> > The only thing I can think is happening is that by routing the
> traffic
> > via the 10gb L3 interface, something is causing the traffic burst to
> > amplify, overrunning the output port. Has anyone seen this, and does
> > anyone know how to mitigate this?
> >
> > ----
> > Matthew Huff             | 1 Manhattanville Rd
> > Director of Operations   | Purchase, NY 10577
> > OTA Management LLC       | Phone: 914-460-4039
> > aim: matthewbhuff        | Fax:   914-460-4139
> >
> > _______________________________________________
> > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/