[c-nsp] Weird Multicast microburst amplification issue

[Chuck Church]

Hmmm.  If it's not spanning tree, I'd have to say it's something not working
right with IGMP, and the server's port is getting more streams than it
should.  Have you checked the IGMP port association to see what it's
subscribed to?

Chuck


-----Original Message-----
From: Matthew Huff [mailto:mhuff at ox.com] 
Sent: Friday, December 09, 2011 2:48 PM
To: 'Chuck Church'; 'cisco-nsp'
Subject: RE: [c-nsp] Weird Multicast microburst amplification issue

Yes. The problem only occurs when connected to any other switch than the
source switch. We have over 300 servers, it isn't anything with a specific
server, but rather the nature of not being on the same switch as the source.
The data is a high packet count, bursty traffic. However, the drops only
occur on the 6748 module  via the layer 2 output on a server subscribing to
that multicast traffic. This occurs if we turn layer 2 flowcontrol on or
off. No pause packets are generated from the server, rather this is 100%
related to the output ASIC queues on the 6748 module.



----
Matthew Huff             | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039
aim: matthewbhuff        | Fax:   914-460-4139


> -----Original Message-----
> From: Chuck Church [mailto:chuckchurch at gmail.com]
> Sent: Friday, December 09, 2011 2:43 PM
> To: Matthew Huff; 'cisco-nsp'
> Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
> 
> Can you move the source server over to switch B to see if the problem
> still exists on switch B then, or moves to switch A?  Anything showing
> up in the logs?
> 
> Chuck
> 
> 
> -----Original Message-----
> From: Matthew Huff [mailto:mhuff at ox.com]
> Sent: Friday, December 09, 2011 2:25 PM
> To: 'Chuck Church'; 'cisco-nsp'
> Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
> 
> Unfortunately, it isn't something simple like that. The output drops
> are continuously happening. The network is very stable. There are not
> other issues during this time. It's something amplifying the burst of
> the stream, either the multicast replication or passing through the
> layer 3 interface.
> 
> IF we run a test with a server on switch a, a client on switch a, and a
> client on switch b, only the client on switch b is seeing the problem.
> The problem isn't passing the data from switch-a to b, but rather
> something during the transmission that changes the shape of the data to
> be a heavier burst.
> 
> 
> 
> ----
> Matthew Huff             | 1 Manhattanville Rd
> Director of Operations   | Purchase, NY 10577
> OTA Management LLC       | Phone: 914-460-4039
> aim: matthewbhuff        | Fax:   914-460-4139
> 
> 
> > -----Original Message-----
> > From: Chuck Church [mailto:chuckchurch at gmail.com]
> > Sent: Friday, December 09, 2011 2:11 PM
> > To: Matthew Huff; 'cisco-nsp'
> > Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
> >
> > Are there multiple streams passing through the switch?  A spanning
> > tree recalculation will cause IGMP to flush associations, and flood
> > all streams out all ports until they're relearned.  Portfast will fix
> > it, as will a multicast-specific interface command, would need to
> look
> > it up.
> >
> > Chuck
> >
> > -----Original Message-----
> > From: cisco-nsp-bounces at puck.nether.net
> > [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Matthew Huff
> > Sent: Friday, December 09, 2011 1:49 PM
> > To: 'cisco-nsp (cisco-nsp at puck.nether.net)'
> > Subject: [c-nsp] Weird Multicast microburst amplification issue
> >
> > We have a multicast data stream (real-time ticker data) that by its
> > nature is very bursty.
> >
> > When we connect a source server via gigabit Ethernet to our
> > 6500/sup720 switch via a 6748 module and a destination server via
> > gigabit to  the same or different module in the same switch,
> > everything works fine. If the destination server is on a different
> > switch connected by a layer3 10GB connection then we have significant
> > output drops on the Ethernet connected to the destination server.
> >
> > All switches are 6509/sup720 with 6748 line cards. QoS is disabled
> > globally.
> > The servers are identical. The output drops only occur on the
> Ethernet
> > drop connected to the server.
> >
> > The only thing I can think is happening is that by routing the
> traffic
> > via the 10gb L3 interface, something is causing the traffic burst to
> > amplify, overrunning the output port. Has anyone seen this, and does
> > anyone know how to mitigate this?
> >
> > ----
> > Matthew Huff             | 1 Manhattanville Rd
> > Director of Operations   | Purchase, NY 10577
> > OTA Management LLC       | Phone: 914-460-4039
> > aim: matthewbhuff        | Fax:   914-460-4139
> >
> > _______________________________________________
> > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/