[c-nsp] Weird Multicast microburst amplification issue

Matthew Huff mhuff at ox.com
Fri Dec 9 15:18:11 EST 2011


Yes, only the correct stream. I've opened a case with Cisco. I'm suspecting that the multicast replication engine is doing something that causes it to amplify the bursty nature of the traffic causing the microburst overruns.


----
Matthew Huff             | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039
aim: matthewbhuff        | Fax:   914-460-4139


> -----Original Message-----
> From: Chuck Church [mailto:chuckchurch at gmail.com]
> Sent: Friday, December 09, 2011 3:01 PM
> To: Matthew Huff; 'cisco-nsp'
> Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
> 
> Hmmm.  If it's not spanning tree, I'd have to say it's something not
> working right with IGMP, and the server's port is getting more streams
> than it should.  Have you checked the IGMP port association to see what
> it's subscribed to?
> 
> Chuck
> 
> 
> -----Original Message-----
> From: Matthew Huff [mailto:mhuff at ox.com]
> Sent: Friday, December 09, 2011 2:48 PM
> To: 'Chuck Church'; 'cisco-nsp'
> Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
> 
> Yes. The problem only occurs when connected to any other switch than
> the source switch. We have over 300 servers, it isn't anything with a
> specific server, but rather the nature of not being on the same switch
> as the source.
> The data is a high packet count, bursty traffic. However, the drops
> only occur on the 6748 module  via the layer 2 output on a server
> subscribing to that multicast traffic. This occurs if we turn layer 2
> flowcontrol on or off. No pause packets are generated from the server,
> rather this is 100% related to the output ASIC queues on the 6748
> module.
> 
> 
> 
> ----
> Matthew Huff             | 1 Manhattanville Rd
> Director of Operations   | Purchase, NY 10577
> OTA Management LLC       | Phone: 914-460-4039
> aim: matthewbhuff        | Fax:   914-460-4139
> 
> 
> > -----Original Message-----
> > From: Chuck Church [mailto:chuckchurch at gmail.com]
> > Sent: Friday, December 09, 2011 2:43 PM
> > To: Matthew Huff; 'cisco-nsp'
> > Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
> >
> > Can you move the source server over to switch B to see if the problem
> > still exists on switch B then, or moves to switch A?  Anything
> showing
> > up in the logs?
> >
> > Chuck
> >
> >
> > -----Original Message-----
> > From: Matthew Huff [mailto:mhuff at ox.com]
> > Sent: Friday, December 09, 2011 2:25 PM
> > To: 'Chuck Church'; 'cisco-nsp'
> > Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
> >
> > Unfortunately, it isn't something simple like that. The output drops
> > are continuously happening. The network is very stable. There are not
> > other issues during this time. It's something amplifying the burst of
> > the stream, either the multicast replication or passing through the
> > layer 3 interface.
> >
> > IF we run a test with a server on switch a, a client on switch a, and
> > a client on switch b, only the client on switch b is seeing the
> problem.
> > The problem isn't passing the data from switch-a to b, but rather
> > something during the transmission that changes the shape of the data
> > to be a heavier burst.
> >
> >
> >
> > ----
> > Matthew Huff             | 1 Manhattanville Rd
> > Director of Operations   | Purchase, NY 10577
> > OTA Management LLC       | Phone: 914-460-4039
> > aim: matthewbhuff        | Fax:   914-460-4139
> >
> >
> > > -----Original Message-----
> > > From: Chuck Church [mailto:chuckchurch at gmail.com]
> > > Sent: Friday, December 09, 2011 2:11 PM
> > > To: Matthew Huff; 'cisco-nsp'
> > > Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
> > >
> > > Are there multiple streams passing through the switch?  A spanning
> > > tree recalculation will cause IGMP to flush associations, and flood
> > > all streams out all ports until they're relearned.  Portfast will
> > > fix it, as will a multicast-specific interface command, would need
> > > to
> > look
> > > it up.
> > >
> > > Chuck
> > >
> > > -----Original Message-----
> > > From: cisco-nsp-bounces at puck.nether.net
> > > [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Matthew
> Huff
> > > Sent: Friday, December 09, 2011 1:49 PM
> > > To: 'cisco-nsp (cisco-nsp at puck.nether.net)'
> > > Subject: [c-nsp] Weird Multicast microburst amplification issue
> > >
> > > We have a multicast data stream (real-time ticker data) that by its
> > > nature is very bursty.
> > >
> > > When we connect a source server via gigabit Ethernet to our
> > > 6500/sup720 switch via a 6748 module and a destination server via
> > > gigabit to  the same or different module in the same switch,
> > > everything works fine. If the destination server is on a different
> > > switch connected by a layer3 10GB connection then we have
> > > significant output drops on the Ethernet connected to the
> destination server.
> > >
> > > All switches are 6509/sup720 with 6748 line cards. QoS is disabled
> > > globally.
> > > The servers are identical. The output drops only occur on the
> > Ethernet
> > > drop connected to the server.
> > >
> > > The only thing I can think is happening is that by routing the
> > traffic
> > > via the 10gb L3 interface, something is causing the traffic burst
> to
> > > amplify, overrunning the output port. Has anyone seen this, and
> does
> > > anyone know how to mitigate this?
> > >
> > > ----
> > > Matthew Huff             | 1 Manhattanville Rd
> > > Director of Operations   | Purchase, NY 10577
> > > OTA Management LLC       | Phone: 914-460-4039
> > > aim: matthewbhuff        | Fax:   914-460-4139
> > >
> > > _______________________________________________
> > > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > archive at http://puck.nether.net/pipermail/cisco-nsp/




More information about the cisco-nsp mailing list