[c-nsp] Weird Multicast microburst amplification issue
Frank Bulk
frnkblk at iname.com
Sun Dec 11 00:37:20 EST 2011
What if you interconnect the two switches with 1G rather than 10?
Frank
-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net
[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Matthew Huff
Sent: Friday, December 09, 2011 2:18 PM
To: 'Chuck Church'; 'cisco-nsp'
Subject: Re: [c-nsp] Weird Multicast microburst amplification issue
Yes, only the correct stream. I've opened a case with Cisco. I'm suspecting
that the multicast replication engine is doing something that causes it to
amplify the bursty nature of the traffic causing the microburst overruns.
----
Matthew Huff | 1 Manhattanville Rd
Director of Operations | Purchase, NY 10577
OTA Management LLC | Phone: 914-460-4039
aim: matthewbhuff | Fax: 914-460-4139
> -----Original Message-----
> From: Chuck Church [mailto:chuckchurch at gmail.com]
> Sent: Friday, December 09, 2011 3:01 PM
> To: Matthew Huff; 'cisco-nsp'
> Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
>
> Hmmm. If it's not spanning tree, I'd have to say it's something not
> working right with IGMP, and the server's port is getting more streams
> than it should. Have you checked the IGMP port association to see what
> it's subscribed to?
>
> Chuck
>
>
> -----Original Message-----
> From: Matthew Huff [mailto:mhuff at ox.com]
> Sent: Friday, December 09, 2011 2:48 PM
> To: 'Chuck Church'; 'cisco-nsp'
> Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
>
> Yes. The problem only occurs when connected to any other switch than
> the source switch. We have over 300 servers, it isn't anything with a
> specific server, but rather the nature of not being on the same switch
> as the source.
> The data is a high packet count, bursty traffic. However, the drops
> only occur on the 6748 module via the layer 2 output on a server
> subscribing to that multicast traffic. This occurs if we turn layer 2
> flowcontrol on or off. No pause packets are generated from the server,
> rather this is 100% related to the output ASIC queues on the 6748
> module.
>
>
>
> ----
> Matthew Huff | 1 Manhattanville Rd
> Director of Operations | Purchase, NY 10577
> OTA Management LLC | Phone: 914-460-4039
> aim: matthewbhuff | Fax: 914-460-4139
>
>
> > -----Original Message-----
> > From: Chuck Church [mailto:chuckchurch at gmail.com]
> > Sent: Friday, December 09, 2011 2:43 PM
> > To: Matthew Huff; 'cisco-nsp'
> > Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
> >
> > Can you move the source server over to switch B to see if the problem
> > still exists on switch B then, or moves to switch A? Anything
> showing
> > up in the logs?
> >
> > Chuck
> >
> >
> > -----Original Message-----
> > From: Matthew Huff [mailto:mhuff at ox.com]
> > Sent: Friday, December 09, 2011 2:25 PM
> > To: 'Chuck Church'; 'cisco-nsp'
> > Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
> >
> > Unfortunately, it isn't something simple like that. The output drops
> > are continuously happening. The network is very stable. There are not
> > other issues during this time. It's something amplifying the burst of
> > the stream, either the multicast replication or passing through the
> > layer 3 interface.
> >
> > IF we run a test with a server on switch a, a client on switch a, and
> > a client on switch b, only the client on switch b is seeing the
> problem.
> > The problem isn't passing the data from switch-a to b, but rather
> > something during the transmission that changes the shape of the data
> > to be a heavier burst.
> >
> >
> >
> > ----
> > Matthew Huff | 1 Manhattanville Rd
> > Director of Operations | Purchase, NY 10577
> > OTA Management LLC | Phone: 914-460-4039
> > aim: matthewbhuff | Fax: 914-460-4139
> >
> >
> > > -----Original Message-----
> > > From: Chuck Church [mailto:chuckchurch at gmail.com]
> > > Sent: Friday, December 09, 2011 2:11 PM
> > > To: Matthew Huff; 'cisco-nsp'
> > > Subject: RE: [c-nsp] Weird Multicast microburst amplification issue
> > >
> > > Are there multiple streams passing through the switch? A spanning
> > > tree recalculation will cause IGMP to flush associations, and flood
> > > all streams out all ports until they're relearned. Portfast will
> > > fix it, as will a multicast-specific interface command, would need
> > > to
> > look
> > > it up.
> > >
> > > Chuck
> > >
> > > -----Original Message-----
> > > From: cisco-nsp-bounces at puck.nether.net
> > > [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Matthew
> Huff
> > > Sent: Friday, December 09, 2011 1:49 PM
> > > To: 'cisco-nsp (cisco-nsp at puck.nether.net)'
> > > Subject: [c-nsp] Weird Multicast microburst amplification issue
> > >
> > > We have a multicast data stream (real-time ticker data) that by its
> > > nature is very bursty.
> > >
> > > When we connect a source server via gigabit Ethernet to our
> > > 6500/sup720 switch via a 6748 module and a destination server via
> > > gigabit to the same or different module in the same switch,
> > > everything works fine. If the destination server is on a different
> > > switch connected by a layer3 10GB connection then we have
> > > significant output drops on the Ethernet connected to the
> destination server.
> > >
> > > All switches are 6509/sup720 with 6748 line cards. QoS is disabled
> > > globally.
> > > The servers are identical. The output drops only occur on the
> > Ethernet
> > > drop connected to the server.
> > >
> > > The only thing I can think is happening is that by routing the
> > traffic
> > > via the 10gb L3 interface, something is causing the traffic burst
> to
> > > amplify, overrunning the output port. Has anyone seen this, and
> does
> > > anyone know how to mitigate this?
> > >
> > > ----
> > > Matthew Huff | 1 Manhattanville Rd
> > > Director of Operations | Purchase, NY 10577
> > > OTA Management LLC | Phone: 914-460-4039
> > > aim: matthewbhuff | Fax: 914-460-4139
> > >
> > > _______________________________________________
> > > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
More information about the cisco-nsp
mailing list