[c-nsp] FWSM traffic distribution across internal etherchannel

Fri Oct 30 15:01:25 EDT 2009

First, I wanted to give thanks to David for helping me track down this issue
and for providing insight into the workings of the FWSM.

To recap the issue I was seeing the majority of outbound traffic from the
FWSM was exiting on the 3rd and 6th port of the ether-channel while the
inbound traffic to the FWSM was pretty much equally distributed across the
six links. Some more info on our setup… Behind this FWSM we have several
high profile web properties so the majority of the traffic is http. Also,
everything behind the FWSM is NATed with static NAT and we were using
src-dst-ip for the ether-channel load-balancing algorithm on the 6500's.

Generally traffic on the FWSM will exit the same port it was received on
with several exceptions

1) Traffic inspected by the CP
2) Fragmented traffic
3) Packets forwarded between the NP's

We were able to rule out the inspected and fragmented traffic pretty easily
which just left option 3.

The command 'show np 1 stats | inc blade' displays the number of packets
that are forwarded from one NP to the other. When we issued this command
twice with a 5 second pause in between we saw a significant increase in the
counters.

Here is an example of what happens when a client makes a connection to one
of our sites

1) Packet comes from client destined to the server.
2) The SUP hashes the packet based on the src and dst IP and sends it out
port 1 (NP 1) of the FWSM
3) Connection is created on NP1
4) IP header in the packet is NATed by the FWSM and sent out to destination
server
5) Server replies back to the client
6) SUP hashes the packet again, but since one of the IPs has been NATed by
the FWSM the packet is now hashed to port 5 (NP2)
7) FWSM receives the packet, but since the connection for this flow resides
on NP1 it forwards the packet from NP2 to NP1
8) once the packet is processed by NP1 it is forwarded out port 3 (if we
reversed it and NP2 did the processing it would have been forwarded out port
6)

So the reason we saw the majority of traffic egress ports 3 and 6 was
because the ether-channel hash and the static NAT. By changing the
ether-channel load-balancing algorithm to src-dst-port we were able to get
equal distribution of traffic out of the FWSM.

On Wed, Oct 28, 2009 at 3:51 AM, <NMaio at guesswho.com> wrote:

>  David,
>
> It appears that I might have misunderstood the original question since it
> was only pertaining to traffic from the FWSM.  My apologies.
>
> Thanks,
>
> Nick
>
>
>
>
>
> *From:* David White, Jr. (dwhitejr) [mailto:dwhitejr at cisco.com]
> *Sent:* Tuesday, October 27, 2009 10:32 PM
> *To:* Nicholas Maio
> *Cc:* j4bles at gmail.com; cisco-nsp at puck.nether.net
>
> *Subject:* Re: [c-nsp] FWSM traffic distribution across internal
> etherchannel
>
>
>
> Hi Nick,
>
> Changing the SUP's load-balancing algorithm (which is what is described in
> the link provided) only affects the traffic that egresses the switch and
> ingresses the FWSM.  It does not impact the packet distribution in the
> reverse direction (egress the FWSM and ingress on the switch).
>
> I didn't indicate that I would need to know the traffic profile to
> determine the correct SUP load-balancing algorithm, but rather to explain
> why ports 3 and 6 were mainly utilized for traffic egressing the FWSM -
> which was Jack's original question.
>
> Sincerely,
>
> David.
>
> NMaio at guesswho.com wrote:
>
> David,
>
> The section named "Customizing the FWSM Internal Interface" in the following page
>
> http://www.cisco.com/en/US/docs/security/fwsm/fwsm31/configuration/guide/switch_f.html
>
> would be helpful.
>
>
>
> As you stated you would need to know the traffic profile to detemine the correct algorithm but why would you say that there aren't any commands to change this?  The command is not run in the fwsm but rather the switch/router.
>
> Nick
>
>
>
>
>
> ________________________________________
>
> From: cisco-nsp-bounces at puck.nether.net [cisco-nsp-bounces at puck.nether.net] On Behalf Of David White, Jr. (dwhitejr) [dwhitejr at cisco.com]
>
> Sent: Tuesday, October 27, 2009 8:29 PM
>
> To: jack b
>
> Cc: cisco-nsp at puck.nether.net
>
> Subject: Re: [c-nsp] FWSM traffic distribution across internal etherchannel
>
>
>
> Hi Jack,
>
>
>
> Yes, it is most likely that this is normal.  There are no CLI commands
>
> on the FWSM to adjust this. I would have to understand your traffic
>
> profile along with your config to tell you why the given profile is
>
> almost exclusively utilizing ports 3 and 6.
>
>
>
> Sincerely,
>
>
>
> David.
>
>
>
>
>
> jack b wrote:
>
>
>
> I have a FWSM running 2.3(4)11 in slot 4 of a 6509. I have noticed that I am
>
> getting unequal traffic distribution on the links that make up the ether
>
> channel bundle  between the FWSM and 6509.
>
>
>
> Here is a snapshot of the traffic distribution
>
>
>
> 4/1    in 28.99mbps    out 458.10mbps
>
> 4/2    in 12.37mbps    out 248.31mbps
>
> 4/3    in 960.86mbps  out 294.95mbps
>
> 4/4    in 34.07mbps    out 505.22mbps
>
> 4/5    in 15.08mbps    out 243.10mbps
>
> 4/6    in 950.63mbps  out 262.68mbps
>
>
>
> In is traffic from the FWSM to the switch and out is traffic from the switch
>
> to the FWSM.
>
>
>
> Is this normal operation, or is there a way to distribute the traffic from
>
> the FWSM to the switch more evenly?
>
> _______________________________________________
>
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
>
> https://puck.nether.net/mailman/listinfo/cisco-nsp
>
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>
>
>
>
>
>
> _______________________________________________
>
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
>
> https://puck.nether.net/mailman/listinfo/cisco-nsp
>
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>
>
>