[f-nsp] High load on the MLX lp's cpu - how to avoid a fallback to software routing in case of one arm routing situations

[Richard A Steenbergen]

On Sun, Dec 10, 2006 at 04:08:19PM +0100, Gunther Stammwitz wrote:
> Hello,
> 
> I just wanted to document to the list that there is a bug (oh sorry..
> foundry is calling this some sort of "feature" and doesn't want to change
> it) in the netiron mlx software 3.1 that causes the line card to fall back
> to cpu switching instead of hardware routing in one-arm-routing-situations.
> This means if traffic is entering and leaving the router on the same
> physical interface - different vlans or ves don't matter - parts of it will
> be cpu switched. Foundry is calling this one arm routing.
...
> The explanation for the behavior is that the high CPU on the LP was a result
> of ICMP redirects. 

No actually that is a feature, and Foundry is 100% correct (well ok at 
least 90% correct) in their handling of it.

The original theory behind ICMP redirect is that rather than try to 
achieve optimal routing at the host level by having every host running an 
active routing protocol (which would have been RIP at the time, a Very Bad 
Idea (tm) by modern standards), hosts would be able to send all packets to 
their default gateway by default (hence the name :P). If that router 
turned out to know about a "better" path on that first hop, it could 
inform the host via a simple stateless mechanism (aka an ICMP redirect) 
which required very little work on both the router and the host compared 
to maintaining full routing protocol sessions and exchanging a full table 
with every host. The RFC specifies that the host MUST accept the redirect, 
so in theory this should only need to be once per host->dst flow, and only 
"on-demand".

Per RFC1812 the conditions for a router sending an ICMP redirect are:

o The packet is being forwarded out the same physical interface that
  it was received from,
o The IP source address in the packet is on the same Logical IP
  (sub)network as the next-hop IP address, and
o The packet does not contain an IP source route option.

All of which your situation meets, thus Foundry is technically correct in 
send out ICMP redirect. It is unfortunate for you that generating an ICMP 
redirect is a CPU-based operation (as it is with every other router), but 
it is technically speaking a perfectly correct and standardized behavior 
which you are more than welcome to disable if you don't like it.

Now, lets stop talking technicalities and talk reality for a minute. The 
reality is that ICMP redirects are a stupid idea on a modern network. They 
made sense back when the Internet was powered by very slow CPU-based 
routers and comparatively much faster bridges or hubs, where you really 
didn't want to continue to send a stream of packets to a router 
unnecessarily only to have it turn around and come back to the same 
interface. Unfortunately in a modern high-speed network it is actually 
more work to stop and cpu-process the packet and its ICMP redirect than it 
is to just hardware forward the thing in a one-armed fashion. People also 
now realize that accepting arbitrary packets from the Internet which force 
your host to install arbitrary new routes in its RIB is a pretty bad idea 
from a security perspective (at the VERY least, it is a big DoS vector).

The funny thing about RFC's is that times change, sometimes they don't get 
updated, and behaviors that made sense on paper 10 years ago no longer 
make sense in an operational environment today. For example, page 53 of 
the same RFC defines a behavior for source-address selection of ICMP 
messages that if followed would render traceroute completely 
non-functional. The RFC has never been updated, it is simply a de facto 
standard that folks don't do it that way. The ICMP message right before 
redirect is source quench, another one which doesn't make sense and isn't 
implemented in modern networks.

There are really three good solutions to your problem:

#1 Don't run single armed router configurations. This is a ghetto hack 
only sensibly implemented when you a cheap bastard who needs to use 
outrageously overpriced gear (coughjunipercough). You have to be pretty 
damn cheap to have this problem with Foundry, which is why most of the 
world doesn't care about this issue.

#2 Just turn off ICMP redirects globally. There is a reason you have the 
option to disable it, and this is it. You probably want to do this anyways 
no matter what else you do, since you have essentially nothing to gain but 
plenty to lose from having redirect generation enabled in the event of a 
DoS or worm. Most sensible people running decent sized networks are doing 
the same.

#3 If you're really determined to get a "better" solution, you might try 
pointing out to Foundry that the requirement for "same physical interface" 
SHOULD be updated to say same physical and logical interface (i.e. you 
shouldn't send a redirect for packets received on Ethernet1/1 vlan 98 to a 
next-hop on Ethernet1/1 vlan 99). Mind you 802.1q wasn't even an offical 
standard until 1998, so the RFC editors had no way to know that this would 
be an issue, but this is very much a document in need of an update. Given 
Foundry's long history of repeated violating offical protocol and RFC 
requirements in the pursuit of a more "correct" behavior, you might have 
more luck if you tried explaining it this way. :)

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)