[f-nsp] BigIron/NetIron experiences?

[Sam Stickland]

Hi Glenn,

On Mon, 28 Feb 2005, Glenn Tan wrote:

> Hi,
>
> We currently run a network made up of 100% Cisco devices.
> We now have to setup a new IP POP and my finance director has asked
> me to decide on something more affordable, so I'm thinking along the
> lines of using the BigIron 4000 (or NetIron 400) as core layer 3
> switches and the NetIron stackable routers (NSR24?) at the
> distribution level.
>
> I would like to hear experiences from fellow Foundry users before we
> try to implement this, as we had a whole stack of Extreme Summit
> switches go terribly wrong (dropped packets and locked up at under
> 10Mbit/s load).
>
> Here are our requirements...
> All the switches must be able to do L2 ingress/egress rate limiting
> and support BGP.
> The core switches should be able to handle at least 2 full BGP tables
> and about 30 peering sessions.
> And of course the core switches should cost less than the Catalyst 6500
> series, which I'm almost certain they will.
>
> I've read some horror stories about Foundry's BGP implementation, does
> anyone know what that is about?

In general we've found the recent builds of Ironware to provide solid BGP 
and OSPF. (This certainly couldn't be said of earlier versions!). The CLI 
is similar enough to cisco's that (with a few exceptions) anyone trained 
on cisco ought to be able to operate it.

It's goes with out saying that should only pick Jetcore based hardware for 
work on L3. Ironcore based hardware will melt under a large number of 
flows. Jetcore hardware, while still technically flow based, only examines 
the first 64 bytes of every packet for flow creation, which gives it far 
more deterministic properties (similar enough to prefix based routing that 
some people believe it is).

On the BigIron's we've experienced the following problems:

* No separate graphing of virtual interfaces. On a cisco 6500 you can 
graph the virtual L3 interfaces (with some gotchas regarding mls, of 
course). On the BigIron's you can only graph L2 ports. This struck me as 
odd considering it looks like this information can be reconstructed from 
the sFlow information, and might therefore be accessible to the hardware, 
but I've had conversations with Foundry engineers who have assured me this 
is a hardware limitation and cannot be fixed in software. This limitation 
does not exist on the NetIrons.

* Enabling port monitoring disables sFlow. Bare this in mind if you plan 
to use sFlow for billing and port monitoring for diagnositics.

* Slightly different next-hop lookup. cisco appears to implement a fully 
recursive next-hop lookup, but the Foundry's don't seem to. This has 
caused us a few problems in the past when static routes injected into 
OSPF weren't configured on a directly connected router (yes... I know 
that's bad anyway ;) ).

* Changes to BGP session config only take affect when the session is 
cleared. This seems harmless to start with but consider that the following 
commands will cause a full table leak!

 	neighbor x.y.z.w remote-as xxxx
 	neighbor x.y.z.w route-map out RM-PEER-OUT

The session is marked as configued after the first command is entered, so 
the second doesn't take affect until 'clear ip bgp nei x.y.z.w soft out' 
is entered.

However, under Foundry's CLI you can apply a route-map before specifying 
an AS number, so the following is fine:

 	neighbor x.y.z.w peer-group PM-PEER
 	neighbor x.y.z.w remote-as xxxx

* I haven't played with it to much, but Foundry's QoS appears to be sorely 
lacking when compared to cisco's.

* No DSCP marking of incoming traffic. Dispite being able to policy route 
and queue based on DSCP values, none of the Foundrys can mark incoming 
traffic with a DSCP value. They can mark IP Precendence via an inbound 
rate-limiting, but bear in mind that the exceed action "mark and transmit" 
appears to just drop the traffic on some versions of Ironware. (ie. It 
still actually rate limits).

Sam