[c-nsp] Designing for WAN failover - OSPF/STP

[Code Monkey]

Hi,

I have a quite agreeable problem :-) I have an MPLS/OSPF network on
one site. I am setting up a new site, interconnected to the old one by
two 100Mbps WAN fiber links. It's not dark fiber but it might as well
be, MPLS and VLANs work as desired over the links (MTU OK etc.)

On each of the two sites I have or will have a minimum of two switches
(3550, 3548, 3508) and a minimum of two routers (NPE G1, Juniper
J4350, NPE-300).

How do I get the best possible redundancy and load balancing over the
two WANs while best using my capacity and respecting hard QoS
requirements? OSPF 10-second failover is acceptable, it's what I've
got elsewhere, but of course sub-second would be nice.

In a similar situation with NPE-300s I couldn't get VLANs to work over
the WANs, so I connected them directly to one router on each site, and
interconnected the routers on each site to two switches, with some
fancy VLANning to avoid a router having two IPs in the same network
while ensuring that the routers could communicate if a switch failed.
No SPF at all. Failover was by OSPF, I'm getting quite familiar with
that.

The problem with that was with setting up intersite L2 VLANs, and the
fact that while packets from any one router had direct L2 to two of
the other routers, connection to the last router had to go through one
of the of the two other routers.  When an NPE-300 is running three BGP
feeds and handling say 30 Mbps of traffic with access-lists, you don't
like the idea of pushing 30 more Mbps through it when you think it
could be avoided...

Now that VLANs work over the WANs, I'd like to attach each end of each
WAN to a different switch, connect all the routers on each site to
both switches (using two interfaces on each router), and connect the
switches together too since a lot of the traffic is between servers
connected in the same VLAN, but often on different switches.

I hoped SPF would be able to automagically direct all traffic, but the
further I get into the documentation the less I understand :-( I think
I understand that SPF only establishes one path through the network,
and that there are tricks so that if that path goes down the failover
to another path is very fast, but will I be able to use both my 100
Mbps WANs concurrently? How will I be able to connect a router to two
interconnected switches so that if a switch fails the only causalties
are the servers directly connected to that one switch and half of the
bandwidth available between the sites? Currently my really very
important servers, like DNS resolvers, have two interfaces in
different L3 networks connected to different switches, and run OSPF
with a router connected to the same switch.

Other options I could look at are setting up OSPF on the 3550s, but
I've never really grokked OSPF on a switch (does it work with MPLS?
VLANs?), some sort of channel bonding between the WAN links (but that
would mean putting both on the same switch, I don't want that because
I can't reroute if I have to replace a switch, and the links are
different types with different operators and different RTTs so I'd
like flows to stay on one link unless failover), playing with VLANs
again (but I don't see how I could have a company-wide L2 VLAN
impervious to one WAN link failure that way), or . . .?

Any ideas or advice welcome (I'll take URLs and book references too,
anything goes)!