[c-nsp] Design Recommendation for Dual WAN Connectivity with Multiple MPLS Service Providers

[Muhammad Atif Jauhar]

Greetings of the day,

Thank you for the detailed discussion and recommendations. We confirm that
this design is intended for an enterprise environment.

We have one additional query regarding traffic utilization across the two
MPLS WAN links. While we understand that a primary/secondary design using
BGP Local Preference is the safest and most deterministic approach, we
would like to explore whether there are recommended methods to *load
balance traffic across both WAN links* without introducing routing loops,
asymmetric paths, or latency-related issues.

One option under consideration is to manually prefer specific prefixes or
destination routes on WAN Link-1 and others on WAN Link-2. While this is
feasible, we would like to understand if there are alternative or more
scalable approaches that are commonly adopted in enterprise MPLS
environments, such as:

   -

   Controlled BGP-based load sharing using selective Local Preference per
   prefix
   -

   Use of BGP communities to influence provider-side routing for inbound
   traffic distribution
   -

   Application-aware or policy-based traffic steering (where appropriate)
   -

   Any recommended techniques that allow link utilization optimization
   while maintaining per-flow symmetry and stability

We would appreciate your guidance on whether active/active utilization is
advisable in this scenario, and if so, what design guardrails should be
applied to ensure predictable behavior and operational simplicity.

Your insights and best-practice recommendations would be valuable for
finalizing the design.


Regards,

Muhammad Atif Jauhar


On Wed, 21 Jan 2026 at 23:16, Brian Knight <ml at knight-networks.com> wrote:

> On 2026-01-21 07:49, Muhammad Atif Jauhar via cisco-nsp wrote:
> > Greetings,
> >
> > We are planning to deploy two WAN links from different Service
> > Providers,
> > both offering MPLS connectivity with BGP peering. As part of the design
> > review, we would like to align on the best-practice approach to ensure
> > a
> > stable, loop-free, and predictable routing architecture.
> > Given the use of multiple MPLS providers with potentially different
> > latency
> > characteristics, our key objectives are to avoid routing loops, prevent
> > asymmetric traffic issues, and ensure controlled traffic engineering
> > and
> > failover behavior.
> > Any recommendations or considerations based on your experience,
> > particularly with respect to MPLS interconnects and BGP traffic
> > engineering.
>
> Sounds like you have an enterprise network connecting to NSPs. Your
> network is probably not an NSP network itself.
>
> It's not strictly on topic for this mailing list, but I'll provide
> what's worked well for me.
>
> Use a different RFC6996 private ASN for each site.
>
> If you're doing a green-field deployment, use 10/8 RFC1918 space. Assign
> a /16 to each site, so your second octet becomes your site prefix, and
> your VLANs become your third octet.
>
> Choose one NSP as primary, the other as backup. At each site, on your
> incoming route policy from the primary NSP, set all routes to high
> localpref, say, 1000. Set all routes from secondary provider to a lower
> localpref, say 900.
>
> Use BFD with BGP for fastest failure detection. Ask your carriers to
> enable it on your BGP sessions if they support it. There are two
> settings with BFD, minimum interval and multiplier. Review those
> settings to see what fits your needs best. Detail here:
> https://www.cisco.com/c/en/us/td/docs/ios/12_0s/feature/guide/fs_bfd.html
>
> Within each site, install two CE routers. Connect Router A to your
> primary NSP, router B to the secondary. Connect the two routers together
> back-to-back and configure iBGP across that link. Be sure to set
> next-hop-self on the neighbor.
>
> Pick two hub sites; one a primary site, the other a secondary site. Set
> up the hub sites to announce all routes including those from the spoke
> sites. Have the spokes announce only locally originated routes.
>
> Connect those routers to the LAN in each location. Think about how
> you'll integrate routing with your LAN gear.
>
> * If you use an IGP internally at any site, avoid routing loops by
> taking care not to redistribute locally originated routes from BGP back
> into your IGP.
>
> * If you use static routing, make sure a FHRP is set up on the CEs to
> provide failover.
>
> Consider setting up VPN backup through GREoIPSec or VTI tunnels to your
> two hub sites. Run BGP across those tunnels, same BGP process as you use
> for connecting to your NSPs.
>
> Use front-door VRF and ACLs to secure the Internet-facing interfaces on
> the routers.
>
> For routes received via VPN to the primary hub, set those to even lower
> localpref, say 800.
>
> For VPN to the secondary hub, set those to lowest localpref, say 700.
>
> If you have any back-door links between sites, like Metro Ethernet,
> switched Ethernet, or optical waves, integrate those by using BGP as
> well. Announce locally originated routes with a higher local preference
> than the MPLS networks, say, 1100.
>
> But be careful sharing WAN routes learned via MPLS across a backdoor
> link. Don't set those to a higher local preference than the MPLS network
> itself. Avoid if possible.
>
> Also, don't forget about the key management aspects for routers: syslog,
> AAA / TACACS, availability and performance monitoring (SNMP), NTP,
> Netflow, and FTP repository for router code. Use a loopback IP on each
> router for your management traffic. Avoid SFTP or SCP to your code repo
> due to known performance issues with OpenSSH and file transfers.
>
> Consider setting up out-of-band access. Best is to build a small
> parallel WAN/LAN over VPN with a firewall, LAN switch, and serial
> console server. Get a separate Internet connection for the firewall
> (business broadband works), and use it only for out-of-band access.
>
> If you decide to configure QOS over the WAN, try to use standard packet
> markings like EF for voice media and AF31 for signaling. Check with your
> providers on what markings they support. The less re-marking of traffic
> you have to do, the better.
>
> > Regards,
> >
> > Atif.
>
> Hope all this made sense. I'm happy to clarify anything.
>
> This config served me well in my enterprise days and later on as an NSP
> integrator. 25 years of experience in those spaces.
>
> Hope this helps,
>
> -Brian
>