[c-nsp] Design Recommendation for Dual WAN Connectivity with Multiple MPLS Service Providers

[Brian Knight]

On 2026-01-21 22:40, Muhammad Atif Jauhar wrote:

> Greetings of the day,
> 
> Thank you for the detailed discussion and recommendations. We confirm 
> that this design is intended for an enterprise environment.
> 
> We have one additional query regarding traffic utilization across the 
> two MPLS WAN links. While we understand that a primary/secondary design 
> using BGP Local Preference is the safest and most deterministic 
> approach, we would like to explore whether there are recommended 
> methods to load balance traffic across both WAN links without 
> introducing routing loops, asymmetric paths, or latency-related issues.

I have implemented this for customers with Viptela and Versa SD-WAN 
products in the past.

The SD-WAN tool set tends to do load sharing more effectively and is 
much more supportable than trying to load share using destination-based 
routing alone. This is especially true for your application-aware 
requirement.

> One option under consideration is to manually prefer specific prefixes 
> or destination routes on WAN Link-1 and others on WAN Link-2. While 
> this is feasible, we would like to understand if there are alternative 
> or more scalable approaches that are commonly adopted in enterprise 
> MPLS environments, such as:
> 
> *
> 
> Controlled BGP-based load sharing using selective Local Preference per 
> prefix
> *
> 
> Use of BGP communities to influence provider-side routing for inbound 
> traffic distribution
> *
> 
> Application-aware or policy-based traffic steering (where appropriate)
> *
> 
> Any recommended techniques that allow link utilization optimization 
> while maintaining per-flow symmetry and stability
> 
> We would appreciate your guidance on whether active/active utilization 
> is advisable in this scenario, and if so, what design guardrails should 
> be applied to ensure predictable behavior and operational simplicity.
> 
> Your insights and best-practice recommendations would be valuable for 
> finalizing the design.
> 
> Regards,
> Muhammad Atif Jauhar
> 
> On Wed, 21 Jan 2026 at 23:16, Brian Knight <ml at knight-networks.com> 
> wrote:
> 
>> On 2026-01-21 07:49, Muhammad Atif Jauhar via cisco-nsp wrote:
>>> Greetings,
>>> 
>>> We are planning to deploy two WAN links from different Service
>>> Providers,
>>> both offering MPLS connectivity with BGP peering. As part of the 
>>> design
>>> review, we would like to align on the best-practice approach to 
>>> ensure
>>> a
>>> stable, loop-free, and predictable routing architecture.
>>> Given the use of multiple MPLS providers with potentially different
>>> latency
>>> characteristics, our key objectives are to avoid routing loops, 
>>> prevent
>>> asymmetric traffic issues, and ensure controlled traffic engineering
>>> and
>>> failover behavior.
>>> Any recommendations or considerations based on your experience,
>>> particularly with respect to MPLS interconnects and BGP traffic
>>> engineering.
>> 
>> Sounds like you have an enterprise network connecting to NSPs. Your
>> network is probably not an NSP network itself.
>> 
>> It's not strictly on topic for this mailing list, but I'll provide
>> what's worked well for me.
>> 
>> Use a different RFC6996 private ASN for each site.
>> 
>> If you're doing a green-field deployment, use 10/8 RFC1918 space. 
>> Assign
>> a /16 to each site, so your second octet becomes your site prefix, and
>> your VLANs become your third octet.
>> 
>> Choose one NSP as primary, the other as backup. At each site, on your
>> incoming route policy from the primary NSP, set all routes to high
>> localpref, say, 1000. Set all routes from secondary provider to a 
>> lower
>> localpref, say 900.
>> 
>> Use BFD with BGP for fastest failure detection. Ask your carriers to
>> enable it on your BGP sessions if they support it. There are two
>> settings with BFD, minimum interval and multiplier. Review those
>> settings to see what fits your needs best. Detail here:
>> https://www.cisco.com/c/en/us/td/docs/ios/12_0s/feature/guide/fs_bfd.html
>> 
>> Within each site, install two CE routers. Connect Router A to your
>> primary NSP, router B to the secondary. Connect the two routers 
>> together
>> back-to-back and configure iBGP across that link. Be sure to set
>> next-hop-self on the neighbor.
>> 
>> Pick two hub sites; one a primary site, the other a secondary site. 
>> Set
>> up the hub sites to announce all routes including those from the spoke
>> sites. Have the spokes announce only locally originated routes.
>> 
>> Connect those routers to the LAN in each location. Think about how
>> you'll integrate routing with your LAN gear.
>> 
>> * If you use an IGP internally at any site, avoid routing loops by
>> taking care not to redistribute locally originated routes from BGP 
>> back
>> into your IGP.
>> 
>> * If you use static routing, make sure a FHRP is set up on the CEs to
>> provide failover.
>> 
>> Consider setting up VPN backup through GREoIPSec or VTI tunnels to 
>> your
>> two hub sites. Run BGP across those tunnels, same BGP process as you 
>> use
>> for connecting to your NSPs.
>> 
>> Use front-door VRF and ACLs to secure the Internet-facing interfaces 
>> on
>> the routers.
>> 
>> For routes received via VPN to the primary hub, set those to even 
>> lower
>> localpref, say 800.
>> 
>> For VPN to the secondary hub, set those to lowest localpref, say 700.
>> 
>> If you have any back-door links between sites, like Metro Ethernet,
>> switched Ethernet, or optical waves, integrate those by using BGP as
>> well. Announce locally originated routes with a higher local 
>> preference
>> than the MPLS networks, say, 1100.
>> 
>> But be careful sharing WAN routes learned via MPLS across a backdoor
>> link. Don't set those to a higher local preference than the MPLS 
>> network
>> itself. Avoid if possible.
>> 
>> Also, don't forget about the key management aspects for routers: 
>> syslog,
>> AAA / TACACS, availability and performance monitoring (SNMP), NTP,
>> Netflow, and FTP repository for router code. Use a loopback IP on each
>> router for your management traffic. Avoid SFTP or SCP to your code 
>> repo
>> due to known performance issues with OpenSSH and file transfers.
>> 
>> Consider setting up out-of-band access. Best is to build a small
>> parallel WAN/LAN over VPN with a firewall, LAN switch, and serial
>> console server. Get a separate Internet connection for the firewall
>> (business broadband works), and use it only for out-of-band access.
>> 
>> If you decide to configure QOS over the WAN, try to use standard 
>> packet
>> markings like EF for voice media and AF31 for signaling. Check with 
>> your
>> providers on what markings they support. The less re-marking of 
>> traffic
>> you have to do, the better.
>> 
>>> Regards,
>>> 
>>> Atif.
>> 
>> Hope all this made sense. I'm happy to clarify anything.
>> 
>> This config served me well in my enterprise days and later on as an 
>> NSP
>> integrator. 25 years of experience in those spaces.
>> 
>> Hope this helps,
>> 
>> -Brian