[j-nsp] L3VPNs and on-prem DDoS scrubbing architecture

[Saku Ytti]

On Tue, 2 Apr 2024 at 18:25, Michael Hare via juniper-nsp
<juniper-nsp at puck.nether.net> wrote:

> We're a US research and education ISP and we've been tasked for coming up with an architecture to allow on premise DDoS scrubbing with an appliance.   As a first pass I've created an cleanL3VPN routing-instance to function as a clean VRF that uses rib-groups to mirror the relevant parts of inet.0.   It is in production and is working great for customer learned BGP routes.  It falls apart when I try to protect a directly attached destination that has a mac address in inet.0.  I think I understand why and the purpose of this message is to see if anyone has been in a similar situation and has thoughts/advice/warnings about alternative designs.
>
> To explain what I see, I noticed that mac address based nexthops don't seem to be copied from inet.0 into cleanL3VPN.inet.0.  I assume this means that mac-address based forwarding must be referencing inet.0 [see far below].   This obviously creates a loop once the best path in inet.0 becomes a BGP /32.  For example when I'm announcing a /32 for 1.2.3.4 out of a locally attached 1.2.3.0/26, traceroute implies the packet enters inet.0, is sent to 5.6.7.8 as the nexthop correctly, arrives in cleanL3VPN which decides to forward to 5.6.7.8 in a loop, even though the BGP /32 isn't part of cleanL3VPN [see below], cleanL3VPN Is dependent on inet.0 for resolution.  Even if I could copy inet.0 mac addresses into cleanL3VPN, eventually the mac address would age out of inet.0 because the /32 would no longer be directly connected.  If I want to be able to protect locally attached destinations so I think my design is unworkable, I think my solutions are

If I understand you correctly, the problem is not that you can't copy
direct into CleanVRF, the problem is that ScrubberPE that does clean
lookup in in CleanVRF, has label stack of [EgressPE TableLabel],
instead of [EgressPE EgressCE], this causes the EgressPE to do IP
lookup, which will then see the Direct/32 advertised by the scrubber,
causing loop. While what you want is end-to-end MPLS lookup, so that
egressPE MPLS lookup has egressMAC.

I believe in BGP-LU you could fix this, without actually paying for
duplicate RIB/FIB and without opportunistically copying routes to
CleanVRF, every prefix would be scrubbable by default. You'd have
per-ce for rest, but per-prefix for connected routes, I believe then
you would have [EgressPE EgressMAC_CE] label for connected routes, so
each host route would have their own label, allowing mac rewrite
without additional local IP lookup.

I'm not sure if this is the only way, I'm not sure if there would be a
way in CleanVRF to force each direct/32 to have a label as well,
avoiding the egress IP lookup loops. One doesn't immediately spring to
mind, but technically implementation could certainly allow such mode.

-- 
  ++ytti