[j-nsp] QFX DDOS Violations

[Saku Ytti]

Hey,

Before any potential trashing, I'd like to say that as far as I am
aware Juniper (MX) is the only platform on the market which isn't
trivial to DoS off the network, despite any protection users may have
tried to configure.

> How do you identify the source problem of DDOS violations that junos logs
> for QFX? For example what interface that is causing the problem?

I assume you are talking about QFX10k with Paradise (PE) chipset. I'm
not very familiar with it, but I know something about it when sold in
PTX10k quise, but there are significant differences. Answers are from
the PTX10k perspective. If you are talking about QFX5k many of the
answers won't apply, but the ukern side answers should help
troubleshoot it further, certainly with QFX5k the situation is worse
than it would be on QFX10k.

> DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for
> protocol/exception  VXLAN:aggregate exceeded its allowed bandwidth at fpc 0
> for 30 times, started at...
>
> The configured rate for VXLAN is 500pps, ddos protection is seeing rates
> over 150 000pps

Do you mean you've configured:
'set system ddos-protection protocols vxlan aggregate bandwidth 500'.
What exactly are you seeing? What does 'show ddos-protection protocols
vxlan' say?Also 'start shell pfe network fpcX' + 'show ddos scfd
proto-states vxlan'

Paradise (unlike Triton and Trio) does not support PPS policing at
all. So when you configure a PPS policer, what actually gets
programmed is 500pps*1500B bps. I've tried to argue this is a poor
default, 64B being superior choice.
In paradise 500pps would admit 500*(1500/64) or about 12kpps per
Paradise if those VXLAN packets were small. These would then be
policed by the LC CPU ukern into 500 pps for all the Paradise chips
living inside that LC CPU, before sending to RE over bme0.
After DDoS but before Paradise admits packet to the LC_CPU it goes
through VoQ, where most packets are classified as VoQ#2 which is
10Mbps wide with no burstability (classification, width and
burstability is being changed on later images). So extremely trivial
rates will cause congestion on the VoQ#2 and a lot of protocols will
be competing for 10Mbps access to LC CPU, like BGP, ISIS, OSPF, LDP,
ND, ARP.

> This is an spine/leaf setup, one theory is that the vxlan traffic that most
> of our QFX boxes are activation ddos protection for is actually vxlan
> services running inside the vxlans, for example we have kubernetes clusters
> using vxlan. Is that a sane theory?

Not enough information to speculate.
In many cases ddos classification is wrong. You can review in the PFE,
'show filter' => HOSTBOND_IPv4_FILTER then 'show filter index X
program'. You can also capture punted packets on interface where RE
meets FPC (I think bme0 here), in the bme0 interface TNP headers are
in top of the punted packets and in the TNP headers you will see what
ddos classification was used, you can turn the number into name by
looking at the 'show ddos scfd proto-statates'.


I naively wish I could set my ddos-protocol classification and voq
classification manually in 'lo0 filter', because the infrastructure
allows for great protection, but particularly when choosing which VoQ
packets share there is no obvious single best solution, it depends on
the environment. Like I could put RSVP, ISIS, LDP on single VoQ, as
they never compete with customers, BGP in another as they will compete
with customers and operators for me, and so forth. But of course this
wish is naive, as the solution the vendor offers is already too
complex for customers to use and giving more rope would just make the
mean config worse.

-- 
  ++ytti