[j-nsp] Network, trouble after customer created a loop *inside* a VM host

[Michele Bergonzoni]

>            |
>      ProCurve 2824    RSTP bridge id 32k
>            |
>        Windows Host
...

> Because of a mistake, the customer accidentally bridged his two VMs
> together as well which caused a loop inside the Host. So far, so good.
>
> The trouble begins at this point because immediately we saw partial
> network outages resulting in router messages like this:
>
> Nov  7 14:30:47  cr0 l2ald[2545]: L2ALD_MAC_MOVE_NOTIFICATION: MAC Moves
> detected in the system

VMware vswitches, as far as I know, simply drop any received BPDUs. So a 
loop, even an internal one among VMs, cannot result in the upstream 
switch (the Procurve 2824) seeing its own BPDU, a condition that most 
switches would detect as a looped port (like an old type-1 IBM cable) 
that they put in blocking or similar state (this is not true for older 
Procurves, due to a bug, but AFAIK the 2824 is not old enough anyway).

So STP didn't protect you and you faced the loop.

When the loop occurs, frames that your switches send down keep coming 
back up, and you see MAC flaps. The 2824 is not smart enough to warn you 
about that.

I did a VM loop myself once, when I tried to use a Linux VM as a VTEP 
for VXLAN (at least I think it was a loop, for sure it was service 
disruption).

I think it is hard to protect from this kind of event. Broadcast were 
probably not enough to trigger broadcast control, but enough to hog your 
CPU with reflected ARP and other control plane packets. You can probably 
mitigatae it by rate-limiting packets to the control plane (RFC6192).

You might try to use PVST in your EXs, if they support it, because it 
uses a different MAC, but I bet the vswitch would drop that as well.

Some switch vendor have proprietary protocols that sends multicast out a 
port and listen for it coming back. Cisco does that, and you can disable 
it with "no keepalive", but I don't know if the vswitch drops this as 
well, or forwards it as a normal multicast.

You can get a small PC, hook an interface to that VLAN, write a program 
to do your keepalive-like functionality (with a perfectly normal 
multicast MAC), and hook it to a script to disable the port when that 
frame is received. This would be of course very, very inelegant.

I am curious to read what others will suggest.

Regards,
                   Bergonz

-- 
Ing. Michele Bergonzoni - Laboratori Guglielmo Marconi S.p.a.
Phone:+39-051-6781926 e-mail: bergonz at labs.it
alt.advanced.networks.design.configure.operate