[outages] HE FMT2 down, various network foo eminating

[William R. Lorenz]

On Sat, 26 Sep 2009, Jeremy Chadwick wrote:

> I will never forget how HE refused to use VLANs to segregate customers 
> on a layer 2 level, instead preferring some strange layer 3 
> implementation.  When we witnessed an unexpected massive (7-8mbit/sec) 
> increase in inbound traffic, only to find that the destination IPs of 
> these packets were for another customer in a completely different 
> netblock/area of the Fremont facility, we were told by support "that's 
> impossible".  Full tcpdump captures were given, and we were told "this 
> makes no sense, this can't happen".  4-5 hours later, we were told the 
> root cause was "a customer who had misconfigured their load balancer".

This will happen if using Cisco's private VLANs (PVLANs) in an isolated or 
community mode on the switch.  Because of the way most load balancers work 
in the layer 2 environment, the traffic ends up being broadcast to all 
switch ports in the group.  My guess is they had customers bridged over to 
the next-hop using PVLANs, so that customers were theoretically isolated 
at a layer 2 level.  Of course, PVLANs don't provide any true isolation.

This is just speculation about their setup based on what you describe, but 
I've seen this in practice with PVLANs, and it seems it to be plausible.

-- 
William R. Lorenz