[c-nsp] Metro Ethernet nightmares (L2PT, PBB-VPLS, Load Balancing, EVPN)

[Holger L]

Hello everybody,

We are trying to set up a Metro Ethernet over 4 POPs using MEF Standards
and facing some nightmares here.

Our Target:
We like to provide E-LAN and E-Line services (on EPL, EVPL) to our
customers, as transparent as possible, which means CDP, VTP, STP, RSTP,
PVST and RPVST should pass. Resiliency in our core network should kick in
below 200ms.

Networkdiagram:
   CustA---CPE1---R1---R2---CustB
                  |     |
          CustB---R4---R3---CPE2---CustA

Hardware:
Core Router R 1-4:
    Cisco 7604, RSP720-3c-10GE, 76-ES+T-4TG, 76-ES+T-40G, IOS 15.3.1-S1
CPE 1-2:
    Cisco ME3400E, IOS 12.2.58-SE2

I tell you the whole story because I'll be glad for any idea or feedback.

Our  first try we gave VPLS with MPLS-TE using Cisco ME3800 as cores,
which was a complete disaster but thats another very loong story. Now we
got 7600 core routers, which should do the job.

So, on the 76ers we set up OSPF + MPLS-TE + VPLS for the core and Q-in-Q
for the access network between CPEs and core routers. We used L2PT on all
customer ports, but this was not as transparent as we suspected. L2PT
behaves different on different Cisco hardware and produces inconsistency
throughout the network and results in dropped CDP, VTP, STP, RSTP, PVST
etc.

To solve this we switched to PBB-VPLS because PBB proved to be completely
transparent throughout our core network. Anyways, since we still had to
get customers traffic transparent to the core network we made a small hack
and aktivated L2PT on CPEs customer ports and on CPEs uplink port. This
means customers traffic gets L2PT "tunneled" through the CPE and is back
to normal on the uplink, accept that it is Q-in-Q tagged. Right now I
don't feel good on this hack but Cisco didn't object on it.
Do you have a better idea how to get customer traffic transparent through
the CPEs?

Next we wanted to upgrade our core node links from 10Gbps to 20Gbps using
the unused interfaces on our ES+T-4TG Linecards.
   R1===R2
   ||   ||
   R4===R3

First, I tried to etherchannel them together but faced the problem that
there is no suitable load balancing hash. It can't be balanced by MAC
because in PBB you only see core node macs, which would produce bad load
balancing by 4 different MACs. Neither it can be balanced by MPLS label
because in PBB all ISIDs are transfered in one bridge-domain, thus in one
vfi and results in only one label to reach the next-hop, which produces
very bad load balancing.

Second, I tried to use RSVP for load balancing and created two primary
MPLS-TE tunnels between every two routers.
R1 Te2/1----[Tunnel R1-R2-1]----Te2/3 R2
R1 Te2/2----[Tunnel R1-R2-2]----Te2/4 R2
...
Now I got two equal routes for each next-hop in the routing tables and
made some tests, well, without success. It seems RSVP load balances by
MPLS label too (correct me if I am wrong) and as already mentioned
PBB-VPLS results in just a few lables and produces bad load balancing.
Any idea how to solve this?

Yesterday I read something about EVPN, PBB-EVPN, BGP-EVPN which might
solve all our trouble, but I neither find any configuration guides nor
anything about EVPN in the feature navigator. Do you know something about
this feature and how to configure it?

Could someone please give me a push in the right direction on all this?
Any help or idea is highly appreciated.

Best Regards,
Holger