[j-nsp] Segment Routing Real World Deployment (was: VPC mc-lag)

[Mark Tinka]

On 10/Jul/18 00:46, Pavel Lunin wrote:


>
>
> I have no doubt that you know how to run MPLS in the access smoothly.
> However, choosing the right gear for this role has always been a hard
> job. Those folks who chose Brocade CES some 5-7 years ago, where are
> they now?
>
> The problem is that most real-world networkers have not enough
> understanding of MPLS internals, or time, or both to check all those
> hardware and software limits and rather look at the vendor's specs in
> terms of "supported/not supported". This approach works _relatively_
> well in many cases like choosing a classic switch or a firewall or
> even an MX/ASR-like full-feature PE. But for the MPLS in the access
> you need to tear the guts out of your vendor, test everything yourself
> in all possible scenarios and still be extremely suspicious about
> every single thing. Moreover a lot of people have some
> commercial/political limits in choosing hardware.
>
> So, while MPLS in the access looks like a good idea, and there are
> people who manage to run it well, I know more failure than success
> stories.

Truth!

I was there, back in 2008, beating the guts out of Juniper, Cisco and
Brocade, to build me a box that had a 1U form factor, had between 20 -
40 1Gbps ports, supported IP/MPLS in full, and was priced around
US$5,000 - US$8,000 per unit. Boy, was that an experience. I needed to
migrate a network that was carrying nearly 6,000 l2vpn/l3vpn services
from classic Huawei- and Cisco-driven 802.1Q-based Metro-E to IP/MPLS.

For the existing platforms in 2008:

  * The Cisco ME6500 was more of a switch than a router.

  * The Cisco 3750ME was more of a switch than a router, although it had
    some MPLS tendencies.

  * The MX (960 and 480, at the time), were the right gear, but too big
    and pricey.

By the start of 2009:

  * Juniper promised they'd have a 1U, 48-port version of the MX80. I
    was happy.

  * Cisco had shipped me a test board (no protective chassis at the
    time) of the ME3600X.

  * I began testing the Brocade CES/CER2000 NetIron.

By the end of 2009:

  * Juniper disappointed me with their lack of vision on where things
    were going in the Metro. When I say that the MX204 is their first
    real attempt at having something that can work in the Metro, I'm not
    just talking :-)...

  * We dumped the NetIron because while it was working well, and had a
    large FIB for the period (512,000 IPv4 entries), we wanted to stick
    with 2 vendors. They also didn't support QPPB, which we wanted at
    the time.

  * We settled on the ME3600X. Because we insisted on QPPB, we managed
    to convince Cisco to re-spin the Nile ASIC before the box was FCS.

If I'm not mistaken, we were the first network in the world, at the
time, to go full IP/MPLS in the Access, at a large scale (some 100+
nodes by the end of 2013).

So yes, the battle was long and hard-fought. I suppose the biggest
hurdle to overcome was the temptation to slip back into classic
802.1Q-based Metro-E deployments, just with newer kit. I spent a whole
year convincing the business that that was the wrong approach. Wasn't
easy, but in 2018, I believe that operation is now several hundred units
large (on the ASR920, I believe), and managing thousands of l2vpn/l3vpn
Enterprise customers.


>
> Good question, indeed. In my opinion there are still a lot of folks
> out there who build DC networks with vPC, FEX, VirtualChassis, Fusion
> etc, which is finally the old good VLANs in a vendor packaged black
> magic box. Sooner or later those VLANs need to go across multiple
> sites. It's nearly improbable that having such a design, you'll mange
> to build a EVPN-VXLAN-hipster-buzz-based DCI. So VPLS is still their
> best friend. I've seen some of them who understand that it's evil, and
> some who believe that it's OK, both had no choice.
>
> However my original point was rather about pseudo-wires than VPLS. I
> mean, I don't see a lot of pseudo-wires in the wild. Mostly because PW
> is a kind of hard to sell. Customers can be of two types: those who
> love Metro Ethernet and those who don't. It's true for real customers,
> whose requirements are amplified by the sales people, and internal
> infrastructure folks.
>
> Those who love L2 because "it's better and easier" usually don't know
> what a pseudowire is. And they just don't care. "Like a switch" is
> what they are looking for.
>
> Those who avoid metro-ethernet just don't need pseudowires, certainly
> automesh Kompella-style. L3VPN works well for them, or they buy L1
> between their routers, or go EVPN.
>
> A pseudo-wire is a kind of side application in my experience, even
> though technically it's simple and powerful. Not that it doesn't exist
> as a commercial service, but mostly used for internal infrastructure
> needs on an occasional basis.
>
> So I tend to think, that if your business can make money out of
> pseudo-wires, it's not about your network design, you are just lucky ;)

In our market (primarily Africa, and parts of Europe as end points),
customers that require point-to-point connectivity are divided into 2
categories; those that are technically-astute, and those that only know
enough to get what they want.

The technical customers will be concerned about whether the transport is
MPLS- or DWDM-based, whether it's Ethernet or SDH, how policing and
queuing is done, e.t.c. The non-technical customers will just want a
scalable point-to-point service, across a medium that they can afford,
which tends to be Ethernet.

Generally, customers interested in DWDM would be those that need
anything over 10Gbps, or are the type that have a strong desire to run a
"clean" backbone, even if that's not their core business.

Customers that take our EoMPLS service have never been interested in
whether it's provisioned via LDP, BGP, VPLS, EVPN, e.t.c. And this
includes a bunch of well-known global companies that know their IP/MPLS.

Maybe it's just our side of the seas.

Mark.