[j-nsp] Opinions on fusion provider edge

[James Bensley]

On Wed, 7 Nov 2018 at 13:03, Antti Ristimäki <antti.ristimaki at csc.fi> wrote:
> Wrt the original question about possible issues with Fusion, we have faced quite a many. Currently one of the biggest pains is to get CoS configured properly on Fusion ports. We have a case open, where any CoS scheduler change stops traffic forwarding out from the cascade port, if one has explicitly configured schedulers for the cascade port logical control (32769) and data (32770) units. This is pretty irritating, as traffic between the extended customer facing port and the CE device works just fine, keeping e.g. BGP up and running, but traffic to/from the core does not work.
>
> I'm also somewhat concerned about the fact that the whole Fusion thing is more or less a black box and as such much more difficult to debug than traditional technologies.
>
> From monitoring point of view it also a bit challenge that not all information related to satellite ports is not available via SNMP. E.g. queue specific counters are not available but have to be queried via CLI command, and IIRC also ifOutDiscards is not recorded for the extended ports.

My experiences with SP Fusion so far have been to add more ports to
routers in a cost effective manner. Filling a chassis with 1G/10G
ports isn't super cost efficient as line cards are expensive and ports
are rarely run near 100% utilisation for an extended period of time,
so it makes for a poor ROI. At $job-1 we went down the road of using
SP Fusion as a layer 1 extension to PEs, using a 40G uplinks to the
router as the aggregation link. Operators have been doing this for
years using dumb layer 2 switches as a layer 2 extension. There is
nothing wrong with layer 2 aggregation switches in my opinion, the
only technical advantage in my opinion to using SP Fusion for a layer
1 extension to a router compared to a layer 2 switch is that SP Fusion
is one device to configure and monitor instead of two. Unless we had
deployed thousands of aggregation + satellite devices it's not really
having any major positive impact on my monitoring licensing costs
thought. Equally when using a typical router + layer 2 switch
extension,  the config that goes into the layer 2 switch is so basic,
touching two devices instead of one again seems like a negligible
disadvantage to me.

The benefit we had from SP Fusion is that, and I'm guessing here, is
that Juniper wanted guinea pigs; they sold us QFX5100s as SP Fusion
devices plus line cards for the MX's for cheaper than we could buy
line cards + EXs, and guinea pigs we were. It took quite a bit of
effort to get the QFXs onto the correct code version in stand alone
mode. We also had to upgrade our MXs to 17.1 (this was not long after
it's release) and use the then new RE-64s because we needed HQoS over
Fusion and this was the only way it was supported. It was then more
hassle to get the QFXs to migrate into Fusion mode and download their
special firmware blob from the MXs. We had to get JTAC to help us and
even they struggled. Another issue is that we were a heavy users of
Inter-AS MPLS option B's and they aren't supported over SP Fusion
links. There is technically no reason why it wouldn't work, as Fusion
is a layer 1 extension, however, Inter-AS Opt B isn't one of the
features they test when releasing new Fusion code versions, so it's
officially unsupported, so we still had to deploy EX's for Opt B
links.

A colleague of mine worked on a separate project which was a DC Fusion
deployment and similar issues and it took him a lot of headache and
JTAC assistance to get that deployment working.

In my current $job we have/had a QFX switch stack in the office (so
nothing to do with Fusion) at that has been very troublesome. As per
some of the other threads on this list we've had lots of problems with
QFX switches and certain optics not working, either in stacked mode or
on certain code versions. Again, this went to JTAC, they couldn't fix
it, eventually we fixed it by trying various different code versions
and breaking the stack out.

So overall, not impressed with the QFX5100s at all.

Cheers,
James.