[j-nsp] 40G QSFP problems on QFX5100 after 16.1R6

[Chris]

Hi,

On 24/04/2018 3:16 PM, Sebastian Wiesinger wrote:
> Hello,
> 
> we've noticed problems with third party vendors QSFP 40G optics after
> upgrading our JunOS on QFX5100. The problems manifest as a general
> instablility on the QSFP links with symptoms like:
> 
> * Links take minutes to come up
> * Links go down randomly
> * Links show CRC/Align errors and packets get dropped

Yes, I have 10 QFX5100-48S and I have been experiencing the same issues.

All 10 devices have third party QSFP+ optics/DAC's (fs.com coded for 
Juniper). So far I have had similar issues to you but not in all cases:

* 4 of the QFX devices are on 16.1R3. These 4 devices each have 1 x 
QSFP+-40G-LR4 and 2 x QSFP+-40G-CU3M. I have not had any issues at all 
with these.

* 4 of the QFX devices are on 17.3R1.
	- 2 devices with QSFP+-40G-LR4: These are what we have been mainly 
experiencing issues with. Initially these had QSFP+40GE-IR4 optics and I 
blamed the issues on the length of the fibre run being a problem (it was 
quite close to the link budget). One of the links was flapping, and 
currently we are having a problem where we see CRC/Align errors. The 
optics are verified to be good testing in other equipment with the same 
patch cables.
	- 2 of the devices with QSFP+-40G-CU1M: No issues.
	- 2 of the devices with QSFP+40GE-IR4: No issues.
	- 2 of the devies with QSFP+-40G-CU5M: No issues.

* 2 of the QFX devices were on 17.3R1 but I have upgraded them to 18.1R1 
yesterday. Before the upgrade I had some problems where certain traffic 
wasn't working when it cross the virtual chassis (the virtual chassis 
connection is over a pair of QSFP+-40G-CU5M links). Simply disabling the 
virtual chassis port then enabling it again one by one fixed the 
problem. The problem reoccured after a while so I elected to try 
upgrading to 18.1R1 to see if it made any difference, that specific 
problem has not occured since. I opened a JTAC case for the specific 
problem with certain traffic not working and didn't get anywhere - I was 
told to reboot the device which I said is not acceptable.
	- Both devices have QSFP+-40G-CU5M (virtual chassis): This had the 
issue noted above.
	- Both devices have QSFP+-40G-LR4: These have the same issue with 
traffic not working in some cases, or traffic will be super slow.

I can't keep switching firmware around to try and resolve this/isolate 
to a specific revision, but it is interesting that you also note you 
have not experienced any issues with 16.1, the same as us. If you get a 
proper answer to what this issue is I would really like to know, but it 
looks like I will probably have to downgrade to 16.1 due to these issues 
as they are impacting services.

I have just ordered some EX4600's for a new office fitout along with 
some 40G DAC's and 40G QSFP+ interfaces from fs.com as well. I am 
curious to see if I have the same issues with those, I suspect that 
would be a yes.

Thanks