[j-nsp] Link establishment issues with 1Gbps SX/LX SFPs on QFX5110

[Timothy Creswick]

> Can confirm I've seen exactly the same issue on QFX5110 18.4R1.8 with the only way I can get fs.com LX 1G
> transceivers working being turning off auto-neg.

Thanks. We've been looking at the Broadcom tools on the QFX and note some apparently quite major differences in the way that the interfaces negotiate across versions.

In the older satellite verions (e.g. 2.0R1.1), GE interfaces appear to have the correct auto neg state set on the chipset. This seems to coincide with it using GMII between the SFP and the board. This reflects the way the ports are configured in Junos: 

            ena/    speed/ link auto    STP                  lrn  inter   max  loop
      port  link    duplex scan neg?   state   pause  discrd ops   face frame  back
       xe4  up      1G  FD   HW  Yes  Forward          None    F   GMII  1604 
       xe6  up      1G  FD   HW  No   Forward          None    F   GMII  1522 
      xe10  up      1G  FD   HW  Yes  Forward          None    F   GMII  1526 
      xe11  up      1G  FD   HW  Yes  Forward          None    F   GMII  1526 
      xe12  up      1G  FD   HW  Yes  Forward          None    F   GMII  1526

On newer versions (e.g. 3.5R2.2), GE interfaces are always shown as auto-neg disabled on the chipset:

                 ena/    speed/ link auto    STP                  lrn  inter   max  loop
           port  link    duplex scan neg?   state   pause  discrd ops   face frame  back
       ge0(  3)  up      1G  FD   HW  No   Forward          None    F  SGMII  1518      
       ge1(  4)  up      1G  FD   HW  No   Forward          None    F  SGMII  1518      
       ge2(  6)  down    1G  FD   HW  No   Forward          None    F  SGMII  1518      
       ge3( 12)  !ena    1G  FD   HW  No   Forward          None    F  SGMII  1518      
       ge4( 13)  up      1G  FD   HW  No   Forward          None    F  SGMII  1526      
       ge5( 15)  up      1G  FD   HW  No   Forward          None    F  SGMII  1526      

They will show as up here and in Junos but not on the remote device.

Forcefully setting the interface to '[gig]ether-options auto-negotiate' does not change the bcm setting.

If you forcefully change the bcm setting (i.e. tell the chipset to auto-neg), the port goes down both in the bcm shell and in Junos.

Interestingly, some of these ports work, others do not. We haven't been able to identify a pattern and JTAC are still looking. 

We also can't explain why these ports indicate they are SGMII.

At the behest of JTAC we put these on 3.5R1-S4.0. This has the characteristic that in the following example we have a MM (port 1) and SM (port 3) transceiver. Both are Flexoptix correctly coded (we've tried others). Port 1 gets link successfully, Port 3 does not. Note that now the working port is back to GMII:

                 ena/    speed/ link auto    STP                  lrn  inter   max  loop
           port  link    duplex scan neg?   state   pause  discrd ops   face frame  back
       ge0(  1)  up      1G  FD   HW  Yes  Forward          None    F   GMII  1518
       ge1(  3)  down   10M  FD   HW  Yes  Forward          None    F  SGMII  1518

Can anyone shed any more light on this?

Regards,

Tim