[j-nsp] Link establishment issues with 1Gbps SX/LX SFPs on QFX5110

[Shamen Snyder]

What version of code are you using? When I worked at Juniper I delt with a
ISP that had this issue and I believe 17.3R3 did not have the issue at all.

There was an issue with BCM and the Juniper code. You can manually force
the port up in the BCM shell (not that is a viable long term solution).

FPC0( vty)# set dcbcm bcmshell "ps 45"

HW (unit 0)
                 ena/    speed/ link auto    STP                  lrn
 inter   max  loop
           port  link    duplex scan neg?   state   pause  discrd ops
face frame  back
       ge0( 45)  down    1G  FD   HW  Yes    Block          None   FA
 SGMII  1518

FPC0( vty)# set dcbcm bcmshell "port 45 an=1 e=t"

HW (unit 0)

FPC0( vty)# set dcbcm bcmshell "ps 45"

HW (unit 0)
                 ena/    speed/ link auto    STP                  lrn
 inter   max  loop
           port  link    duplex scan neg?   state   pause  discrd ops
face frame  back
       ge0( 45)  !ena    1G  FD None  No     Block          None    D
 SGMII  9412




On Tue, Jun 25, 2019 at 1:15 AM Timothy Creswick <timothy at creswick.eu>
wrote:

> > Can confirm I've seen exactly the same issue on QFX5110 18.4R1.8 with
> the only way I can get fs.com LX 1G
> > transceivers working being turning off auto-neg.
>
> Thanks. We've been looking at the Broadcom tools on the QFX and note some
> apparently quite major differences in the way that the interfaces negotiate
> across versions.
>
> In the older satellite verions (e.g. 2.0R1.1), GE interfaces appear to
> have the correct auto neg state set on the chipset. This seems to coincide
> with it using GMII between the SFP and the board. This reflects the way the
> ports are configured in Junos:
>
>             ena/    speed/ link auto    STP                  lrn  inter
>  max  loop
>       port  link    duplex scan neg?   state   pause  discrd ops   face
> frame  back
>        xe4  up      1G  FD   HW  Yes  Forward          None    F   GMII
> 1604
>        xe6  up      1G  FD   HW  No   Forward          None    F   GMII
> 1522
>       xe10  up      1G  FD   HW  Yes  Forward          None    F   GMII
> 1526
>       xe11  up      1G  FD   HW  Yes  Forward          None    F   GMII
> 1526
>       xe12  up      1G  FD   HW  Yes  Forward          None    F   GMII
> 1526
>
> On newer versions (e.g. 3.5R2.2), GE interfaces are always shown as
> auto-neg disabled on the chipset:
>
>                  ena/    speed/ link auto    STP                  lrn
> inter   max  loop
>            port  link    duplex scan neg?   state   pause  discrd ops
>  face frame  back
>        ge0(  3)  up      1G  FD   HW  No   Forward          None    F
> SGMII  1518
>        ge1(  4)  up      1G  FD   HW  No   Forward          None    F
> SGMII  1518
>        ge2(  6)  down    1G  FD   HW  No   Forward          None    F
> SGMII  1518
>        ge3( 12)  !ena    1G  FD   HW  No   Forward          None    F
> SGMII  1518
>        ge4( 13)  up      1G  FD   HW  No   Forward          None    F
> SGMII  1526
>        ge5( 15)  up      1G  FD   HW  No   Forward          None    F
> SGMII  1526
>
> They will show as up here and in Junos but not on the remote device.
>
> Forcefully setting the interface to '[gig]ether-options auto-negotiate'
> does not change the bcm setting.
>
> If you forcefully change the bcm setting (i.e. tell the chipset to
> auto-neg), the port goes down both in the bcm shell and in Junos.
>
> Interestingly, some of these ports work, others do not. We haven't been
> able to identify a pattern and JTAC are still looking.
>
> We also can't explain why these ports indicate they are SGMII.
>
> At the behest of JTAC we put these on 3.5R1-S4.0. This has the
> characteristic that in the following example we have a MM (port 1) and SM
> (port 3) transceiver. Both are Flexoptix correctly coded (we've tried
> others). Port 1 gets link successfully, Port 3 does not. Note that now the
> working port is back to GMII:
>
>                  ena/    speed/ link auto    STP                  lrn
> inter   max  loop
>            port  link    duplex scan neg?   state   pause  discrd ops
>  face frame  back
>        ge0(  1)  up      1G  FD   HW  Yes  Forward          None    F
>  GMII  1518
>        ge1(  3)  down   10M  FD   HW  Yes  Forward          None    F
> SGMII  1518
>
> Can anyone shed any more light on this?
>
> Regards,
>
> Tim
> _______________________________________________
> juniper-nsp mailing list juniper-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>