[j-nsp] [c-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco

james list jameslist72 at gmail.com
Sun Feb 11 08:24:34 EST 2024


On Cisco I see physical goes down (initializing), what does that mean?

While on Juniper when the issue happens I always see:

show log messages | last 440 | match LACPD_TIMEOUT
Jan 25 21:32:27.948 2024  MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Jan 26 18:41:12.514 2024  MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Jan 28 05:07:20.283 2024  MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Jan 29 04:06:51.768 2024  MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Jan 30 03:09:43.923 2024  MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Feb  5 18:13:20.158 2024  MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Feb  6 02:17:23.703 2024  MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Feb  6 22:00:23.758 2024  MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Feb  9 09:29:35.728 2024  MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Feb  9 16:39:35.813 2024  MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT

Il giorno dom 11 feb 2024 alle ore 14:10 Saku Ytti <saku at ytti.fi> ha
scritto:

> Hey James,
>
> You shared this off-list, I think it's sufficiently material to share.
>
> 2024 Feb  9 16:39:36 NEXUS1
> %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface
> port-channel101 is down (No operational members)
> 2024 Feb  9 16:39:36 NEXUS1 %ETH_PORT_CHANNEL-5-PORT_DOWN:
> port-channel101: Ethernet1/44 is down
> Feb  9 16:39:35.813 2024  MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5:
> lacp current while timer expired current Receive State: CURRENT
> Feb  9 16:39:35.813 2024  MX1 lacpd[31632]: LACP_INTF_DOWN: ae49:
> Interface marked down due to lacp timeout on member et-0/1/5
>
> We can't know the order of events here, due to no subsecond precision
> enabled on Cisco end.
>
> But if failure would start from interface down, it would take 3seconds
> for Juniper to realise LACP failure. However we can see that it
> happens in less than 1s, so we can determine the interface was not
> down first, the first problem was Juniper not receiving 3 consecutive
> LACP PDUs, 1s apart, prior to noticing any type of interface state
> related problems.
>
> Is this always the order of events? Does it always happen with Juniper
> noticing problems receiving LACP PDU first?
>
>
> On Sun, 11 Feb 2024 at 14:55, james list via juniper-nsp
> <juniper-nsp at puck.nether.net> wrote:
> >
> > Hi
> >
> > 1) cable has been replaced with a brand new one, they said that to check
> an
> > MPO 100 Gbs cable is not that easy
> >
> > 3) no errors reported on both side
> >
> > 2) here the output of cisco and juniper
> >
> > NEXUS1# sh interface eth1/44 transceiver details
> > Ethernet1/44
> >     transceiver is present
> >     type is QSFP-100G-SR4
> >     name is CISCO-INNOLIGHT
> >     part number is TR-FC85S-NC3
> >     revision is 2C
> >     serial number is INL27050TVT
> >     nominal bitrate is 25500 MBit/sec
> >     Link length supported for 50/125um OM3 fiber is 70 m
> >     cisco id is 17
> >     cisco extended id number is 220
> >     cisco part number is 10-3142-03
> >     cisco product id is QSFP-100G-SR4-S
> >     cisco version id is V03
> >
> > Lane Number:1 Network Lane
> >            SFP Detail Diagnostics Information (internal calibration)
> >
> >
> ----------------------------------------------------------------------------
> >                 Current              Alarms                  Warnings
> >                 Measurement     High        Low         High          Low
> >
> >
> ----------------------------------------------------------------------------
> >   Temperature   30.51 C        75.00 C     -5.00 C     70.00 C
> 0.00 C
> >   Voltage        3.28 V         3.63 V      2.97 V      3.46 V
> 3.13 V
> >   Current        6.40 mA       12.45 mA     3.25 mA    12.45 mA
>  3.25
> > mA
> >   Tx Power       0.98 dBm       5.39 dBm  -12.44 dBm    2.39 dBm
>  -8.41
> > dBm
> >   Rx Power      -1.60 dBm       5.39 dBm  -14.31 dBm    2.39 dBm
> -10.31
> > dBm
> >   Transmit Fault Count = 0
> >
> >
> ----------------------------------------------------------------------------
> >   Note: ++  high-alarm; +  high-warning; --  low-alarm; -  low-warning
> >
> > Lane Number:2 Network Lane
> >            SFP Detail Diagnostics Information (internal calibration)
> >
> >
> ----------------------------------------------------------------------------
> >                 Current              Alarms                  Warnings
> >                 Measurement     High        Low         High          Low
> >
> >
> ----------------------------------------------------------------------------
> >   Temperature   30.51 C        75.00 C     -5.00 C     70.00 C
> 0.00 C
> >   Voltage        3.28 V         3.63 V      2.97 V      3.46 V
> 3.13 V
> >   Current        6.40 mA       12.45 mA     3.25 mA    12.45 mA
>  3.25
> > mA
> >   Tx Power       0.62 dBm       5.39 dBm  -12.44 dBm    2.39 dBm
>  -8.41
> > dBm
> >   Rx Power      -1.18 dBm       5.39 dBm  -14.31 dBm    2.39 dBm
> -10.31
> > dBm
> >   Transmit Fault Count = 0
> >
> >
> ----------------------------------------------------------------------------
> >   Note: ++  high-alarm; +  high-warning; --  low-alarm; -  low-warning
> >
> > Lane Number:3 Network Lane
> >            SFP Detail Diagnostics Information (internal calibration)
> >
> >
> ----------------------------------------------------------------------------
> >                 Current              Alarms                  Warnings
> >                 Measurement     High        Low         High          Low
> >
> >
> ----------------------------------------------------------------------------
> >   Temperature   30.51 C        75.00 C     -5.00 C     70.00 C
> 0.00 C
> >   Voltage        3.28 V         3.63 V      2.97 V      3.46 V
> 3.13 V
> >   Current        6.40 mA       12.45 mA     3.25 mA    12.45 mA
>  3.25
> > mA
> >   Tx Power       0.87 dBm       5.39 dBm  -12.44 dBm    2.39 dBm
>  -8.41
> > dBm
> >   Rx Power       0.01 dBm       5.39 dBm  -14.31 dBm    2.39 dBm
> -10.31
> > dBm
> >   Transmit Fault Count = 0
> >
> >
> ----------------------------------------------------------------------------
> >   Note: ++  high-alarm; +  high-warning; --  low-alarm; -  low-warning
> >
> > Lane Number:4 Network Lane
> >            SFP Detail Diagnostics Information (internal calibration)
> >
> >
> ----------------------------------------------------------------------------
> >                 Current              Alarms                  Warnings
> >                 Measurement     High        Low         High          Low
> >
> >
> ----------------------------------------------------------------------------
> >   Temperature   30.51 C        75.00 C     -5.00 C     70.00 C
> 0.00 C
> >   Voltage        3.28 V         3.63 V      2.97 V      3.46 V
> 3.13 V
> >   Current        6.40 mA       12.45 mA     3.25 mA    12.45 mA
>  3.25
> > mA
> >   Tx Power       0.67 dBm       5.39 dBm  -12.44 dBm    2.39 dBm
>  -8.41
> > dBm
> >   Rx Power       0.11 dBm       5.39 dBm  -14.31 dBm    2.39 dBm
> -10.31
> > dBm
> >   Transmit Fault Count = 0
> >
> >
> ----------------------------------------------------------------------------
> >   Note: ++  high-alarm; +  high-warning; --  low-alarm; -  low-warning
> >
> >
> >
> > MX1> show interfaces diagnostics optics et-1/0/5
> > Physical interface: et-1/0/5
> >     Module temperature                        :  38 degrees C / 100
> degrees
> > F
> >     Module voltage                            :  3.2740 V
> >     Module temperature high alarm             :  Off
> >     Module temperature low alarm              :  Off
> >     Module temperature high warning           :  Off
> >     Module temperature low warning            :  Off
> >     Module voltage high alarm                 :  Off
> >     Module voltage low alarm                  :  Off
> >     Module voltage high warning               :  Off
> >     Module voltage low warning                :  Off
> >     Module temperature high alarm threshold   :  78 degrees C / 172
> degrees
> > F
> >     Module temperature low alarm threshold    :  -5 degrees C / 23
> degrees F
> >     Module temperature high warning threshold :  75 degrees C / 167
> degrees
> > F
> >     Module temperature low warning threshold  :  0 degrees C / 32
> degrees F
> >     Module voltage high alarm threshold       :  3.6300 V
> >     Module voltage low alarm threshold        :  2.9700 V
> >     Module voltage high warning threshold     :  3.4640 V
> >     Module voltage low warning threshold      :  3.1340 V
> >     Laser bias current high alarm threshold   :  104.999 mA
> >     Laser bias current low alarm threshold    :  7.999 mA
> >     Laser bias current high warning threshold :  104.999 mA
> >     Laser bias current low warning threshold  :  9.999 mA
> >     Laser output power high alarm threshold   :  5.3703 mW / 7.30 dBm
> >     Laser output power low alarm threshold    :  0.0794 mW / -11.00 dBm
> >     Laser output power high warning threshold :  3.1623 mW / 5.00 dBm
> >     Laser output power low warning threshold  :  0.1995 mW / -7.00 dBm
> >     Laser rx power high alarm threshold       :  4.4668 mW / 6.50 dBm
> >     Laser rx power low alarm threshold        :  0.0251 mW / -16.00 dBm
> >     Laser rx power high warning threshold     :  3.5481 mW / 5.50 dBm
> >     Laser rx power low warning threshold      :  0.0630 mW / -12.01 dBm
> >   Lane 0
> >     Laser bias current                        :  41.588 mA
> >     Laser output power                        :  1.702 mW / 2.31 dBm
> >     Laser receiver power                      :  1.102 mW / 0.42 dBm
> >     Laser bias current high alarm             :  Off
> >     Laser bias current low alarm              :  Off
> >     Laser bias current high warning           :  Off
> >     Laser bias current low warning            :  Off
> >     Laser receiver power high alarm           :  Off
> >     Laser receiver power low alarm            :  Off
> >     Laser receiver power high warning         :  Off
> >     Laser receiver power low warning          :  Off
> >     Tx loss of signal functionality alarm     :  Off
> >     Rx loss of signal alarm                   :  Off
> >     Tx laser disabled alarm                   :  Off
> >   Lane 1
> >     Laser bias current                        :  42.324 mA
> >     Laser output power                        :  1.376 mW / 1.39 dBm
> >     Laser receiver power                      :  2.001 mW / 3.01 dBm
> >     Laser bias current high alarm             :  Off
> >     Laser bias current low alarm              :  Off
> >     Laser bias current high warning           :  Off
> >     Laser bias current low warning            :  Off
> >     Laser receiver power high alarm           :  Off
> >     Laser receiver power low alarm            :  Off
> >     Laser receiver power high warning         :  Off
> >     Laser receiver power low warning          :  Off
> >     Tx loss of signal functionality alarm     :  Off
> >     Rx loss of signal alarm                   :  Off
> >     Tx laser disabled alarm                   :  Off
> >   Lane 2
> >     Laser bias current                        :  41.066 mA
> >     Laser output power                        :  1.659 mW / 2.20 dBm
> >     Laser receiver power                      :  1.328 mW / 1.23 dBm
> >     Laser bias current high alarm             :  Off
> >     Laser bias current low alarm              :  Off
> >     Laser bias current high warning           :  Off
> >     Laser bias current low warning            :  Off
> >     Laser receiver power high alarm           :  Off
> >     Laser receiver power low alarm            :  Off
> >     Laser receiver power high warning         :  Off
> >     Laser receiver power low warning          :  Off
> >     Tx loss of signal functionality alarm     :  Off
> >     Rx loss of signal alarm                   :  Off
> >     Tx laser disabled alarm                   :  Off
> >   Lane 3
> >     Laser bias current                        :  37.970 mA
> >     Laser output power                        :  1.304 mW / 1.15 dBm
> >     Laser receiver power                      :  1.370 mW / 1.37 dBm
> >     Laser bias current high alarm             :  Off
> >     Laser bias current low alarm              :  Off
> >     Laser bias current high warning           :  Off
> >     Laser bias current low warning            :  Off
> >     Laser receiver power high alarm           :  Off
> >     Laser receiver power low alarm            :  Off
> >     Laser receiver power high warning         :  Off
> >     Laser receiver power low warning          :  Off
> >     Tx loss of signal functionality alarm     :  Off
> >     Rx loss of signal alarm                   :  Off
> >     Tx laser disabled alarm                   :  Off
> >
> > Il giorno dom 11 feb 2024 alle ore 13:12 Havard Eidnes <he at uninett.no>
> ha
> > scritto:
> >
> > > > DC technicians states cable are the same in both DCs and
> > > > direct, no patch panel
> > >
> > > Things I would look at:
> > >
> > >  * Has all the connectors been verified clean via microscope?
> > >
> > >  * Optical levels relative to threshold values (may relate to the
> > >    first).
> > >
> > >  * Any end seeing any input errors?  (May relate to the above
> > >    two.)  On the Juniper you can see some of this via PCS
> > >    ("Physical Coding Sublayer") unexpected events independently
> > >    of whether you have payload traffic, not sure you can do the
> > >    same on the Nexus boxes.
> > >
> > > Regards,
> > >
> > > - Håvard
> > >
> > _______________________________________________
> > juniper-nsp mailing list juniper-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/juniper-nsp
>
>
>
> --
>   ++ytti
>


More information about the juniper-nsp mailing list