[j-nsp] [c-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
james list
jameslist72 at gmail.com
Sun Feb 11 08:24:34 EST 2024
On Cisco I see physical goes down (initializing), what does that mean?
While on Juniper when the issue happens I always see:
show log messages | last 440 | match LACPD_TIMEOUT
Jan 25 21:32:27.948 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Jan 26 18:41:12.514 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Jan 28 05:07:20.283 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Jan 29 04:06:51.768 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Jan 30 03:09:43.923 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Feb 5 18:13:20.158 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Feb 6 02:17:23.703 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Feb 6 22:00:23.758 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Feb 9 09:29:35.728 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Feb 9 16:39:35.813 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Il giorno dom 11 feb 2024 alle ore 14:10 Saku Ytti <saku at ytti.fi> ha
scritto:
> Hey James,
>
> You shared this off-list, I think it's sufficiently material to share.
>
> 2024 Feb 9 16:39:36 NEXUS1
> %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface
> port-channel101 is down (No operational members)
> 2024 Feb 9 16:39:36 NEXUS1 %ETH_PORT_CHANNEL-5-PORT_DOWN:
> port-channel101: Ethernet1/44 is down
> Feb 9 16:39:35.813 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5:
> lacp current while timer expired current Receive State: CURRENT
> Feb 9 16:39:35.813 2024 MX1 lacpd[31632]: LACP_INTF_DOWN: ae49:
> Interface marked down due to lacp timeout on member et-0/1/5
>
> We can't know the order of events here, due to no subsecond precision
> enabled on Cisco end.
>
> But if failure would start from interface down, it would take 3seconds
> for Juniper to realise LACP failure. However we can see that it
> happens in less than 1s, so we can determine the interface was not
> down first, the first problem was Juniper not receiving 3 consecutive
> LACP PDUs, 1s apart, prior to noticing any type of interface state
> related problems.
>
> Is this always the order of events? Does it always happen with Juniper
> noticing problems receiving LACP PDU first?
>
>
> On Sun, 11 Feb 2024 at 14:55, james list via juniper-nsp
> <juniper-nsp at puck.nether.net> wrote:
> >
> > Hi
> >
> > 1) cable has been replaced with a brand new one, they said that to check
> an
> > MPO 100 Gbs cable is not that easy
> >
> > 3) no errors reported on both side
> >
> > 2) here the output of cisco and juniper
> >
> > NEXUS1# sh interface eth1/44 transceiver details
> > Ethernet1/44
> > transceiver is present
> > type is QSFP-100G-SR4
> > name is CISCO-INNOLIGHT
> > part number is TR-FC85S-NC3
> > revision is 2C
> > serial number is INL27050TVT
> > nominal bitrate is 25500 MBit/sec
> > Link length supported for 50/125um OM3 fiber is 70 m
> > cisco id is 17
> > cisco extended id number is 220
> > cisco part number is 10-3142-03
> > cisco product id is QSFP-100G-SR4-S
> > cisco version id is V03
> >
> > Lane Number:1 Network Lane
> > SFP Detail Diagnostics Information (internal calibration)
> >
> >
> ----------------------------------------------------------------------------
> > Current Alarms Warnings
> > Measurement High Low High Low
> >
> >
> ----------------------------------------------------------------------------
> > Temperature 30.51 C 75.00 C -5.00 C 70.00 C
> 0.00 C
> > Voltage 3.28 V 3.63 V 2.97 V 3.46 V
> 3.13 V
> > Current 6.40 mA 12.45 mA 3.25 mA 12.45 mA
> 3.25
> > mA
> > Tx Power 0.98 dBm 5.39 dBm -12.44 dBm 2.39 dBm
> -8.41
> > dBm
> > Rx Power -1.60 dBm 5.39 dBm -14.31 dBm 2.39 dBm
> -10.31
> > dBm
> > Transmit Fault Count = 0
> >
> >
> ----------------------------------------------------------------------------
> > Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
> >
> > Lane Number:2 Network Lane
> > SFP Detail Diagnostics Information (internal calibration)
> >
> >
> ----------------------------------------------------------------------------
> > Current Alarms Warnings
> > Measurement High Low High Low
> >
> >
> ----------------------------------------------------------------------------
> > Temperature 30.51 C 75.00 C -5.00 C 70.00 C
> 0.00 C
> > Voltage 3.28 V 3.63 V 2.97 V 3.46 V
> 3.13 V
> > Current 6.40 mA 12.45 mA 3.25 mA 12.45 mA
> 3.25
> > mA
> > Tx Power 0.62 dBm 5.39 dBm -12.44 dBm 2.39 dBm
> -8.41
> > dBm
> > Rx Power -1.18 dBm 5.39 dBm -14.31 dBm 2.39 dBm
> -10.31
> > dBm
> > Transmit Fault Count = 0
> >
> >
> ----------------------------------------------------------------------------
> > Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
> >
> > Lane Number:3 Network Lane
> > SFP Detail Diagnostics Information (internal calibration)
> >
> >
> ----------------------------------------------------------------------------
> > Current Alarms Warnings
> > Measurement High Low High Low
> >
> >
> ----------------------------------------------------------------------------
> > Temperature 30.51 C 75.00 C -5.00 C 70.00 C
> 0.00 C
> > Voltage 3.28 V 3.63 V 2.97 V 3.46 V
> 3.13 V
> > Current 6.40 mA 12.45 mA 3.25 mA 12.45 mA
> 3.25
> > mA
> > Tx Power 0.87 dBm 5.39 dBm -12.44 dBm 2.39 dBm
> -8.41
> > dBm
> > Rx Power 0.01 dBm 5.39 dBm -14.31 dBm 2.39 dBm
> -10.31
> > dBm
> > Transmit Fault Count = 0
> >
> >
> ----------------------------------------------------------------------------
> > Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
> >
> > Lane Number:4 Network Lane
> > SFP Detail Diagnostics Information (internal calibration)
> >
> >
> ----------------------------------------------------------------------------
> > Current Alarms Warnings
> > Measurement High Low High Low
> >
> >
> ----------------------------------------------------------------------------
> > Temperature 30.51 C 75.00 C -5.00 C 70.00 C
> 0.00 C
> > Voltage 3.28 V 3.63 V 2.97 V 3.46 V
> 3.13 V
> > Current 6.40 mA 12.45 mA 3.25 mA 12.45 mA
> 3.25
> > mA
> > Tx Power 0.67 dBm 5.39 dBm -12.44 dBm 2.39 dBm
> -8.41
> > dBm
> > Rx Power 0.11 dBm 5.39 dBm -14.31 dBm 2.39 dBm
> -10.31
> > dBm
> > Transmit Fault Count = 0
> >
> >
> ----------------------------------------------------------------------------
> > Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
> >
> >
> >
> > MX1> show interfaces diagnostics optics et-1/0/5
> > Physical interface: et-1/0/5
> > Module temperature : 38 degrees C / 100
> degrees
> > F
> > Module voltage : 3.2740 V
> > Module temperature high alarm : Off
> > Module temperature low alarm : Off
> > Module temperature high warning : Off
> > Module temperature low warning : Off
> > Module voltage high alarm : Off
> > Module voltage low alarm : Off
> > Module voltage high warning : Off
> > Module voltage low warning : Off
> > Module temperature high alarm threshold : 78 degrees C / 172
> degrees
> > F
> > Module temperature low alarm threshold : -5 degrees C / 23
> degrees F
> > Module temperature high warning threshold : 75 degrees C / 167
> degrees
> > F
> > Module temperature low warning threshold : 0 degrees C / 32
> degrees F
> > Module voltage high alarm threshold : 3.6300 V
> > Module voltage low alarm threshold : 2.9700 V
> > Module voltage high warning threshold : 3.4640 V
> > Module voltage low warning threshold : 3.1340 V
> > Laser bias current high alarm threshold : 104.999 mA
> > Laser bias current low alarm threshold : 7.999 mA
> > Laser bias current high warning threshold : 104.999 mA
> > Laser bias current low warning threshold : 9.999 mA
> > Laser output power high alarm threshold : 5.3703 mW / 7.30 dBm
> > Laser output power low alarm threshold : 0.0794 mW / -11.00 dBm
> > Laser output power high warning threshold : 3.1623 mW / 5.00 dBm
> > Laser output power low warning threshold : 0.1995 mW / -7.00 dBm
> > Laser rx power high alarm threshold : 4.4668 mW / 6.50 dBm
> > Laser rx power low alarm threshold : 0.0251 mW / -16.00 dBm
> > Laser rx power high warning threshold : 3.5481 mW / 5.50 dBm
> > Laser rx power low warning threshold : 0.0630 mW / -12.01 dBm
> > Lane 0
> > Laser bias current : 41.588 mA
> > Laser output power : 1.702 mW / 2.31 dBm
> > Laser receiver power : 1.102 mW / 0.42 dBm
> > Laser bias current high alarm : Off
> > Laser bias current low alarm : Off
> > Laser bias current high warning : Off
> > Laser bias current low warning : Off
> > Laser receiver power high alarm : Off
> > Laser receiver power low alarm : Off
> > Laser receiver power high warning : Off
> > Laser receiver power low warning : Off
> > Tx loss of signal functionality alarm : Off
> > Rx loss of signal alarm : Off
> > Tx laser disabled alarm : Off
> > Lane 1
> > Laser bias current : 42.324 mA
> > Laser output power : 1.376 mW / 1.39 dBm
> > Laser receiver power : 2.001 mW / 3.01 dBm
> > Laser bias current high alarm : Off
> > Laser bias current low alarm : Off
> > Laser bias current high warning : Off
> > Laser bias current low warning : Off
> > Laser receiver power high alarm : Off
> > Laser receiver power low alarm : Off
> > Laser receiver power high warning : Off
> > Laser receiver power low warning : Off
> > Tx loss of signal functionality alarm : Off
> > Rx loss of signal alarm : Off
> > Tx laser disabled alarm : Off
> > Lane 2
> > Laser bias current : 41.066 mA
> > Laser output power : 1.659 mW / 2.20 dBm
> > Laser receiver power : 1.328 mW / 1.23 dBm
> > Laser bias current high alarm : Off
> > Laser bias current low alarm : Off
> > Laser bias current high warning : Off
> > Laser bias current low warning : Off
> > Laser receiver power high alarm : Off
> > Laser receiver power low alarm : Off
> > Laser receiver power high warning : Off
> > Laser receiver power low warning : Off
> > Tx loss of signal functionality alarm : Off
> > Rx loss of signal alarm : Off
> > Tx laser disabled alarm : Off
> > Lane 3
> > Laser bias current : 37.970 mA
> > Laser output power : 1.304 mW / 1.15 dBm
> > Laser receiver power : 1.370 mW / 1.37 dBm
> > Laser bias current high alarm : Off
> > Laser bias current low alarm : Off
> > Laser bias current high warning : Off
> > Laser bias current low warning : Off
> > Laser receiver power high alarm : Off
> > Laser receiver power low alarm : Off
> > Laser receiver power high warning : Off
> > Laser receiver power low warning : Off
> > Tx loss of signal functionality alarm : Off
> > Rx loss of signal alarm : Off
> > Tx laser disabled alarm : Off
> >
> > Il giorno dom 11 feb 2024 alle ore 13:12 Havard Eidnes <he at uninett.no>
> ha
> > scritto:
> >
> > > > DC technicians states cable are the same in both DCs and
> > > > direct, no patch panel
> > >
> > > Things I would look at:
> > >
> > > * Has all the connectors been verified clean via microscope?
> > >
> > > * Optical levels relative to threshold values (may relate to the
> > > first).
> > >
> > > * Any end seeing any input errors? (May relate to the above
> > > two.) On the Juniper you can see some of this via PCS
> > > ("Physical Coding Sublayer") unexpected events independently
> > > of whether you have payload traffic, not sure you can do the
> > > same on the Nexus boxes.
> > >
> > > Regards,
> > >
> > > - Håvard
> > >
> > _______________________________________________
> > juniper-nsp mailing list juniper-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/juniper-nsp
>
>
>
> --
> ++ytti
>
More information about the juniper-nsp
mailing list