[j-nsp] [c-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco

james list jameslist72 at gmail.com
Sun Feb 11 07:49:55 EST 2024


Hi
there are no errors on both interfaces (Cisco and Juniper).

here following logs of one event on both side, config and LACP stats.

LOGS of one event time 16:39:

CISCO
2024 Feb  9 16:39:36 NEXUS1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN:
Interface port-channel101 is down (No operational members)
2024 Feb  9 16:39:36 NEXUS1 %ETHPORT-5-IF_DOWN_PARENT_DOWN: Interface
port-channel101.2303 is down (Parent interface is down)
2024 Feb  9 16:39:36 NEXUS1 %BGP-5-ADJCHANGE:  bgp- [xxx] (xxx) neighbor
172.16.6.17 Down - sent:  other configuration change
2024 Feb  9 16:39:36 NEXUS1 %ETH_PORT_CHANNEL-5-FOP_CHANGED:
port-channel101: first operational port changed from Ethernet1/44 to none
2024 Feb  9 16:39:36 NEXUS1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel101:
Ethernet1/44 is down
2024 Feb  9 16:39:36 NEXUS1 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface
port-channel101,bandwidth changed to 100000 Kbit
2024 Feb  9 16:39:36 NEXUS1 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface
Ethernet1/44 is down (Initializing)
2024 Feb  9 16:39:36 NEXUS1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN:
Interface port-channel101 is down (No operational members)
2024 Feb  9 16:39:36 NEXUS1 %ETHPORT-5-SPEED: Interface port-channel101,
operational speed changed to 100 Gbps
2024 Feb  9 16:39:36 NEXUS1 %ETHPORT-5-IF_DUPLEX: Interface
port-channel101, operational duplex mode changed to Full
2024 Feb  9 16:39:36 NEXUS1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface
port-channel101, operational Receive Flow Control state changed to off
2024 Feb  9 16:39:36 NEXUS1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface
port-channel101, operational Transmit Flow Control state changed to off
2024 Feb  9 16:39:39 NEXUS1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel101:
Ethernet1/44 is up
2024 Feb  9 16:39:39 NEXUS1 %ETH_PORT_CHANNEL-5-FOP_CHANGED:
port-channel101: first operational port changed from none to Ethernet1/44
2024 Feb  9 16:39:39 NEXUS1 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface
port-channel101,bandwidth changed to 100000000 Kbit
2024 Feb  9 16:39:39 NEXUS1 %ETHPORT-5-IF_UP: Interface Ethernet1/44 is up
in Layer3
2024 Feb  9 16:39:39 NEXUS1 %ETHPORT-5-IF_UP: Interface port-channel101 is
up in Layer3
2024 Feb  9 16:39:39 NEXUS1 %ETHPORT-5-IF_UP: Interface
port-channel101.2303 is up in Layer3
2024 Feb  9 16:39:43 NEXUS1 %BGP-5-ADJCHANGE:  bgp- [xxx] (xxx) neighbor
172.16.6.17 Up



Feb  9 16:39:35.813 2024  MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp
current while timer expired current Receive State: CURRENT
Feb  9 16:39:35.813 2024  MX1 lacpd[31632]: LACP_INTF_DOWN: ae49: Interface
marked down due to lacp timeout on member et-0/1/5
Feb  9 16:39:35.819 2024  MX1 kernel: lag_bundlestate_ifd_change: bundle
ae49: bundle IFD minimum bandwidth or minimum links not met, Bandwidth
(Current : Required) 0 : 100000000000 Number of links (Current : Required)
0 : 1
Feb  9 16:39:35.815 2024  MX1 lacpd[31632]: LACP_INTF_MUX_STATE_CHANGED:
ae49: et-0/1/5: Lacp state changed from COLLECTING_DISTRIBUTING to
ATTACHED, actor port state : |EXP|-|-|-|IN_SYNC|AGG|SHORT|ACT|, partner
port state : |-|-|DIS|COL|OUT_OF_SYNC|AGG|SHORT|ACT|
Feb  9 16:39:35.869 2024  MX1 rpd[31866]: bgp_ifachange_group:10697:
NOTIFICATION sent to 172.16.6.18 (External AS xxx): code 6 (Cease) subcode
6 (Other Configuration Change), Reason: Interface change for the peer-group
Feb  9 16:39:35.909 2024  MX1 mib2d[31909]: SNMP_TRAP_LINK_DOWN: ifIndex
684, ifAdminStatus up(1), ifOperStatus down(2), ifName ae49
Feb  9 16:39:36.083 2024  MX1 lacpd[31632]: LACP_INTF_MUX_STATE_CHANGED:
ae49: et-0/1/5: Lacp state changed from ATTACHED to
COLLECTING_DISTRIBUTING, actor port state :
|-|-|DIS|COL|IN_SYNC|AGG|SHORT|ACT|, partner port state :
|-|-|DIS|COL|IN_SYNC|AGG|SHORT|ACT|
Feb  9 16:39:36.089 2024  MX1 kernel: lag_bundlestate_ifd_change: bundle
ae49 is now Up. uplinks 1 >= min_links 1
Feb  9 16:39:36.089 2024  MX1 kernel: lag_bundlestate_ifd_change: bundle
ae49: bundle IFD minimum bandwidth or minimum links not met, Bandwidth
(Current : Required) 0 : 100000000000 Number of links (Current : Required)
0 : 1
Feb  9 16:39:36.085 2024  MX1 lacpd[31632]: LACP_INTF_MUX_STATE_CHANGED:
ae49: et-0/1/5: Lacp state changed from COLLECTING_DISTRIBUTING to
ATTACHED, actor port state : |-|-|-|-|IN_SYNC|AGG|SHORT|ACT|, partner port
state : |-|-|-|-|OUT_OF_SYNC|AGG|SHORT|ACT|
Feb  9 16:39:39.095 2024  MX1 lacpd[31632]: LACP_INTF_MUX_STATE_CHANGED:
ae49: et-0/1/5: Lacp state changed from ATTACHED to
COLLECTING_DISTRIBUTING, actor port state :
|-|-|DIS|COL|IN_SYNC|AGG|SHORT|ACT|, partner port state :
|-|-|-|-|IN_SYNC|AGG|SHORT|ACT|
Feb  9 16:39:39.101 2024  MX1 kernel: lag_bundlestate_ifd_change: bundle
ae49 is now Up. uplinks 1 >= min_links 1
Feb  9 16:39:39.109 2024  MX1 mib2d[31909]: SNMP_TRAP_LINK_UP: ifIndex 684,
ifAdminStatus up(1), ifOperStatus up(1), ifName ae49
Feb  9 16:39:41.190 2024  MX1 rpd[31866]: bgp_recv: read from peer
172.16.6.18 (External AS xxx) failed: Unknown error: 48110976


CONFIG:

CISCO

NEXUS1# sh run int port-channel 101

interface port-channel101
  description <[To MX1|Et-0/1/5]>
  mtu 9216
  no ip redirects

NEXUS01# sh run int port-channel 101.2303

interface port-channel101.2303
  description <[To MX1|Et-0/1/5]>
  mtu 9216
  encapsulation dot1q 2303
  vrf member SIA
  bfd ipv4 interval 250 min_rx 250 multiplier 3
  no ip redirects
  ip address 172.16.6.18/30
  no shutdown

JUNIPER

MX1> show configuration interfaces ae49
description "link to NEXUS01";
flexible-vlan-tagging;
mtu 9192;
encapsulation flexible-ethernet-services;
aggregated-ether-options {
    lacp {
        active;
        periodic fast;
    }
}
unit 2303 {
    vlan-id 2303;
    family inet {
        mtu 1500;
        address 172.16.6.17/30;
    }
}

LACP counters:


CISCO

NEXUS01# sh lacp counters
NOTE: Clear lacp counters to get accurate statistics

------------------------------------------------------------------------------
                             LACPDUs                      Markers/Resp
LACPDUs
Port              Sent                Recv                  Recv Sent  Pkts
Err
------------------------------------------------------------------------------
port-channel101
Ethernet1/44       6123011              6118981                0      0
 0

NEXUS1# sh lacp interface eth1/44
Interface Ethernet1/44 is up
  Channel group is 101 port channel is Po101
  PDUs sent: 6123014
  PDUs rcvd: 6118984
  Markers sent: 0
  Markers rcvd: 0
  Marker response sent: 0
  Marker response rcvd: 0
  Unknown packets rcvd: 0
  Illegal packets rcvd: 0
Lag Id: [ [(7f, c4-9-b7-64-30-38, 32, 7f, 18), (8000, b0-8b-cf-83-49-5b,
64, 8000, 1ad)] ]
Operational as aggregated link since Fri Feb  9 16:39:39 2024

Local Port: Eth1/44   MAC Address= b0-8b-cf-83-49-5b
  System Identifier=0x8000,  Port Identifier=0x8000,0x1ad
  Operational key=100
  LACP_Activity=active
  LACP_Timeout=Short Timeout (1s)
  Synchronization=IN_SYNC
  Collecting=true
  Distributing=true
  Partner information refresh timeout=Short Timeout (3s)
Actor Admin State=63
Actor Oper State=63
Neighbor: 0x18
  MAC Address= c4-9-b7-64-30-38
  System Identifier=0x7f,  Port Identifier=0x7f,0x18
  Operational key=50
  LACP_Activity=active
  LACP_Timeout=short Timeout (1s)
  Synchronization=IN_SYNC
  Collecting=true
  Distributing=true
Partner Admin State=63
Partner Oper State=63
Aggregate or Individual(True=1)= 1

JUNIPER

MX1> show lacp interfaces ae49 extensive
Aggregated interface: ae49
    LACP state:       Role   Exp   Def  Dist  Col  Syn  Aggr  Timeout
 Activity
      et-0/1/5       Actor    No    No   Yes  Yes  Yes   Yes     Fast
 Active
      et-0/1/5     Partner    No    No   Yes  Yes  Yes   Yes     Fast
 Active
    LACP protocol:        Receive State  Transmit State          Mux State
      et-0/1/5                  Current   Fast periodic Collecting
distributing
    LACP info:        Role     System             System       Port
Port    Port
                             priority         identifier   priority
number     key
      et-0/1/5       Actor        127  c4:09:b7:64:30:38        127
24      50
      et-0/1/5     Partner      32768  b0:8b:cf:83:49:5b      32768
 429     100

Il giorno dom 11 feb 2024 alle ore 13:07 Gert Doering <gert at greenie.muc.de>
ha scritto:

> HI,
>
> On Sun, Feb 11, 2024 at 12:50:32PM +0100, james list wrote:
> > 2024 Feb  9 16:39:36 NEXUS1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN:
> > Interface port-channel101 is down (No operational members)
>
> So there is no *BGP* problem here, but a lower layer issue.
>
> Let me repeat that part about "error counters on the interface"...
>
> gert
> --
> "If was one thing all people took for granted, was conviction that if you
>  feed honest figures into a computer, honest figures come out. Never
> doubted
>  it myself till I met a computer with a sense of humor."
>                              Robert A. Heinlein, The Moon is a Harsh
> Mistress
>
> Gert Doering - Munich, Germany
> gert at greenie.muc.de
>


More information about the juniper-nsp mailing list