[j-nsp] EX4300: Framing error with macsec enabled

Sun Apr 19 03:37:23 EDT 2020

Dear experts,
I've an EX4300 (Junos 17.3R3-S3.3) which have a constant Framing error
counter increase also if the traffic is very low.
Interface is connected to a WAN link from a carrier and bw is 1 Gbs but
traffic max is actually 100 Mbs and on average 10 Mbs.
On this interface I've enabled macsec, if I disable macsec the issue is not
in place but unfortunately macsec is mandatory to be kept enabled.

I cannot sniff since the packet is encrypted but to me it seems that
traffic is not lost, if I have 100 Mbs inside from WAN I see 100 Mbs
outside to DataCenter.

Due to the fact that monitoring system contantly raise an alert, I'd like
to know how to fix it or at least let the EX4300 do not raise the counter
increase.

I've opened a JTAC case but they found a PR which is currently related to a
Broadcom chipset raising framing errors during spikes (ie 70% of the
interface bandwidth).

https://kb.juniper.net/InfoCenter/index?page=content&id=KB32264&actp=METADATA

Also enabling flow-control as described in the KB do not change the
behaviour.

I'm wondering if there could the option we're receiving some sort on
"unknown protocol" from the carrier (I remeber Cisco has something like
that) or could be an harware issue..

On the other side of the link, the other EX4300 (side B) do not experience
the same issue but the traffic is mostly from side B to side A.

Here an example of the output, statistics cleared and after 1 minute I get
12 framing errors with 2 Mbs running on the WAN link:

@EX4300-A> show interfaces ae0 extensive
Physical interface: ae0, Enabled, Physical link is Up
  Interface index: 220, SNMP ifIndex: 549, Generation: 131
  Description: xxx
  Link-level type: Ethernet, MTU: 9192, Speed: 1Gbps, BPDU Error: None,
Ethernet-Switching Error: None, MAC-REWRITE Error: None,
  Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled,
Minimum links needed: 1, Minimum bandwidth needed: 1bps
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x0
  Current address: cc:e5:94:11:43:23, Hardware address: cc:e5:94:11:43:23
  Last flapped   : 2020-04-19 02:05:05 CEST (06:50:45 ago)
  Statistics last cleared: 2020-04-19 09:11:22 CEST (00:01:00 ago)
  Traffic statistics:
   Input  bytes  :             10014863              2205456 bps
   Output bytes  :              4095720               582456 bps
   Input  packets:                33292                  624 pps
   Output packets:                33023                  568 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Input errors:
    Errors: 12, Drops: 0, Framing errors: 12, Runts: 0, Giants: 0, Policed
discards: 0, Resource errors: 0
  Output errors:
    Carrier transitions: 0, Errors: 0, Drops: 0, MTU errors: 0, Resource
errors: 0
  Egress queues: 12 supported, 11 in use

@EX4300-A> show interfaces ge-0/0/0 extensive
Physical interface: ge-0/0/0, Enabled, Physical link is Up
  Interface index: 649, SNMP ifIndex: 509, Generation: 140
  Description: WAN link
  Link-level type: Ethernet, MTU: 9192, LAN-PHY mode, Link-mode:
Full-duplex, Speed: 1000mbps, BPDU Error: None, Loop Detect PDU Error: None,
  Ethernet-Switching Error: None, Source filtering: Disabled
  Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback:
Disabled, Flow control: Enabled, Auto-negotiation: Enabled,
  Remote fault: Online, Media type: Copper, IEEE 802.3az Energy Efficient
Ethernet: Disabled, Auto-MDIX: Enabled
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x0
  Link flags     : None
  CoS queues     : 12 supported, 12 maximum usable queues
  Hold-times     : Up 0 ms, Down 0 ms
  Current address: cc:e5:94:11:43:23, Hardware address: cc:e5:94:11:43:23
  Last flapped   : 2020-03-28 18:43:04 CET (3w0d 13:30 ago)
  Statistics last cleared: 2020-04-19 09:11:18 CEST (00:02:18 ago)
  Traffic statistics:
   Input  bytes  :             21782579               932296 bps
   Output bytes  :             17898068               498704 bps
   Input  packets:                76844                  569 pps
   Output packets:                82594                  590 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Input errors:
    Errors: 28, Drops: 0, Framing errors: 28, Runts: 0, Policed discards:
0, L3 incompletes: 0, L2 channel errors: 0,
    L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0

Here part of the config:

@EX4300-A> show configuration interfaces ge-0/0/0 | display set
set interfaces ge-0/0/0 ether-options auto-negotiation
set interfaces ge-0/0/0 ether-options flow-control
set interfaces ge-0/0/0 ether-options 802.3ad ae0

@EX4300-A> show configuration interfaces ae0 | display set
set interfaces ae0 mtu 9192
set interfaces ae0 aggregated-ether-options flow-control
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options lacp periodic fast
set interfaces ae0 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae0 unit 0 family ethernet-switching vlan members 2228
set interfaces ae0 unit 0 family ethernet-switching vlan members 2552-2553
set interfaces ae0 unit 0 family ethernet-switching filter input QOS

@EX4300-A> show configuration security macsec | display set
set security macsec connectivity-association MAC security-mode static-cak
set security macsec connectivity-association MAC pre-shared-key ckn xxxx
set security macsec connectivity-association MAC pre-shared-key cak
"tttttvvvv"
set security macsec connectivity-association MAC exclude-protocol lldp
set security macsec connectivity-association MAC exclude-protocol lacp
set security macsec interfaces ge-0/0/0 connectivity-association MAC

Dear all, an help is appreciated and welcomme, please let me thank in
advance anyone will give an hint.

Cheers
James