[j-nsp] [EXT] EX4300: Framing error with macsec enabled
Richard McGovern
rmcgovern at juniper.net
Tue Apr 21 09:27:56 EDT 2020
Chuck, I thought you were running both LLDP and LACP outside the MACSEC tunnel, no?
(Optional) Exclude a protocol from MACsec:
[edit security macsec connectivity-association connectivity-association-name]
user at switch# set exclude-protocol protocol-name
For instance, if you did not want Link Level Discovery Protocol (LLDP) to be secured using MACsec:
[edit security macsec connectivity-association ca-dynamic1]
user at switch# set exclude-protocol lldp
When this option is enabled, MACsec is disabled for all packets of the specified protocol—in this case, LLDP—that are sent or received on the link.
BEST PRACTICEWe recommend that any protocol other than MACsec being used on the MACsec connection, such as LLDP, LACP, STP, or layer 3 routing protocols, should be excluded and moved outside of the MACsec tunnel.
Is this not working properly for LLDP?
Rich
Richard McGovern
Sr Sales Engineer, Juniper Networks
978-618-3342
I’d rather be lucky than good, as I know I am not good
I don’t make the news, I just report it
On 4/19/20, 7:31 PM, "Chuck Anderson" <cra at WPI.EDU> wrote:
Well, that was an easy fix on my MX480s:
set protocols lldp interface xe-0/0/1 disable
Now I'm not seeing CRC errors incrementing 2-3 times per minute on the EX3400s connected directly to the MX480s.
I'm not excluding any protocols from MACsec--LLDP runs end-to-end between the EX3400s just fine.
Check if the carrier is running LLDP or CDP or similar.
On Sun, Apr 19, 2020 at 07:16:46PM -0400, Chuck Anderson wrote:
> Yes, I see CRC errors on EX3400s with MACsec termination, but only on one side.
>
> Here is my topology:
>
> From A to B:
>
> [EX3400-A]-->--[push-vlan-tag-on-MX480]-->-L2 vlan-->-[Carrier-ASR9k-pop-vlan-tag]-->--[EX3400-B]
> MACsec L2 connection L2 xconnect MACsec
>
> From B to A:
>
> [EX3400-A]--<--[pop-vlan-tag-on-MX480]--<-L2 vlan--<-[Carrier-ASR9k-push-vlan-tag]--<--[EX3400-B]
> MACsec L2 connection L2 xconnect MACsec
>
> I also have a redundant path with EX3400-C (different local switch) and EX3400-B (same remote switch).
>
> I see the CRC errors increasing at a rate of about 2-3 per minute, but only on EX3400-A and EXX3400-C.
>
> All EX3400s were initially running 15.1X53-D57. Now A and C are running 18.2R3-S2 and B is running 15.1X53-D592. But the problem has been consistent throughout all releases, no improvement with upgrades.
>
> I wonder if something the carrier's ASR9k is sending down the VLAN towards EX3400-A and -C is causing this? If not, maybe it is the MX480s sending something locally to EX3400-A and -C?
>
> The following PRs don't seem relevant--I'm not doing anywhere close to 60% utilization:
>
> https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1261567
>
> And I'm not seeing "runts":
>
> https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1469663
>
> I'm only seeing Framing errors (CRC/Align errors):
>
> admin at ex3400-a> show interfaces extensive xe-0/2/0 |match 22791
> Errors: 227911, Drops: 0, Framing errors: 227911, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
> CRC/Align errors 227911 0
>
> A few seconds later, it increased to 227913:
>
> MAC statistics: Receive Transmit
> Total octets 17953647117156 3221741316352
> Total packets 13200126465 7010832956
> Unicast packets 13194022205 7004785539
> Broadcast packets 5272 0
> Multicast packets 6098988 6047417
> CRC/Align errors 227913 0
> FIFO errors 0 0
> MAC control frames 0 0
> MAC pause frames 0 0
> Oversized frames 0
> Jabber frames 0
> Fragment frames 0
> VLAN tagged frames 13196813130
> Code violations 0
>
> Rate is only 24 Mbps, 2200 pps:
>
> admin at ex3400-a> show interfaces extensive xe-0/2/0 |match "bps|pps"
> Link-level type: Ethernet, MTU: 9192, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None, MAC-REWRITE Error: None,
> Input bytes : 17954037433802 24242136 bps
> Output bytes : 3221749625049 498504 bps
> Input packets: 13200416854 2190 pps
> Output packets: 7010941872 830 pps
>
>
>
> On Sun, Apr 19, 2020 at 09:37:23AM +0200, james list wrote:
> > Dear experts,
> > I've an EX4300 (Junos 17.3R3-S3.3) which have a constant Framing error
> > counter increase also if the traffic is very low.
> > Interface is connected to a WAN link from a carrier and bw is 1 Gbs but
> > traffic max is actually 100 Mbs and on average 10 Mbs.
> > On this interface I've enabled macsec, if I disable macsec the issue is not
> > in place but unfortunately macsec is mandatory to be kept enabled.
> >
> > I cannot sniff since the packet is encrypted but to me it seems that
> > traffic is not lost, if I have 100 Mbs inside from WAN I see 100 Mbs
> > outside to DataCenter.
> >
> > Due to the fact that monitoring system contantly raise an alert, I'd like
> > to know how to fix it or at least let the EX4300 do not raise the counter
> > increase.
> >
> > I've opened a JTAC case but they found a PR which is currently related to a
> > Broadcom chipset raising framing errors during spikes (ie 70% of the
> > interface bandwidth).
> >
> > https://kb.juniper.net/InfoCenter/index?page=content&id=KB32264&actp=METADATA
> >
> > Also enabling flow-control as described in the KB do not change the
> > behaviour.
> >
> > I'm wondering if there could the option we're receiving some sort on
> > "unknown protocol" from the carrier (I remeber Cisco has something like
> > that) or could be an harware issue..
> >
> > On the other side of the link, the other EX4300 (side B) do not experience
> > the same issue but the traffic is mostly from side B to side A.
> >
> > Here an example of the output, statistics cleared and after 1 minute I get
> > 12 framing errors with 2 Mbs running on the WAN link:
> >
> > @EX4300-A> show interfaces ae0 extensive
> > Physical interface: ae0, Enabled, Physical link is Up
> > Interface index: 220, SNMP ifIndex: 549, Generation: 131
> > Description: xxx
> > Link-level type: Ethernet, MTU: 9192, Speed: 1Gbps, BPDU Error: None,
> > Ethernet-Switching Error: None, MAC-REWRITE Error: None,
> > Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled,
> > Minimum links needed: 1, Minimum bandwidth needed: 1bps
> > Device flags : Present Running
> > Interface flags: SNMP-Traps Internal: 0x0
> > Current address: cc:e5:94:11:43:23, Hardware address: cc:e5:94:11:43:23
> > Last flapped : 2020-04-19 02:05:05 CEST (06:50:45 ago)
> > Statistics last cleared: 2020-04-19 09:11:22 CEST (00:01:00 ago)
> > Traffic statistics:
> > Input bytes : 10014863 2205456 bps
> > Output bytes : 4095720 582456 bps
> > Input packets: 33292 624 pps
> > Output packets: 33023 568 pps
> > IPv6 transit statistics:
> > Input bytes : 0
> > Output bytes : 0
> > Input packets: 0
> > Output packets: 0
> > Input errors:
> > Errors: 12, Drops: 0, Framing errors: 12, Runts: 0, Giants: 0, Policed
> > discards: 0, Resource errors: 0
> > Output errors:
> > Carrier transitions: 0, Errors: 0, Drops: 0, MTU errors: 0, Resource
> > errors: 0
> > Egress queues: 12 supported, 11 in use
> >
> >
> > @EX4300-A> show interfaces ge-0/0/0 extensive
> > Physical interface: ge-0/0/0, Enabled, Physical link is Up
> > Interface index: 649, SNMP ifIndex: 509, Generation: 140
> > Description: WAN link
> > Link-level type: Ethernet, MTU: 9192, LAN-PHY mode, Link-mode:
> > Full-duplex, Speed: 1000mbps, BPDU Error: None, Loop Detect PDU Error: None,
> > Ethernet-Switching Error: None, Source filtering: Disabled
> > Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback:
> > Disabled, Flow control: Enabled, Auto-negotiation: Enabled,
> > Remote fault: Online, Media type: Copper, IEEE 802.3az Energy Efficient
> > Ethernet: Disabled, Auto-MDIX: Enabled
> > Device flags : Present Running
> > Interface flags: SNMP-Traps Internal: 0x0
> > Link flags : None
> > CoS queues : 12 supported, 12 maximum usable queues
> > Hold-times : Up 0 ms, Down 0 ms
> > Current address: cc:e5:94:11:43:23, Hardware address: cc:e5:94:11:43:23
> > Last flapped : 2020-03-28 18:43:04 CET (3w0d 13:30 ago)
> > Statistics last cleared: 2020-04-19 09:11:18 CEST (00:02:18 ago)
> > Traffic statistics:
> > Input bytes : 21782579 932296 bps
> > Output bytes : 17898068 498704 bps
> > Input packets: 76844 569 pps
> > Output packets: 82594 590 pps
> > IPv6 transit statistics:
> > Input bytes : 0
> > Output bytes : 0
> > Input packets: 0
> > Output packets: 0
> > Input errors:
> > Errors: 28, Drops: 0, Framing errors: 28, Runts: 0, Policed discards:
> > 0, L3 incompletes: 0, L2 channel errors: 0,
> > L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
> >
> >
> > Here part of the config:
> >
> > @EX4300-A> show configuration interfaces ge-0/0/0 | display set
> > set interfaces ge-0/0/0 ether-options auto-negotiation
> > set interfaces ge-0/0/0 ether-options flow-control
> > set interfaces ge-0/0/0 ether-options 802.3ad ae0
> >
> >
> > @EX4300-A> show configuration interfaces ae0 | display set
> > set interfaces ae0 mtu 9192
> > set interfaces ae0 aggregated-ether-options flow-control
> > set interfaces ae0 aggregated-ether-options lacp active
> > set interfaces ae0 aggregated-ether-options lacp periodic fast
> > set interfaces ae0 unit 0 family ethernet-switching interface-mode trunk
> > set interfaces ae0 unit 0 family ethernet-switching vlan members 2228
> > set interfaces ae0 unit 0 family ethernet-switching vlan members 2552-2553
> > set interfaces ae0 unit 0 family ethernet-switching filter input QOS
> >
> >
> > @EX4300-A> show configuration security macsec | display set
> > set security macsec connectivity-association MAC security-mode static-cak
> > set security macsec connectivity-association MAC pre-shared-key ckn xxxx
> > set security macsec connectivity-association MAC pre-shared-key cak
> > "tttttvvvv"
> > set security macsec connectivity-association MAC exclude-protocol lldp
> > set security macsec connectivity-association MAC exclude-protocol lacp
> > set security macsec interfaces ge-0/0/0 connectivity-association MAC
> >
> >
> > Dear all, an help is appreciated and welcomme, please let me thank in
> > advance anyone will give an hint.
> >
> > Cheers
> > James
More information about the juniper-nsp
mailing list