[c-nsp] Xconnect me3600x - asr920 stops receiving packets
Fredrik Lund
Fredrik.Lund at nc-spectrum.no
Fri Sep 8 07:11:52 EDT 2017
Hi Spyros,
When this is happening mpls ping and mpls traceroute returns fine and doesnt reveal any problems. - What do you mean with "find the exact point its failing"?
What we discovered is if we do shut/no shut on the outgoing interface for the xconnect on the ME3600 it works.
Other MPLS traffic being forwarded through the same ME3600 isnt affected as far as we can see.
We are running IOS 15.4(3)S5
-----Original Message-----
From: Spyros Kakaroukas [mailto:s.kakaroukas at connecticore.com]
Sent: fredag, 8 september, 2017 12.39
To: Fredrik Lund <Fredrik.Lund at nc-spectrum.no>; cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] Xconnect me3600x - asr920 stops receiving packets
Hey Fredrik,
I’ve seen LSPs failing after reconvergence on me3600x in the past, though it has been very rare. IIRC, it hasn’t even happened on our current IOS version ( 15.4(3)S6a ) , but that might be just luck. When it happened, I would use mpls ping and mpls traceroute to find the exact point it’s failing at and then look at my topology map for redundant paths. Then I would just drain the traffic from them and shut/no shut the interface on both sides. That would fix it ( ie, I could push traffic around even via the interface that was previously eating said packets, and it would work, even after an actual reconvergence event ).
Assuming you run a somewhat recent IOS version and can reproduce the issue at will, I would definitely contact TAC.
My thoughts and words are my own.
Spyros
On 08/09/2017, 13:10, "cisco-nsp on behalf of Fredrik Lund" <cisco-nsp-bounces at puck.nether.net on behalf of Fredrik.Lund at nc-spectrum.no> wrote:
Hi again,
In advanced sorry for the long mail.
It seems as whenever theres a ospf route change somewhere it stops.
Acces and core interface on ASR920 is NNI, so theres a lot of xconnects on them and all of them are working parallel to this.
ME3600#show mpls l2transport binding 99300663
Destination Address: x.x.x.x,VC ID: 99300663
Local Label: 106
Cbit: 1, VC Type: Ethernet, GroupID: 0
MTU: 9216, Interface Desc: XXXXXXXX
VCCV: CC Type: CW [1], RA [2]
CV Type: LSPV [2], BFD/Raw [5]
Remote Label: 245
Cbit: 1, VC Type: Ethernet, GroupID: 32
MTU: 9216, Interface Desc: XXXXXXXX
VCCV: CC Type: CW [1], RA [2]
CV Type: LSPV [2]
ME3600#show mpls forwarding-table labels 106 det
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
106 No Label l2ckt(2) 10616 Gi0/1 point2point
MAC/Encaps=0/0, MRU=0, Label Stack{}
No output feature configured
ME3600#show mpls l2 vc 99300663 det
Local interface: Gi0/1 up, line protocol up, Eth VLAN 1566 up
Interworking type is Ethernet
Destination address X.X.64.125, VC ID: 99300663, VC status: up
Output interface: Vl1414, imposed label stack {68 315}
Preferred path: not configured
Default path: active
Next hop: X.X.72.113
Create time: 01:13:03, last status change time: 00:03:37
Last label FSM state change time: 00:03:37
Signaling protocol: LDP, peer X.X.64.125:0 up
Targeted Hello: X.X.72.30(LDP Id) -> X.X.64.125, LDP is UP
Graceful restart: not configured and not enabled
Non stop routing: not configured and not enabled
Status TLV support (local/remote) : enabled/supported
LDP route watch : enabled
Label/status state machine : established, LruRru
Last local dataplane status rcvd: No fault
Last BFD dataplane status rcvd: Not sent
Last BFD peer monitor status rcvd: No fault
Last local AC circuit status rcvd: No fault
Last local AC circuit status sent: No fault
Last local PW i/f circ status rcvd: No fault
Last local LDP TLV status sent: No fault
Last remote LDP TLV status rcvd: No fault
Last remote LDP ADJ status rcvd: No fault
MPLS VC labels: local 326, remote 315
Group ID: local 0, remote 32
MTU: local 9216, remote 9216
Remote interface description: XXXXXXXX
Sequencing: receive disabled, send disabled
Control Word: On (configured: autosense)
Dataplane:
SSM segment/switch IDs: 8217/8209 (used), PWID: 2
VC statistics:
transit packet totals: receive 54, send 1061
transit byte totals: receive 4860, send 83446
transit packet drops: receive 0, seq error 0, send 0
ME3600#show int gi0/1
GigabitEthernet0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is b0aa.77a0.2001 (bia b0aa.77a0.2001)
Description: XXXXXXXXXXX
MTU 9800 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, media type is BX10D
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:05, output 00:00:09, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1888
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 859000 bits/sec, 335 packets/sec
30 second output rate 1641000 bits/sec, 347 packets/sec
3163048656 packets input, 1480017234601 bytes, 0 no buffer
Received 12043091 broadcasts (11979514 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 11979514 multicast, 0 pause input
0 input packets with dribble condition detected
3690514016 packets output, 2340966691569 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
ME3600#sh int vl 1414
Vlan1414 is up, line protocol is up
Hardware is EtherSVI, address is b0aa.77a0.2040 (bia b0aa.77a0.2040)
Description: YYYYYYYYY
Internet address is X.X.72.112/31
MTU 9216 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 2000 bits/sec, 3 packets/sec
9109246 packets input, 754908178 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
61009207 packets output, 10367856892 bytes, 0 underruns
0 output errors, 2 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
-------ASR920----------
ASR920#show mpls l2transport binding 99300663
Destination Address: X.X72.30,VC ID: 99300663
Local Label: 245
Cbit: 1, VC Type: Ethernet, GroupID: 32
MTU: 9216, Interface Desc: XXXXXXXX
VCCV: CC Type: CW [1], RA [2]
CV Type: LSPV [2]
Remote Label: 106
Cbit: 1, VC Type: Ethernet, GroupID: 0
MTU: 9216, Interface Desc: XXXXXXXX
VCCV: CC Type: CW [1], RA [2]
CV Type: LSPV [2], BFD/Raw [5]
ATB-KGB-CO-01#show mpls forwarding-table lab 245 det
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
245 No Label l2ckt(65) 0 Te0/0/26 point2point
MAC/Encaps=0/0, MRU=0, Label Stack{}
No output feature configured
ASR920#show mpls l2 vc 99300663 det
Local interface: Te0/0/26 up, line protocol up, Eth VLAN 1566 up
Interworking type is Ethernet
Destination address: X.X.72.30, VC ID: 99300663, VC status: up
Output interface: Te0/0/25, imposed label stack {230 106}
Preferred path: not configured
Default path: active
Next hop: X.X.65.215
Create time: 00:11:14, last status change time: 00:11:14
Last label FSM state change time: 00:11:14
Signaling protocol: LDP, peer X.X.72.30:0 up
Targeted Hello: X.X.64.125(LDP Id) X.X.72.30, LDP is UP
Graceful restart: not configured and not enabled
Non stop routing: not configured and not enabled
Status TLV support (local/remote) : enabled/supported
LDP route watch : enabled
Label/status state machine : established, LruRru
Last local dataplane status rcvd: No fault
Last BFD dataplane status rcvd: Not sent
Last BFD peer monitor status rcvd: No fault
Last local AC circuit status rcvd: No fault
Last local AC circuit status sent: No fault
Last local PW i/f circ status rcvd: No fault
Last local LDP TLV status sent: No fault
Last remote LDP TLV status rcvd: No fault
Last remote LDP ADJ status rcvd: No fault
MPLS VC labels: local 245, remote 106
Group ID: local 32, remote 0
MTU: local 9216, remote 9216
Remote interface description: XXXXXXXX
Sequencing: receive disabled, send disabled
Control Word: On (configured: autosense)
SSO Descriptor: X.X.72.30/99300663, local label: 245
Dataplane:
SSM segment/switch IDs: 53966/266957 (used), PWID: 65
VC statistics:
transit packet totals: receive 0, send 45
transit byte totals: receive 0, send 2880
transit packet drops: receive 0, seq error 0, send 0
ASR920#sh int te0/0/26
TenGigabitEthernet0/0/26 is up, line protocol is up
Hardware is 24xGE-4x10GE-FIXED, address is 5897.bd91.1c9a (bia 5897.bd91.1c9a)
Description: XXXXXXXXX
MTU 9216 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full Duplex, 10000Mbps, link type is force-up, media type is SFP-LR
output flow-control is unsupported, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 1w3d, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 371
Queueing strategy: Class-based queueing
Output queue: 0/40 (size/max)
5 minute input rate 93913000 bits/sec, 15524 packets/sec
5 minute output rate 35394000 bits/sec, 12194 packets/sec
169136289831 packets input, 160607734990202 bytes, 0 no buffer
Received 299358399 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
2 input errors, 2 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 184325971 multicast, 0 pause input
122518905552 packets output, 46166090926080 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
ASR920#sh int te0/0/25
TenGigabitEthernet0/0/25 is up, line protocol is up
Hardware is 24xGE-4x10GE-FIXED, address is 5897.bd91.1c99 (bia 5897.bd91.1c99)
Description: YYYYYYYYYY
Internet address is X.X.65.214/31
MTU 9216 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 97/255, rxload 5/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 10000Mbps, link type is force-up, media type is SFP-ER
output flow-control is unsupported, input flow-control is on
Asymmetric Carrier-Delay Up Timer is 10 sec
Asymmetric Carrier-Delay Down Timer is 0 msec
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:36, output 00:00:02, output hang never
Last clearing of "show interface" counters never
Input queue: 1/375/1286/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: Class-based queueing
Output queue: 0/40 (size/max)
5 minute input rate 196172000 bits/sec, 66083 packets/sec
5 minute output rate 3806678000 bits/sec, 347550 packets/sec
579601801591 packets input, 216185565795346 bytes, 0 no buffer
Received 222 broadcasts (38 IP multicasts)
0 runts, 0 giants, 0 throttles
113 input errors, 113 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 22997660 multicast, 0 pause input
4720080905737 packets output, 6472879220341068 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
-----Original Message-----
From: cisco-nsp [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of James Bensley
Sent: mandag, 4 september, 2017 21.23
To: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] Xconnect me3600x - asr920 stops receiving packets
On 1 September 2017 at 14:37, Fredrik Lund <Fredrik.Lund at nc-spectrum.no> wrote:
> Hi all,
Hey,
> I am facing a problem with Xconnect between ME3600X and ASR920 which suddenly stops receiving packets.
> A side(ME3600)x:
> VC statistics:
> transit packet totals: receive 31, send 64
> transit byte totals: receive 6054, send 14364
> transit packet drops: receive 0, seq error 0, send 0
>
> B side(ASR920):
> VC statistics:
> transit packet totals: receive 0, send 54
> transit byte totals: receive 0, send 9169
> transit packet drops: receive 0, seq error 0, send 0
>
> When I am doing mpls ping I can see that counters on B side is
> increasing. So I know that LDP is functioning
>
> When I force another output interface on xconnect on the me3600x it works, so it seems like it is something with lsp. But I can't figure out where. We don't use any MPLS TE in our net.
> When I do show mpls forwarding-table x.x.x.x on 3600x I don't see any hits on Bytes Label Switched.
Have you noticed any pattern such it happens after X time, X bytes, X packets? Whats the longest/shortest it has lasted for? What is the traffic you're putting over the xconnect?
Can you share the full config for each side and software versions of the devices?
It's not clear to me which end "stops" the ASR920 or ME3600? Are you sending traffic into the ME3600 and then "bytes/packets sent " stops incrementing or is it the ASR920 that stops showing "bytes/packets received" ?
Can you share the output from;
show mpls l2transport binding <VC ID>
show mpls l2transport vc <VC ID> detail
show int X/X (on access interface and core facing interface) show mpls forwarding <label ID> detail
Do you see anything in the output of "show logging" when it happens, or anything in the logs of any P/PE devices between the A and B end?
Cheers,
James.
_______________________________________________
cisco-nsp mailing list cisco-nsp at puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
This e-mail and any attachment(s) contained within are confidential and are intended only for the use of the individual to whom they are addressed. The information contained in this communication may be privileged, or exempt from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender and delete the communication without retaining any copies. Connecticore SA is not responsible for, nor endorses, any opinion, recommendation, conclusion, solicitation, offer or agreement or any information contained in this communication.
More information about the cisco-nsp
mailing list