[c-nsp] Xconnect me3600x - asr920 stops receiving packets

Spyros Kakaroukas s.kakaroukas at connecticore.com
Fri Sep 8 06:39:22 EDT 2017


Hey Fredrik,

I’ve seen LSPs failing after reconvergence on me3600x in the past, though it has been very rare. IIRC, it hasn’t even happened on our current IOS version ( 15.4(3)S6a ) , but that might be just luck. When it happened, I would use mpls ping and mpls traceroute to find the exact point it’s failing at and then look at my topology map for redundant paths. Then I would just drain the traffic from them and shut/no shut the interface on both sides. That would fix it ( ie, I could push traffic around even via the interface that was previously eating said packets, and it would work, even after an actual reconvergence event ).

Assuming you run a somewhat recent IOS version and can reproduce the issue at will, I would definitely contact TAC.

My thoughts and words are my own.

Spyros

On 08/09/2017, 13:10, "cisco-nsp on behalf of Fredrik Lund" <cisco-nsp-bounces at puck.nether.net on behalf of Fredrik.Lund at nc-spectrum.no> wrote:

    Hi again,
    In advanced sorry for the long mail.

    It seems as whenever theres a ospf route change somewhere it stops.
    Acces and core interface on ASR920 is NNI, so theres a lot of xconnects on them and all of them are working parallel to this.

    ME3600#show mpls l2transport binding 99300663
      Destination Address: x.x.x.x,VC ID: 99300663
        Local Label:  106
            Cbit: 1,    VC Type: Ethernet,    GroupID: 0
            MTU: 9216,   Interface Desc: XXXXXXXX
            VCCV: CC Type: CW [1], RA [2]
                  CV Type: LSPV [2], BFD/Raw [5]
        Remote Label: 245
            Cbit: 1,    VC Type: Ethernet,    GroupID: 32
            MTU: 9216,   Interface Desc: XXXXXXXX
            VCCV: CC Type: CW [1], RA [2]
                  CV Type: LSPV [2]

    ME3600#show mpls forwarding-table labels 106 det
    Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
    Label      Label      or Tunnel Id     Switched      interface
    106        No Label   l2ckt(2)         10616         Gi0/1      point2point
            MAC/Encaps=0/0, MRU=0, Label Stack{}
            No output feature configured

    ME3600#show mpls l2 vc 99300663 det
    Local interface: Gi0/1 up, line protocol up, Eth VLAN 1566 up
      Interworking type is Ethernet
      Destination address X.X.64.125, VC ID: 99300663, VC status: up
        Output interface: Vl1414, imposed label stack {68 315}
        Preferred path: not configured
        Default path: active
        Next hop: X.X.72.113
      Create time: 01:13:03, last status change time: 00:03:37
        Last label FSM state change time: 00:03:37
      Signaling protocol: LDP, peer X.X.64.125:0 up
        Targeted Hello: X.X.72.30(LDP Id) -> X.X.64.125, LDP is UP
        Graceful restart: not configured and not enabled
        Non stop routing: not configured and not enabled
        Status TLV support (local/remote)   : enabled/supported
          LDP route watch                   : enabled
          Label/status state machine        : established, LruRru
          Last local dataplane   status rcvd: No fault
          Last BFD dataplane     status rcvd: Not sent
          Last BFD peer monitor  status rcvd: No fault
          Last local AC  circuit status rcvd: No fault
          Last local AC  circuit status sent: No fault
          Last local PW i/f circ status rcvd: No fault
          Last local LDP TLV     status sent: No fault
          Last remote LDP TLV    status rcvd: No fault
          Last remote LDP ADJ    status rcvd: No fault
        MPLS VC labels: local 326, remote 315
        Group ID: local 0, remote 32
        MTU: local 9216, remote 9216
        Remote interface description: XXXXXXXX
      Sequencing: receive disabled, send disabled
      Control Word: On (configured: autosense)
      Dataplane:
        SSM segment/switch IDs: 8217/8209 (used), PWID: 2
      VC statistics:
        transit packet totals: receive 54, send 1061
        transit byte totals:   receive 4860, send 83446
        transit packet drops:  receive 0, seq error 0, send 0

    ME3600#show int gi0/1
    GigabitEthernet0/1 is up, line protocol is up (connected)
      Hardware is Gigabit Ethernet, address is b0aa.77a0.2001 (bia b0aa.77a0.2001)
      Description: XXXXXXXXXXX
      MTU 9800 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
         reliability 255/255, txload 1/255, rxload 1/255
      Encapsulation ARPA, loopback not set
      Keepalive not set
      Full-duplex, 1000Mb/s, media type is BX10D
      input flow-control is off, output flow-control is unsupported
      ARP type: ARPA, ARP Timeout 04:00:00
      Last input 00:00:05, output 00:00:09, output hang never
      Last clearing of "show interface" counters never
      Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1888
      Queueing strategy: fifo
      Output queue: 0/40 (size/max)
      30 second input rate 859000 bits/sec, 335 packets/sec
      30 second output rate 1641000 bits/sec, 347 packets/sec
         3163048656 packets input, 1480017234601 bytes, 0 no buffer
         Received 12043091 broadcasts (11979514 multicasts)
         0 runts, 0 giants, 0 throttles
         0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
         0 watchdog, 11979514 multicast, 0 pause input
         0 input packets with dribble condition detected
         3690514016 packets output, 2340966691569 bytes, 0 underruns
         0 output errors, 0 collisions, 1 interface resets
         0 unknown protocol drops
         0 babbles, 0 late collision, 0 deferred
         0 lost carrier, 0 no carrier, 0 pause output
         0 output buffer failures, 0 output buffers swapped out

    ME3600#sh int vl 1414
    Vlan1414 is up, line protocol is up
      Hardware is EtherSVI, address is b0aa.77a0.2040 (bia b0aa.77a0.2040)
      Description: YYYYYYYYY
      Internet address is X.X.72.112/31
      MTU 9216 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
         reliability 255/255, txload 1/255, rxload 1/255
      Encapsulation ARPA, loopback not set
      Keepalive not supported
      ARP type: ARPA, ARP Timeout 04:00:00
      Last input 00:00:00, output 00:00:00, output hang never
      Last clearing of "show interface" counters never
      Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
      Queueing strategy: fifo
      Output queue: 0/40 (size/max)
      5 minute input rate 0 bits/sec, 0 packets/sec
      5 minute output rate 2000 bits/sec, 3 packets/sec
         9109246 packets input, 754908178 bytes, 0 no buffer
         Received 0 broadcasts (0 IP multicasts)
         0 runts, 0 giants, 0 throttles
         0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
         61009207 packets output, 10367856892 bytes, 0 underruns
         0 output errors, 2 interface resets
         0 unknown protocol drops
         0 output buffer failures, 0 output buffers swapped out


    -------ASR920----------


    ASR920#show mpls l2transport binding 99300663
      Destination Address: X.X72.30,VC ID: 99300663
        Local Label:  245
            Cbit: 1,    VC Type: Ethernet,    GroupID: 32
            MTU: 9216,   Interface Desc: XXXXXXXX
            VCCV: CC Type: CW [1], RA [2]
                  CV Type: LSPV [2]
        Remote Label: 106
            Cbit: 1,    VC Type: Ethernet,    GroupID: 0
            MTU: 9216,   Interface Desc: XXXXXXXX
            VCCV: CC Type: CW [1], RA [2]
                  CV Type: LSPV [2], BFD/Raw [5]

    ATB-KGB-CO-01#show mpls forwarding-table lab 245 det
    Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
    Label      Label      or Tunnel Id     Switched      interface
    245        No Label   l2ckt(65)        0             Te0/0/26   point2point
            MAC/Encaps=0/0, MRU=0, Label Stack{}
            No output feature configured

    ASR920#show mpls l2 vc 99300663 det
    Local interface: Te0/0/26 up, line protocol up, Eth VLAN 1566 up
      Interworking type is Ethernet
      Destination address: X.X.72.30, VC ID: 99300663, VC status: up
        Output interface: Te0/0/25, imposed label stack {230 106}
        Preferred path: not configured
        Default path: active
        Next hop: X.X.65.215
      Create time: 00:11:14, last status change time: 00:11:14
        Last label FSM state change time: 00:11:14
      Signaling protocol: LDP, peer X.X.72.30:0 up
        Targeted Hello: X.X.64.125(LDP Id) X.X.72.30, LDP is UP
        Graceful restart: not configured and not enabled
        Non stop routing: not configured and not enabled
        Status TLV support (local/remote)   : enabled/supported
          LDP route watch                   : enabled
          Label/status state machine        : established, LruRru
          Last local dataplane   status rcvd: No fault
          Last BFD dataplane     status rcvd: Not sent
          Last BFD peer monitor  status rcvd: No fault
          Last local AC  circuit status rcvd: No fault
          Last local AC  circuit status sent: No fault
          Last local PW i/f circ status rcvd: No fault
          Last local LDP TLV     status sent: No fault
          Last remote LDP TLV    status rcvd: No fault
          Last remote LDP ADJ    status rcvd: No fault
        MPLS VC labels: local 245, remote 106
        Group ID: local 32, remote 0
        MTU: local 9216, remote 9216
        Remote interface description: XXXXXXXX
      Sequencing: receive disabled, send disabled
      Control Word: On (configured: autosense)
      SSO Descriptor: X.X.72.30/99300663, local label: 245
      Dataplane:
        SSM segment/switch IDs: 53966/266957 (used), PWID: 65
      VC statistics:
        transit packet totals: receive 0, send 45
        transit byte totals:   receive 0, send 2880
        transit packet drops:  receive 0, seq error 0, send 0

    ASR920#sh int te0/0/26
    TenGigabitEthernet0/0/26 is up, line protocol is up
      Hardware is 24xGE-4x10GE-FIXED, address is 5897.bd91.1c9a (bia 5897.bd91.1c9a)
      Description: XXXXXXXXX
      MTU 9216 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
         reliability 255/255, txload 1/255, rxload 2/255
      Encapsulation ARPA, loopback not set
      Keepalive not set
      Full Duplex, 10000Mbps, link type is force-up, media type is SFP-LR
      output flow-control is unsupported, input flow-control is on
      ARP type: ARPA, ARP Timeout 04:00:00
      Last input never, output 1w3d, output hang never
      Last clearing of "show interface" counters never
      Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 371
      Queueing strategy: Class-based queueing
      Output queue: 0/40 (size/max)
      5 minute input rate 93913000 bits/sec, 15524 packets/sec
      5 minute output rate 35394000 bits/sec, 12194 packets/sec
         169136289831 packets input, 160607734990202 bytes, 0 no buffer
         Received 299358399 broadcasts (0 IP multicasts)
         0 runts, 0 giants, 0 throttles
         2 input errors, 2 CRC, 0 frame, 0 overrun, 0 ignored
         0 watchdog, 184325971 multicast, 0 pause input
         122518905552 packets output, 46166090926080 bytes, 0 underruns
         0 output errors, 0 collisions, 2 interface resets
         0 unknown protocol drops
         0 babbles, 0 late collision, 0 deferred
         0 lost carrier, 0 no carrier, 0 pause output
         0 output buffer failures, 0 output buffers swapped out

    ASR920#sh int te0/0/25
    TenGigabitEthernet0/0/25 is up, line protocol is up
      Hardware is 24xGE-4x10GE-FIXED, address is 5897.bd91.1c99 (bia 5897.bd91.1c99)
      Description: YYYYYYYYYY
      Internet address is X.X.65.214/31
      MTU 9216 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
         reliability 255/255, txload 97/255, rxload 5/255
      Encapsulation ARPA, loopback not set
      Keepalive set (10 sec)
      Full Duplex, 10000Mbps, link type is force-up, media type is SFP-ER
      output flow-control is unsupported, input flow-control is on
      Asymmetric Carrier-Delay Up Timer is 10 sec
      Asymmetric Carrier-Delay Down Timer is 0 msec
      ARP type: ARPA, ARP Timeout 04:00:00
      Last input 00:00:36, output 00:00:02, output hang never
      Last clearing of "show interface" counters never
      Input queue: 1/375/1286/0 (size/max/drops/flushes); Total output drops: 0
      Queueing strategy: Class-based queueing
      Output queue: 0/40 (size/max)
      5 minute input rate 196172000 bits/sec, 66083 packets/sec
      5 minute output rate 3806678000 bits/sec, 347550 packets/sec
         579601801591 packets input, 216185565795346 bytes, 0 no buffer
         Received 222 broadcasts (38 IP multicasts)
         0 runts, 0 giants, 0 throttles
         113 input errors, 113 CRC, 0 frame, 0 overrun, 0 ignored
         0 watchdog, 22997660 multicast, 0 pause input
         4720080905737 packets output, 6472879220341068 bytes, 0 underruns
         0 output errors, 0 collisions, 2 interface resets
         0 unknown protocol drops
         0 babbles, 0 late collision, 0 deferred
         0 lost carrier, 0 no carrier, 0 pause output
         0 output buffer failures, 0 output buffers swapped out
    -----Original Message-----
    From: cisco-nsp [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of James Bensley
    Sent: mandag, 4 september, 2017 21.23
    To: cisco-nsp at puck.nether.net
    Subject: Re: [c-nsp] Xconnect me3600x - asr920 stops receiving packets

    On 1 September 2017 at 14:37, Fredrik Lund <Fredrik.Lund at nc-spectrum.no> wrote:
    > Hi all,

    Hey,

    > I am facing a problem with Xconnect between ME3600X and ASR920 which suddenly stops receiving packets.
    > A side(ME3600)x:
    > VC statistics:
    >     transit packet totals: receive 31, send 64
    >     transit byte totals:   receive 6054, send 14364
    >     transit packet drops:  receive 0, seq error 0, send 0
    >
    > B side(ASR920):
    > VC statistics:
    >     transit packet totals: receive 0, send 54
    >     transit byte totals:   receive 0, send 9169
    >     transit packet drops:  receive 0, seq error 0, send 0
    >
    > When I am doing mpls ping I can see that counters on B side is
    > increasing. So I know that LDP is functioning
    >
    > When I force another output interface on xconnect on the me3600x it works, so it seems like it is something with lsp. But I can't figure out where. We don't use any MPLS TE in our net.
    > When I do show mpls forwarding-table x.x.x.x on 3600x I don't see any hits on Bytes Label Switched.

    Have you noticed any pattern such it happens after X time, X bytes, X packets? Whats the longest/shortest it has lasted for? What is the traffic you're putting over the xconnect?

    Can you share the full config for each side and software versions of the devices?

    It's not clear to me which end "stops" the ASR920 or ME3600? Are you sending traffic into the ME3600 and then "bytes/packets sent " stops incrementing or is it the ASR920 that stops showing "bytes/packets received" ?

    Can you share the output from;
    show mpls l2transport binding <VC ID>
    show mpls l2transport vc <VC ID> detail
    show int X/X (on access interface and core facing interface) show mpls forwarding <label ID> detail

    Do you see anything in the output of "show logging" when it happens, or anything in the logs of any P/PE devices between the A and B end?

    Cheers,
    James.
    _______________________________________________
    cisco-nsp mailing list  cisco-nsp at puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp
    archive at http://puck.nether.net/pipermail/cisco-nsp/
    _______________________________________________
    cisco-nsp mailing list  cisco-nsp at puck.nether.net
    https://puck.nether.net/mailman/listinfo/cisco-nsp
    archive at http://puck.nether.net/pipermail/cisco-nsp/




This e-mail and any attachment(s) contained within are confidential and are intended only for the use of the individual to whom they are addressed. The information contained in this communication may be privileged, or exempt from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender and delete the communication without retaining any copies. Connecticore SA is not responsible for, nor endorses, any opinion, recommendation, conclusion, solicitation, offer or agreement or any information contained in this communication.


More information about the cisco-nsp mailing list