[c-nsp] EoMPLS VC gettings stuck (Sup32, SXI2a)?

[Gert Doering]

Hi,

"weird happenings" day...  this started out as a query for a problem, now
I found a workaround, and this is now "for the record"... (and in the hope
that someone knows whether this is fixed in a recent IOS).


I have two 6500s happily EoMPLSing to each other (and to other routers
in the network) without any issues so far.

So we set up a new EoMPLS link, which worked fine for a few days.  Very
basic config:

End A (Sup32-10G, SXI2a):

interface GigabitEthernet2/6
 no ip address
 xconnect 1.1.1.74 11240001 encapsulation mpls
end

End B (Sup720-3B, SXF13a):

interface GigabitEthernet3/8
 no ip address
 xconnect 1.1.1.67 11240001 encapsulation mpls
end

The VC is still up and well:


Cisco#sh mpls l2transport vc 11240001 det
Local interface: Gi2/6 up, line protocol up, Ethernet up
  Destination address: 1.1.1.74, VC ID: 11240001, VC status: up
    Output interface: Vl13, imposed label stack {154 83}
    Preferred path: not configured  
    Default path: active
    Next hop: 1.2.3.66
  Create time: 00:09:26, last status change time: 00:09:22
  Signaling protocol: LDP, peer 1.1.1.74:0 up
    Targeted Hello: 1.1.1.67(LDP Id) -> 1.1.1.74
    MPLS VC labels: local 42, remote 83 
    Group ID: local 0, remote 0
    MTU: local 1500, remote 1500
    Remote interface description: SWM: XXX
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 69, send 0
    byte totals:   receive 37069, send 0
    packet drops:  receive 0, send 0


Here, the problem is already visible: the VC is not sending packets.

The gi2/6 port is connected to a fairly active network, so the port is
receiving tons of broadcasts:

GigabitEthernet2/6 is up, line protocol is up (connected)
...
  5 minute input rate 6000 bits/sec, 8 packets/sec
...
     8412 packets input, 903398 bytes, 0 no buffer
     Received 8237 broadcasts (0 IP multicasts)


-> but "send 0", the packets don't go out via the VC.  The *other*
direction actually works (verfied with tcpdump on one of the systems
on the network).


This is something that really baffles me - I can't see any way this 
could fail in a way that has the "vc up" but still not sending packets.

I can find lots of ways that packets might get lost (blackholing in the
network, MTU problems, etc. etc.) but all these would not be visible to
the sending router - so it would still increment its "send" counters(!!),
with the receiving router just not receiving the appropriate number of 
packets.  

But "send 0" is "the packets are getting lost inside the box" weirdness.


So, to troubleshoot this, I tried:

 - shut/no shut (no change)
 - remove vc, change port to "switchport", change back, re-add vc (no change)
 - change VC ID on both ends (no change)
 - change the IGP topology to force a different path and different labes 
   (no change)
 - move the VC to a differnet port (gi2/4 -> gi2/6, and gi2/15)
   (no change)

I might be tempted to blaim the card in question (6516A), but it has one
other active EoMPLS on Gi2/1, which happily shoves packets back and forth
just fine, and up until recently had a second one on Gi2/5...:

Cisco#sh mpls l2transport vc

Local intf     Local circuit              Dest address    VC ID      Status    
-------------  -------------------------- --------------- ---------- ----------
Gi2/1          Ethernet                   1.1.1.64   10010001   UP        
Fa3/18         Ethernet                   1.1.1.71   11210001   UP        
Fa3/19         Ethernet                   1.1.1.71   11210002   UP        
Gi2/15         Ethernet                   1.1.1.74   11240001   UP        
Gi2/5          Ethernet                   1.1.1.78   11300001   ADMIN DOWN


Mmmmh.  Gi2/5 worked until a few days ago, and was temporarily moved
to another box, and just configured to "shutdown", but not removed.

Let's remove the config, just for the fun of it...

Cisco(config)#int g2/5
Cisco(config-if)#no  xconnect 1.1.1.78 11300001 encapsulation mpls


... but what's that?  Now my *other* VC starts forwarding traffic!

Cisco-M-XLI#sh mpls l2transport vc 11240001 det
Local interface: Gi2/15 up, line protocol up, Ethernet up
  Destination address: 1.1.1.74, VC ID: 11240001, VC status: up
    Output interface: Vl13, imposed label stack {154 83}
    Preferred path: not configured  
    Default path: active
    Next hop: 1.2.3.66
  Create time: 00:09:51, last status change time: 00:09:49
  Signaling protocol: LDP, peer 1.1.1.74:0 up
    Targeted Hello: 1.1.1.67(LDP Id) -> 1.1.1.74
    MPLS VC labels: local 289, remote 83 
    Group ID: local 0, remote 0
    MTU: local 1500, remote 1500
    Remote interface description: SWM: XXX
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 76, send 1742
    byte totals:   receive 40571, send 188463
    packet drops:  receive 0, send 0


so it seems that having an EoMPLS configuration on a "shutdown" interface
can block an unrelated EoMPLS VC on the same router, but on a different VC!


- now I don't actually need advice on the problem anymore, but would of
course like to hear more about this - is this a known bug, is this just 
me again?  (I'm going to open a TAC case on this, but right now, Cisco
is somehow unable to get new contracts registered, so this router has
no active contract *sigh*).

gert

-- 
USENET is *not* the non-clickable part of WWW!
                                                           //www.muc.de/~gert/
Gert Doering - Munich, Germany                             gert at greenie.muc.de
fax: +49-89-35655025                        gert at net.informatik.tu-muenchen.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 305 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-nsp/attachments/20100222/f6c309ab/attachment.bin>