[c-nsp] 2 x "new" ME3600's, 1 fails to reach one PE only in VRF's

CiscoNSP List cisconsp_list at hotmail.com
Thu Apr 30 05:42:15 EDT 2015


Just an update to this - It is still with the ME3600 dev team, and they are still are unable to identify a root cause....been with them for over a week....webex sessions, sdcli...still no joy..

Really really frustrating as it is holding up provisioning of any "new" services on these boxes.

Anyone from Cisco on the ME team read this list?  If so, would love to hear from you (Off list if more appropriate).

Cheers.


> From: cisconsp_list at hotmail.com
> To: adam.vitkovsky at gamma.co.uk; cisco-nsp at puck.nether.net
> Date: Thu, 23 Apr 2015 07:36:10 +1030
> Subject: Re: [c-nsp] 2 x "new" ME3600's, 1 fails to reach one PE only in VRF's
> 
> Hi Adam - Been webex'ing quite a bit with TAC on this, and they *believe* it's a bug, and it's currently with ME Dev team.
> 
> Anyway - Router that connects to problem PE definitely has a label for problem PE's loop, this router then connects back to an ASR1001, which also has label for problem PE's loop, ASR1001 connects to ME01, and this is where the label gets dropped....ME01 doesnt assign a local label...ME02 connects to ME01 to get to the rest of the network, so it also doesnt have a label(outgoing) for problem PE's loop.
> 
> Router that connects to problem PE:
> 
> sh mpls forwarding-table xxx.xxx.xxx.222 32
> 
> Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop    
> Label  Label or VC   or Tunnel Id      Switched      interface              
> 1737   Pop Label     xxx.xxx.xxx.222/32 638373045428  Po1.88     xxx.xxx.xxx.126
> 
> ASR1001 that connects to ME01:
> 
> sh mpls forwarding-table xxx.xxx.xxx.222 32
> Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
> L.abel      Label      or Tunnel Id     Switched      interface              
> 40420      30821      xxx.xxx.xxx.222/32   \
>                                        34411028      Gi0/0/2    xxx.xxx.xxx.239
> 
> ME01:
> 
> sh mpls forwarding-table xxx.xxx.xxx.222 32
> Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
> Label      Label      or Tunnel Id     Switched      interface              
> None       40420      xxx.xxx.xxx.222/32   \
>                                        0             Gi0/1      xxx.xxx.xxx.236
> 
> ME02:
> 
> sh mpls forwarding-table xxx.xxx.xxx.222 32
> Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
> Label      Label      or Tunnel Id     Switched      interface              
> 20625      No Label   xxx.xxx.xxx.222/32   \
>                                        0             Gi0/3      xxx.xxx.xxx.232
> 
> 
> Cheers.
> 
> From: Adam.Vitkovsky at gamma.co.uk
> To: cisconsp_list at hotmail.com; cisco-nsp at puck.nether.net
> Subject: RE: [c-nsp] 2 x "new" ME3600's, 1 fails to reach one PE only in VRF's
> Date: Wed, 22 Apr 2015 13:10:30 +0000
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Hi
> 
>  
> I was just trying to understand which router drops the ball.
> Because all other routers have working label switched paths to the remote PE it’s just this one ME that has problems –so with that
>  in mind the remote PE has to advertise label for its loopback correctly.
> But at some point the label is lost and our problem ME doesn’t get the label for this prefix.
> So I wanted to know if it’s the ME –not accepting the label for the remote PE’s loopback for some reason.
> Or if the ME is not even getting the label from the upstream router for some reason –so that is why I wanted to see mpls forwarding
>  table from the router that is upstream to the ME to see if it allocated label for the remote PE’s loopback.
>  
> ME<--PHP<--...<--PE
>  
> Can you please post output from the “sh mpls ldp nei det” –from ME and PHP for the session between them.
> And also on PHP please issue “sh mpls forwarding” for the PE’s loopback  --the PHP router should allocate a label for PE’s loopback.
>  
>  
> adam
> 
>  
> 
> 
> 
> From: CiscoNSP List [mailto:cisconsp_list at hotmail.com]
> 
> 
> Sent: 21 April 2015 04:10
> 
> To: Adam Vitkovsky; cisco-nsp at puck.nether.net
> 
> Subject: RE: [c-nsp] 2 x "new" ME3600's, 1 fails to reach one PE only in VRF's
> 
> 
>  
> 
> 
> Apologies - that was "cant" see a label...not "can"!
> 
> 
>  
> 
> 
>  
> 
> 
> > 
> 
> > Thanks Adam,
> 
> > On the PE that the problem PE connects to, I can see a label:
> 
> > 
> 
> > sh ip cef xxx.xxx.xxx.222/32 detail 
> 
> > xxx.xxx.xxx.222/32, epoch 0
> 
> > NetFlow: Origin AS 0, Peer AS 0, Mask Bits 32
> 
> > local label info: global/1737
> 
> > nexthop xxx.xxx.xxx Port-channel1.88
> 
> > 
> 
> > and for debugging, I have the following enabled(On ME3600), but not getting anything "useful" in the logs...Is there another debug option that would provide why the label is being dropped?
> 
> > 
> 
> > MPLS:
> 
> > MPLS events debugging is on
> 
> > MPLS ldp:
> 
> > LDP Label Information Base (LIB) changes debugging is on
> 
> > LDP label and address advertisements debugging is on
> 
> > LDP Configuration events debugging is on
> 
> > 
> 
> > Cheers.
> 
> > 
> 
> > 
> 
> > From: Adam.Vitkovsky at gamma.co.uk
> 
> > To: cisconsp_list at hotmail.com; 
> cisco-nsp at puck.nether.net
> 
> > Subject: RE: [c-nsp] 2 x "new" ME3600's, 1 fails to reach one PE only in VRF's
> 
> > Date: Mon, 20 Apr 2015 23:13:42 +0000
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > On the ME you can debug why the label is being dropped/not accepted for the “problem”PE’s loopback.
> 
> > Or what label, if any, is allocated for the “problem”PE’s loopback by the penultimate hop router.
> 
> > 
> 
> > 
> 
> > adam
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > From: CiscoNSP List [mailto:cisconsp_list at hotmail.com]
> 
> > 
> 
> > 
> 
> > Sent: 20 April 2015 22:49
> 
> > 
> 
> > To: Adam Vitkovsky; cisco-nsp at puck.nether.net
> 
> > 
> 
> > Subject: RE: [c-nsp] 2 x "new" ME3600's, 1 fails to reach one PE only in VRF's
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > One thing I do notice, is that the "problem" PE doesnt have a label(After nexthop) when issuing "#sh ip cef xxx detail"...all other PE's do?
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > eg. a "working" PE, from this ME3600:
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > sh ip cef xxx.xxx.xxx.217/32 detail 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > xxx.xxx.xxx.217/32, epoch 0
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > local label info: global/20661
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 2 RR sources [heavily shared]
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > nexthop xxx.xxx.xxx.232 GigabitEthernet0/3 label 333
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > vs
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > #sh ip cef xxx.xxx.xxx.222/32 detail 
> 
> > 
> 
> > 
> 
> > xxx.xxx.xxx.222/32, epoch 0
> 
> > 
> 
> > 
> 
> > local label info: global/20625
> 
> > 
> 
> > 
> 
> > nexthop xxx.xxx.xxx.232 GigabitEthernet0/3
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > > From: 
> 
> > cisconsp_list at hotmail.com
> 
> > 
> 
> > > To: adam.vitkovsky at gamma.co.uk;
> 
> > cisco-nsp at puck.nether.net
> 
> > 
> 
> > > Date: Tue, 21 Apr 2015 08:11:52 +1030
> 
> > 
> 
> > > Subject: Re: [c-nsp] 2 x "new" ME3600's, 1 fails to reach one PE only in VRF's
> 
> > 
> 
> > > 
> 
> > 
> 
> > > Thanks Adam - shutdown int vlan36, but unfortunately the issue still persists...I also did a "clear ip route *"
> 
> > 
> 
> > > 
> 
> > 
> 
> > > #sh ip cef xxx.xxx.xxx.222/32 detail 
> 
> > 
> 
> > > xxx.xxx.xxx.222/32, epoch 0
> 
> > 
> 
> > > local label info: global/20625
> 
> > 
> 
> > > nexthop xxx.xxx.xxx.232 GigabitEthernet0/3
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > #ping vrf CUSTA 10.2.4.1 
> 
> > 
> 
> > > Type escape sequence to abort.
> 
> > 
> 
> > > Sending 5, 100-byte ICMP Echos to 10.2.4.1, timeout is 2 seconds:
> 
> > 
> 
> > > .....
> 
> > 
> 
> > > Success rate is 0 percent (0/5)
> 
> > 
> 
> > > 
> 
> > 
> 
> > > If it was due to this bug, wouldnt it impact on "all/most?" traffic...i.e. the box can get to all our other PE's in VRF+Global....it's only VRF traffic to this one PE that appears to fail...Global(as mentioned) to this PE works fine?
> 
> > 
> 
> > > 
> 
> > 
> 
> > > Cheers.
> 
> > 
> 
> > > From: Adam.Vitkovsky at gamma.co.uk
> 
> > 
> 
> > > To: cisconsp_list at hotmail.com;
> 
> 
> > cisco-nsp at puck.nether.net
> 
> > 
> 
> > > Subject: RE: [c-nsp] 2 x "new" ME3600's, 1 fails to reach one PE only in VRF's
> 
> > 
> 
> > > Date: Mon, 20 Apr 2015 11:53:16 +0000
> 
> > 
> 
> > > 
> 
> > 
> 
> > > I'm sorry about that I did not follow up on this problem. 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > IP LFA is not supported on VLAN interfaces (at least not on 15.3(3)S4)
> 
> > 
> 
> > > 
> 
> > 
> 
> > > Bug: 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > Whenever IP LFA is used and a backup path is computed via VLAN interface HW forwarding gets corrupted on the box.
> 
> > 
> 
> > > 
> 
> > 
> 
> > > You are "lucky" though, as when we ran into this the box stopped forwarding all transit traffic (only traffic to the box was not affected fortunately)
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > The workaround is to never use VLAN interfaces with IP FRR.
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > So would you please try shutting down the VLAN36 interface if it helps (provided there are no other VLAN interfaces that can be used as LFA backup link)?
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > > ME02:
> 
> > 
> 
> > > 
> 
> > 
> 
> > > > #sh ip cef xxx.xxx.xxx.222/32 detail 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > > xxx.xxx.xxx.222/32, epoch 0
> 
> > 
> 
> > > 
> 
> > 
> 
> > > > local label info: global/20625
> 
> > 
> 
> > > 
> 
> > 
> 
> > > > 2 RR sources [heavily shared]
> 
> > 
> 
> > > 
> 
> > 
> 
> > > > nexthop xxx.xxx.xxx.232 GigabitEthernet0/3 label [none|674]
> 
> > 
> 
> > > 
> 
> > 
> 
> > > > repair: attached-nexthop xxx.xxx.xxx.234 Vlan36 <==========
> 
> > 
> 
> > > 
> 
> > 
> 
> > > > nexthop xxx.xxx.xxx.234 Vlan36, repair <=======
> 
> > 
> 
> > > 
> 
> > 
> 
> > > >
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > adam
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > This email has been scanned for email related threats and delivered safely by Mimecast.
> 
> > 
> 
> > > For more information please visit http://www.mimecast.com
> 
> > 
> 
> > 
> 
> > > _______________________________________________
> 
> > 
> 
> > > cisco-nsp mailing list cisco-nsp at puck.nether.net
> 
> > 
> 
> > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> 
> > 
> 
> > > archive at http://puck.nether.net/pipermail/cisco-nsp/
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > This email has been scanned for email related threats and delivered safely by Mimecast.
> 
> > For more information please visit http://www.mimecast.com
> 
> 
> > _______________________________________________
> 
> > cisco-nsp mailing list cisco-nsp at puck.nether.net
> 
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> 
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> 
> 
> 
> 
> 
> 
>   This email has been scanned for email related threats and delivered safely by Mimecast.
>  For more information please visit http://www.mimecast.com   		 	   		  
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
 		 	   		  


More information about the cisco-nsp mailing list