[c-nsp] Bad routes in MPLS

[Pete Lumbis]

Generally these kinds of problems are triggered by routing changes. The
software that owns the routing table (show ip route/ show ip cef) needs to
update the hardware TCAM (show mls cef). This is true of both IP prefixes
and MPLS labels.

When you issue "clear ip route" you for the software to hardware
programming to happen again, generally avoiding whatever corner case
happened previously.

This is due to a bug nearly 100% of the time. Tracking down the bug you are
hitting can be a huge hassle. I would strongly recommend you upgrade to the
latest SRE code (....make sure that SRE will support your SIP, I don't
remember...) where there have been a ton of fixes in this area and go from
there.


-Pete


On Tue, Nov 19, 2013 at 6:45 PM, Tony <td_miles at yahoo.com> wrote:

> Hi all,
>
> We've been having an issue recently where we have routes on PE routers
> that look to be ok, but are not forwarding any traffic. Usually this can be
> resolved by doing "clear ip route vrf <vrf_name> <ip_prefix>" which causes
> the PE to re-learn the route and everything works again.
>
> This problem appears to be triggered by routing flap/changes. This can
> either be within a vrf (ie. PE-CE, BGP or OSPF) or our core IGP (PE-PE,
> OSPF). When it is a PE-CE route change it seems to only affect routes
> within that vrf, but when it is an IGP topology change it can affect pretty
> much any vrf.
>
> My initial though was that something is wrong in the stuff that is
> underlying the routing (eg. MPLS, LDP, CEF) and so I started looking, but
> can't see anything that is different from when it works and when it doesn't
> work. I've worked my way through most of "show ip bgp vpnv4", "show mpls
> forwarding-table", "show mpls ip binding", etc. You can also "work around
> the problem" by adding a more specific route, which would bypass the dodgy
> route in question (but not fix the route, so that when the more specific
> route is removed it goes back to not working). I've also found that
> sometimes clearing the route (clear ip route ...) doesn't work, in which
> case the best way I've found to resolve that is to create a static route
> exactly the same on the PE but with a destination of null0, then remove
> this route again. This again causes the bad route to be flushed and
> replaced with one that now works.
>
> This has been happening for a couple of weeks, but was typically only
> affecting a prefix here or there and so we thought it just a bad quirk, but
> more recently we have had some occasions where it has caused widespread
> havoc affecting a good portion of routes on a single PE.
>
> The boxes in question are 7609 with dual sup720-3B and a variety of card
> (mostly SIP-400 + SPA-GE & ES20+) running 12.2(33)SRD4. I'm aware the
> software is a bit out of date and we are looking to schedule a maintenance
> window for upgrading (at this stage a week away from now). The boxes are
> not heavily loaded from a CPU, memory or traffic perspective.
>
> Any thoughts on where I might look to try and see an actual error (ie.
> something corrupt) or any other troubleshooting that could be done. I have
> opened a case with TAC (and as per recent thread, OMFG was that PAINFUL !)
> and will see where that goes. I imagine their first response might be
> upgrade/reboot but I'm hoping not and they might have some sensible
> suggestions.
>
>
> Happy to post output of some of the troubleshooting I've already done, but
> thought that would just add a heap of extra information to this post and
> wuld just cause people's eye's to glaze over.
>
> Thanks,
> Tony.
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>