[c-nsp] BGP Path Selection and next-hop reachability (IGP vs BGP)

[John Neiberger]

I've been doing some more testing and I even talked to a couple of guys
from Cisco Advanced Services and I still don't understand exactly what is
happening.

To summarize, a router has two iBGP paths available for a particular
prefix. The next hop for both paths is learned via OSPF, so the router
selects the path with the lowest IGP metric. Then a network change occurs
such that the next hop for one of the paths is no longer learned via OSPF.
Instead, it is learned via BGP. The router switches to using that path, but
I can't figure out why. It appears that a path with a next hop reachable
via BGP is preferred, but I can't find any documentation that says that.

At first I thought I'd follow the usual path selection criteria for
choosing between two iBGP paths. At that point in the process, the first
question is router ID. Is it switching to the path with the lowest router
ID? Nope. What about cluster length? Nope, it's the same. Lastly, is it
choosing this new path because the neighbor's IP address is lowest?
Again....NO!

So, what the heck? I'm really stumped.


On Fri, Nov 30, 2012 at 2:42 PM, John Neiberger <jneiberger at gmail.com>wrote:

> I ran into an interesting situation where I think I understand what is
> happening, but I can't find any documentation about the path selection
> process that specifically addresses this.
>
> We have a router--let's call it Router A--that has learned a prefix via
> iBGP from two route reflector clients. The next hop addresses in those
> advertisements are the loopback addresses of the advertising routers. Those
> loopback addresses are being advertised into OSPF. So, this router has two
> available paths for this prefix:
>
> 1: 4.4.4.4 (loopback address of first RR client, learned via OSPF)
> 2: 5.5.5.5 (loopback address of second RR client, learned via OSPF)
>
> Now, the weirdness happens when the second router experiences
> unidirectional traffic and stops advertising anything at all to its
> upstream neighbor. Within just a few seconds, OSPF times out, so 5.5.5.5
> disappears from OSPF because that router is now isolated.
>
> Now, you must know that Router A also has learned a default route via
> eBGP. So, the available paths in the BGP table for a particular prefix now
> look like this:
>
> 1: 4.4.4.4 (learned via OSPF)
> 2: 5.5.5.5 (recursively reachable via 0/0 learned from eBGP)
>
> The router switches to the second path, errantly sending packets with a
> next hop of 5.5.5.5--which is actually unreachable--out to the upstream
> router advertising the default route.
>
> Here is the BGP table before the "outage":
>
> R2#show ip bgp 100.100.100.0/24
> BGP routing table entry for 100.100.100.0/24, version 12
> Paths: (3 available, best #3, table Default-IP-Routing-Table)
> Flag: 0x900
>   Advertised to update-groups:
>         1    2    3
>   Local, (Received from a RR-client)
>     5.5.5.5 (metric 102) from 5.5.5.5 (100.100.100.100)
>       Origin incomplete, metric 0, localpref 100, valid, internal
>   Local
>     5.5.5.5 (metric 102) from 23.23.23.3 (35.35.35.3)
>       Origin incomplete, metric 0, localpref 100, valid, internal
>       Originator: 100.100.100.100, Cluster list: 35.35.35.3
>   Local, (Received from a RR-client)
>     4.4.4.4 (metric 101) from 4.4.4.4 (4.4.4.4)
>       Origin incomplete, metric 0, localpref 100, valid, internal, best
>
> The best path is--correctly--through 4.4.4.4. Here is what the BGP table
> looked like when 5.5.5.5 disappeared from OSPF:
>
> R2#show ip bgp 100.100.100.0/24
> BGP routing table entry for 100.100.100.0/24, version 13
> Paths: (3 available, best #1, table Default-IP-Routing-Table)
> Flag: 0x900
>   Advertised to update-groups:
>         1    2    3
>   Local, (Received from a RR-client)
>     5.5.5.5 from 5.5.5.5 (100.100.100.100)
>       Origin incomplete, metric 0, localpref 100, valid, internal, best
>   Local
>     5.5.5.5 from 23.23.23.3 (35.35.35.3)
>       Origin incomplete, metric 0, localpref 100, valid, internal
>       Originator: 100.100.100.100, Cluster list: 35.35.35.3
>   Local, (Received from a RR-client)
>     4.4.4.4 (metric 101) from 4.4.4.4 (4.4.4.4)
>       Origin incomplete, metric 0, localpref 100, valid, internal
>
>
> My question is this: Why specifically does this router switch to the path
> with a next hop learned via BGP? I'm assuming that a next hop reachable by
> eBGP is preferred to a next hop reachable via an IGP, but I don't see
> anything in the stated path selection process that would account for that.
> Nothing else about the paths changed since BGP hadn't changed. The only
> thing that changed was how the second loopback was reachable. It originally
> was OSPF and now is recursively reachable through the default route.
>
> Once the BGP session to the failed router times out, the second path is
> removed and all is well again.
>
> Any thoughts? I feel like I'm missing something obvious. I've been staring
> at this too long.  :)
>