[c-nsp] BGP Path Selection and next-hop reachability (IGP vs BGP)

[John Neiberger]

I ran into an interesting situation where I think I understand what is
happening, but I can't find any documentation about the path selection
process that specifically addresses this.

We have a router--let's call it Router A--that has learned a prefix via
iBGP from two route reflector clients. The next hop addresses in those
advertisements are the loopback addresses of the advertising routers. Those
loopback addresses are being advertised into OSPF. So, this router has two
available paths for this prefix:

1: 4.4.4.4 (loopback address of first RR client, learned via OSPF)
2: 5.5.5.5 (loopback address of second RR client, learned via OSPF)

Now, the weirdness happens when the second router experiences
unidirectional traffic and stops advertising anything at all to its
upstream neighbor. Within just a few seconds, OSPF times out, so 5.5.5.5
disappears from OSPF because that router is now isolated.

Now, you must know that Router A also has learned a default route via eBGP.
So, the available paths in the BGP table for a particular prefix now look
like this:

1: 4.4.4.4 (learned via OSPF)
2: 5.5.5.5 (recursively reachable via 0/0 learned from eBGP)

The router switches to the second path, errantly sending packets with a
next hop of 5.5.5.5--which is actually unreachable--out to the upstream
router advertising the default route.

Here is the BGP table before the "outage":

R2#show ip bgp 100.100.100.0/24
BGP routing table entry for 100.100.100.0/24, version 12
Paths: (3 available, best #3, table Default-IP-Routing-Table)
Flag: 0x900
  Advertised to update-groups:
        1    2    3
  Local, (Received from a RR-client)
    5.5.5.5 (metric 102) from 5.5.5.5 (100.100.100.100)
      Origin incomplete, metric 0, localpref 100, valid, internal
  Local
    5.5.5.5 (metric 102) from 23.23.23.3 (35.35.35.3)
      Origin incomplete, metric 0, localpref 100, valid, internal
      Originator: 100.100.100.100, Cluster list: 35.35.35.3
  Local, (Received from a RR-client)
    4.4.4.4 (metric 101) from 4.4.4.4 (4.4.4.4)
      Origin incomplete, metric 0, localpref 100, valid, internal, best

The best path is--correctly--through 4.4.4.4. Here is what the BGP table
looked like when 5.5.5.5 disappeared from OSPF:

R2#show ip bgp 100.100.100.0/24
BGP routing table entry for 100.100.100.0/24, version 13
Paths: (3 available, best #1, table Default-IP-Routing-Table)
Flag: 0x900
  Advertised to update-groups:
        1    2    3
  Local, (Received from a RR-client)
    5.5.5.5 from 5.5.5.5 (100.100.100.100)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
  Local
    5.5.5.5 from 23.23.23.3 (35.35.35.3)
      Origin incomplete, metric 0, localpref 100, valid, internal
      Originator: 100.100.100.100, Cluster list: 35.35.35.3
  Local, (Received from a RR-client)
    4.4.4.4 (metric 101) from 4.4.4.4 (4.4.4.4)
      Origin incomplete, metric 0, localpref 100, valid, internal


My question is this: Why specifically does this router switch to the path
with a next hop learned via BGP? I'm assuming that a next hop reachable by
eBGP is preferred to a next hop reachable via an IGP, but I don't see
anything in the stated path selection process that would account for that.
Nothing else about the paths changed since BGP hadn't changed. The only
thing that changed was how the second loopback was reachable. It originally
was OSPF and now is recursively reachable through the default route.

Once the BGP session to the failed router times out, the second path is
removed and all is well again.

Any thoughts? I feel like I'm missing something obvious. I've been staring
at this too long.  :)