[c-nsp] BGP ghosts in 12.2(25)S5

[Bernhard Schmidt]

Hi,

the IPv6 prefix 2001:4c90::/32 was withdrawn from the global routing
table due to filters at their upstream some days ago. According to
http://grh.sixxs.net, my routers were the only ones still having that
prefix and announcing it to the distributed looking glass.

I have two NPE-G1 running 12.2(25)S5 Service Provider image, each of
them running a BGP-Session to both upstreams through native GE. Each of
them still knew this prefix via both upstreams (which did not have it
anymore, at least one of them submits to the same routeserver). After
clearing the session the prefix is indeed gone.

The output looks very weird:

BACK1-C7206-OTN-SATIP#sh bgp ipv6 unicast 2001:4c90::/32
BGP routing table entry for 2001:4C90::/32, version 47058
Paths: (4 available, best #3, table Global-IPv6-Table)
Flag: 0x19E0
  Advertised to update-groups:
     1          2          3          5         
  6453 10566 6939 14277 3549 12779 20912, (received-only)
    2001:5A0:701:1::1 from 2001:5A0:701:1::1 (195.219.142.254)
      Origin IGP, localpref 100, valid, external
  8767 1273 6830 6830 6830 6830 6939 14277 3549 12779 20912, (received-only)
    2001:A60:0:201::1:1 from 2001:A60:0:201::1:1 (62.245.135.1)
      Origin IGP, localpref 100, valid, external
      Community: 8767:3000
  8767 3549 11537 17579 9270 2200 20965 3356 6175 12779 20912
    2001:A60:0:201::1:1 from 2001:A60:0:201::1:1 (62.245.135.1)
      Origin IGP, metric 5, localpref 110, valid, external, best
      Community: 3549:1246 3549:30840 8767:3050 8767:6000 29259:2100 29259:2170 29259:2171
  8767 3549 11537 17579 9270 2200 20965 3356 6175 12779 20912, (received-only)
    2001:A60:0:201::1:1 from 2001:A60:0:201::1:1 (62.245.135.1)
      Origin IGP, localpref 100, valid, external
      Community: 3549:1246 3549:30840 8767:3050 8767:6000

there are two received-only entries for the same prefix from the same
neighbor. IIRC this should never happen. I recall seeing similar things in
IPv4, too, and sometimes even with different paths. Mostly 8767 is involved
(a Juniper M-series on their side), but there are several with 6453, too,
which is some Cisco gear.

A very ugly example:

BACK1-C7206-OTN-SATIP#sh bgp ipv6 unicast 2001:1b70::/32
BGP routing table entry for 2001:1B70::/32, version 44391
Paths: (10 available, best #1, table Global-IPv6-Table)
Flag: 0x1960
  Advertised to update-groups:
     1          2          3          5         
  29259 29259 29259 8767 3549 6830 1299 3301
    2001:A60:0:201::1:1 from 2001:A60:0:201::1:1 (62.245.135.1)
      Origin incomplete, metric 5, localpref 110, valid, external, best
      Community: 3549:1256 3549:31528 8767:3050 8767:6000 29259:2100 29259:2170 29259:2171
  29259 29259 29259 8767 3549 6830 1299 3301
    2001:1B10::12 (metric 10) from 2001:1B10::12 (83.170.0.2)
      Origin incomplete, metric 5, localpref 110, valid, internal
      Community: 3549:1256 3549:31528 8767:3050 8767:6000 29259:2100 29259:2170 29259:2171
  29259 29259 29259 6453 10566 6175 6830 6830 6830 1299 3301
    2001:5A0:701:1::1 from 2001:5A0:701:1::1 (195.219.142.254)
      Origin incomplete, metric 20, localpref 110, valid, external
      Community: 29259:2100 29259:2160 29259:2161
  6453 10566 6175 6830 6830 6830 1299 3301, (received-only)
    2001:5A0:701:1::1 from 2001:5A0:701:1::1 (195.219.142.254)
      Origin incomplete, localpref 100, valid, external
  8767 3549 6830 1299 3301, (received-only)
    2001:A60:0:201::1:1 from 2001:A60:0:201::1:1 (62.245.135.1)
      Origin incomplete, localpref 100, valid, external
      Community: 3549:1256 3549:31528 8767:3050 8767:6000
  8767 3549 6830 1299 3301, (received-only)
    2001:A60:0:201::1:1 from 2001:A60:0:201::1:1 (62.245.135.1)
      Origin incomplete, localpref 100, valid, external
      Community: 3549:1256 3549:31528 8767:3050 8767:6000
  8767 3549 6830 1299 3301, (received-only)
    2001:A60:0:201::1:1 from 2001:A60:0:201::1:1 (62.245.135.1)
      Origin incomplete, localpref 100, valid, external
      Community: 3549:1256 3549:31528 8767:3050 8767:6000
  8767 3549 6830 1299 3301, (received-only)
    2001:A60:0:201::1:1 from 2001:A60:0:201::1:1 (62.245.135.1)
      Origin incomplete, localpref 100, valid, external
      Community: 3549:1256 3549:31528 8767:3050 8767:6000
  8767 3549 11537 17579 9270 2200 20965 1299 3301
    2001:A60:0:201::1:1 from 2001:A60:0:201::1:1 (62.245.135.1)
      Origin incomplete, metric 5, localpref 110, valid, external
      Community: 3549:1246 3549:30840 8767:3050 8767:6000 29259:2100 29259:2170 29259:2171
  8767 3549 11537 17579 9270 2200 20965 1299 3301, (received-only)
    2001:A60:0:201::1:1 from 2001:A60:0:201::1:1 (62.245.135.1)
      Origin incomplete, localpref 100, valid, external
      Community: 3549:1246 3549:30840 8767:3050 8767:6000

(I have more, one I just saw had 20 available paths ... with two upstreams
(soft-reconfiguration inbound) and the other backbone router being iBGP
peer without soft-reconfiguration it should be 5 at max).

How can this happen? To my knowledge, if the same neighbor sends you a 
new path for an already known prefix the old one is implicitly withdrawn.
I think those issues started in S5, I can't remember having them in S4.

Bernhard