[c-nsp] some weird routing problems after GSR upgrade

[Primoz Jeroncic]

Hi

After upgrade of one of our c12008 from GRP-B to PRP-1, and with
this required IOS upgrade (from service provider 12.0.30.S5 to
12.0.33.S1) I'm starting to get some really weird issues with
routing.
At the moment my network topology looks like this:

Upstream-1 -- GSR-1 -- GSR-2 -- GSR-3 -- c7401 -- c7206 -- backup upstream

Now show explanation. Route engine line card and IOS were upgraded
on GSR-1.
One prefix is originated and advertised in BGP from GSR-2.
Second prefix is originated and advertised from c7206.
In case if matters, MPLS is enabled from GSR-1 to c7401. C7206 is
not in MPLS enabled network.
Both prefixes are announced through same AS, and if everything is fine,
they both go out through GSR-1 and Upstream-1.

And now my problem. Config is same as it was before. Before everything
was working fine, and when link between c7401 and c7206 went down,
prefix originated on c7206 went out through backup upstream and came
into my network (to GSR-2) through upstream-1 and GSR-1.
After upgrade, things doesn't work like this anymore. Until I clear
route for prefix originated on c7206 on GSR-1 (clear ip route x.x.x.0),
trafic goes from GSR-2 to GSR-3, then to c7401, then back to GSR-3 and
GSR-2, and then another loop.

After link goes down, there's no routes for this prefix in GSR-3 and
c7401, so traffic should go over default route. There is route for
this prefix on GSR-2, but it's pointing to GSR-1, and on GSR-1 there's
route for this prefix through Upstream-1. So based on routing tables
everything looks fine. But traffic still bounces as described above.

I know easiest solution would be to downgrade GSR-1 to previous IOS
which was working. But with older IOS, one of Gigabitethernet linecards
in GSR-1 was crashing constantly every 15 to 20mins. Considering there
is about 200 BGP peerings on that GE, you can imagine that peerings
were down pretty much all the time :) With 12.0.33.S1, this crashes
are history, but this routing problem is something what I didn't have
before.

If anyone has problems like this, or if anyone has any idea how to solve
this, please let me know. I would be extremely greatful for any hint
on this.

Thanks in advance, and if I wasn't clear enough, please drop me note, and
I will try to explain again.

Have fun,
Primoz Jeroncic
Support - IP Connectivity & Routing
-------------------------------------------------------------------
Softnet d.o.o.  tel:  +386 1 562 31 40   |
Borovec 2       fax:  +386 1 562 18 55   |       1 + 1 = 3
1236 Trzin      primoz(at)softnet.si     | for larger values of 1
Slovenija       http://flea.softnet.si/
-------------------------------------------------------------------