[c-nsp] Crash iBGP on 7600 during mVPN reconfiguration

[Anrey Teslenko]

Hello Rob,

I didn't think that "snowball of route-churn/high cpu" was a reason of
problem,
because after each command I looked  statistic of cpu and waited for
normalization load.

In addition, when I first encountered this problem I just  perform command
"no ip vfr IPTV" (only one vrf instanse) and the router, where i did this,
is not bsr or rp candidate and have no problem with cpu.
However when i used workaround of CSCse41600 ( I tried investigate this
after reload,  becouse did not do  "wr mem" ) the problem isn't repeated.

I repeated this is step by step on bsr-candidate also (this is other router,
also 7604), where I observe high cpu utilization. But how i said above:
"after each command I looked  statistic of cpu and waited for normalization
load".

#sh processes cpu
CPU utilization for five seconds: 59%/54%; one minute: 61%; five minutes:
61%

How you see the cpu usage is far to 100%, but befor implementation MVPN
normaly cpu was used to 5%.

I have also another router, which is RP-candidate (also 7604, SRE2), and cpu
utilisation on it lower 5%.
This router also participating in MVPN

>> Did you investigate the high cpu on the bsr-candidate node before
starting down this process of ripping out vrf relavant configs?

 I tried execute fallowing commands:
 On both RP & BSR:
 mls rate-limit multicast ipv4 fib-miss 10 100 (doesn't help)
 no ip pim v1-rp-reachability (doesn't help)

Then I removed first router from bsr-candidate and second one from
rp-candidate,
so I  am not left bsr or rp in my network  any more.
Note: Source multicast was switched off all the time

When this not help. I tried undress multicast configuration on router
which was BSR step by step (slowly).
And how i wrote below  only after comand "no  mdt default"
(vrf wasn't yet deleted) the casus occured.

In logging output i observed
The First --- all LDP sessions are down
%LDP-5-NBRCHG: LDP Neighbor 213.xx.xx.xx:0 (10) is DOWN (Received error
notification from peer: Holddown time expired)

The Second -- status for all iBGP sessions are down ("clear ip bgp *"
doesn't help)
Note: the status eBGP sessions was UP

BGP-5-ADJCHANGE: neighbor 213.xx.xx.xx Down Peer closed the session


2010/10/12 Rob Taylor <robetayl at cisco.com>

> Hi Anrey,
>
> Did you investigate the high cpu on the bsr-candidate node before starting
> down this process of ripping out vrf relavant configs?
>
> Sounds like you may have caused iBGP to go down (I would not term it a
> crash), likely due to high cpu resulting from the removal of multiple vrfs
> at the same time.  Seems that all the configuration removal that you did was
> just a continuous snowball of route-churn/high cpu inducing commands.
>
> If I were you, I would go back to the scenario where you just had high cpu
> on the one device, and where the full multicast configuration was deployed
> as had been successfully tested, and get TAC on the line to see how they are
> related.
>
> Do you have any output captured from when you had high cpu the first time?
>
> Rob
>
>
>
>
> On 10/12/2010 9:32 AM, Anrey Teslenko wrote:
>
>> Hello all,
>>
>> We are trying to implement multicast VPN in our network.
>> On PE routers which we use now installed 12.2 (33r) SRE2    IOS
>> The hardware platform is 7604 RSP720
>>
>> The typical configuration was implemented on routers and was successful
>> tested, but
>> after finishing the tests, we were observed high cpu utilization on
>> bsr-candidate (not on RP)
>> and on it BGP neighbor, which don't participate in multicast routing.
>>
>> However this no so badly how  the removal vrfs (no ip vrf) which
>> participate
>> in MVPN.
>> We were observed crash of all iBGP session after this command was applied.
>>
>> I find only the case CSCse41600 which nearest for my problem, but the
>> router
>> not crashed, only iBGP.
>> Bgp sessions were cleared  but this don't help and only after full reload
>> problem was resolved.
>> Workaround (CSCse41600) doesn't help -- I trying delete MVPN step by step,
>> the first was replaced bgp configuration, then has removed ip pim  and ip
>> forwarding vrf from
>> all used interfaces, then "no multicast routing", only after that I began
>> remove configuration in vrf line by line.
>> However after deleting default tree configuration (no mdt default ) iBGP
>> sessions crashed  again.
>>
>>
>> Can someone please give me some hints to solve this problem?
>> Can it's not  good idea to use SRE2 for MVPN?
>>
>>
>> Thanks in advance!
>> _______________________________________________
>> cisco-nsp mailing list  cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>
>>
>>
>
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>