[c-nsp] BGP Hold time expired/ospf dropping 6500 Sup720-3BXL

[Andy B.]

Hi,

I just fell over this thread while doing a little reseach to solve a
similar situation.

Hardware:

- 6509 with SUP720-3BXL on both ends
- SXF15a
- Uptime: 46 weeks

Problem:

- OSPF (for the loopback between cores) and BGP (mostly customers whom
we send the full table) going up and down all the time:

%OSPF-5-ADJCHG: Process 1, Nbr x.x.x.130 on TenGigabitEthernet4/1 from
FULL to DOWN, Neighbor Down: Dead timer expired
%OSPF-5-ADJCHG: Process 1, Nbr x.x.x.131 on TenGigabitEthernet9/1 from
LOADING to FULL, Loading Done
%BGP-5-ADJCHANGE: neighbor y.y.y.14 Down BGP Notification sent
%BGP-3-NOTIFICATION: sent to neighbor y.y.y.14 4/0 (hold time expired) 0 bytes
%BGP-5-ADJCHANGE: neighbor y.y.y.14 Up

This keeps going on for several hours, and suddenly it stabilizes itself.

Furthermore I use cacti to generate graphs from the core router via
SNMP. I have one VLAN that has around 15 GBPS traffic at peak times,
and as soon as I hit more than 15 GBPS, no more graphs are drawn, core
router console becomes rather unresponsive and OSPF starts to behave
strangely.

What I can rule out is the fiber capacity. I have multiple circuits
and different paths and operators. The OSPF issue happens on all
circuits, not just a specific one. No 10 GE link is used more than
60%. In fact, traffic from inside my backbone to any place outside
remains unaffected (thank God), but the core router itself is pretty
useless. Pinging the core's loopback or any ip loaded on that box
results in a 40-60% packet loss.

CPU usage is not high, it's stable. No unusual processes, just IP
Input and BGP Scanner. More than 50% memory is still free at that
time.

I've had this many times recently, but it really just happens when my
core goes beyond +- 15 GBPS of traffic (outbound). We've been below 15
GBPS for 2 years and it never happaned at that time. Now all this mess
happens almost daily, rendering important billing graphs useless and
annoying full table BGP customers.

Is this a memory issue, due to the router's long uptime? Would
reloading the router help in this case? That's the last thing I would
want to do, but if it helps...

Cheers,

Andy

On Fri, Dec 11, 2009 at 5:22 PM, Drew Weaver <drew.weaver at thenap.com> wrote:
> Howdy all,
>
> Last night I had an interesting encounter on one of my 6509s /w SUP7203-BXL.
>
> This switch has 3x iBGP sessions with full internet tables and is also running OSPF.
>
> Two of the three iBGP sessions randomly dropped with:
>
> %BGP-3-NOTIFICATION: sent to neighbor x.x.x.3 4/0 (hold time expired) 0 bytes, I also noticed that during this period OSPF dropped with Neighbor Down: Dead timer expired
>
> and then re-established, and then failed again, and re-established, and failed again, and so-on, and so-on.
>
> I checked the physical interfaces between this 6500 and the two GSR 12000s it peers with and there were no errors, there was also no obvious spike in traffic that would account for latency that might cause the hold timers to expire. I remember when this system first came online it took a really long time for it to download the full internet tables from the upstream GSRs and also during that time there was a lot of CPU time being eaten up, I am wondering if maybe the first session failing caused sort of a 'performance' domino effect which then caused everything else to fail, the issue eventually corrected itself and stabilized.
>
> This particular box is running 12.2(18)SXF17 so I am less likely to believe it is a software bug.
>
> Does anyone have any tips on both how I can avoid the hold timer issue altogether and also how I can make it so that if a session does go down and re-establish it doesn't totally nail the CPU while it's trying to re-establish/download the routes? A long time ago I also read that increasing the MTU on both ends of a circuit can make BGP tables download faster, I don't know if that's true or not, has anyone else found that?
>
> thanks,
> -Drew
>
>
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>