[c-nsp] 10gig Link Bouncing Consistently

[Justin Krejci]

Cisco NSP,


The Basics:

We've a customer with two separate 10gig links to a pair of our
6509-E's. Both of our 6509-E's have dual Sup720-3BXL and the 6704-10GE
line cards.

Link 1: 6509 Router#1 routed port to Netgear#1. IPv4 only. Multimode
fiber
Link 2: 6509 Router#2 routed port to Netgear#2. IPv4 only. Multimode
fiber
The two Netgears are in a virtual stack. The 6509s are not.
Netgear model: GSM7352Sv2

Beginning several days ago both links began flapping every 5 and a half
hours, both links have this happen within a couple of minutes of each
other.
They will flap for several minutes and then return to normal operation
requiring no user action and the ports never enter err-disable or show
any other symptoms.
Our Ciscos log between 5 and 10 interface drops in the time frame.
The Netgear switches log hundreds of interface drops in the same time
frame.


Further Details:

We've replaced the transceivers on all endpoints.
Light power levels are great at all end points.
All other 10gig ports on the Ciscos are operating without issue, though
they are all using single mode/LR optics.
No other related log entries or other observed/reported issues anywhere
else.
On the Cisco's there are a some old interface errors but the error
counters have not increased over the past several days.
On the Netgears there are a few thousand FCS errors during the the few
minutes of the problem occurring otherwise no errors ever.
Sup720's are not failing over

Beyond the very basic things the Ciscos are also doing OSPF and full BGP
tables. IPv4 and IPv6 is in use. HSRP used in a few places, including
these particular 10gig ports.
Management is in a VRF.

The peculiar thing is the regularity of 5.5 hour cycle which is obvious
in the logs and in the traffic graphs.

Both cisco ports are identically configured except hsrp priority and ip
address.

interface TenGigabitEthernet9/2
ip address XXXX YYYY
ip flow ingress
ip flow egress
standby 9 ip XXX
standby 9 priority 105
standby 9 preempt delay minimum 120
standby 9 authentication md5 key-string 7 XXXX
standby 9 track 1 decrement 20
no cdp enable


Is there some known issue or any suggestions at all at this point?
Compatibility issues?
An IOS upgrade seems to be another thing to try but in reading through
the IOS SXI and SXJ release notes I don't see anything obviously
related.
Or maybe even just a reboot might help.


Thanks for any advice.