[c-nsp] Weird Transceiver failure after upgraded to 6509-E chassis

[Matthew Huff]

Yesterday we upgraded two of our 6509 chassis to 6509-E. Other than the tedious nature of removing/replacing the line cards, everything went well. The configuration was set for complete diagnostics, and everything came up with no errors reported. However, one transceiver on one switch, and two on another are down/down. The diagnostics show no errors nor do the interfaces report any problems. They are all configured as layer 3 devices, with no strange features enabled. Other transceivers on the same module work fine, and other modules of the same type do as well.

Switch1 (6509-E, sup720-3B, module X6704-10GE)
----------------------------------------------
Xenpack-10Gbase-LX4 down/down
Xenpack-10Gbase-LX4 down/down

Switch2 (6509-E, sup720-3B, module X6708-10GE)
----------------------------------------------
X2-10GBase-ER down/down


Config:

interface TenGigabitEthernet7/3
 ip address x.x.x.x 255.255.255.252
 ip pim sparse-dense-mode
 ipv6 address x:x:x:x::x/127
 ipv6 enable
 ipv6 eigrp x

Things I have tried:

1) Reseating the transceivers
2) Reseating the modules
3) Power cycling the modules
4) Tried OS versions  12.2(33).SXJ2, 12.2(33).SXJ6, 15.1(2)SY1
5) Replaced patch cables
6) The X2-10GBase-ER transceiver connects to an extension from a meet-me room. I swapped pairs to try to see if that resolved anything

Nothing worked.

The two switches haven't been reset in over a year, and probably haven't been power-cycled in 3-4 years. It's possible that they just failed due to that, being moved, etc... But, once is happenstance, twice is circumstance, three times is enemy action. I don't like the coincidence of three failing.

I've got an ER optic on order (I don't have any spares), but I'm going to swap out the LX4 optics today. 

Has anyone run into this ever before with no error message or any indication of the problem? They interfaces show the optics, the ER DOM shows low receive levels (varies between -28 and -37), but just won't come up. 



----
Matthew Huff             | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039