[c-nsp] Possible causes for fiber link flap?

[Peter Rathlev]

We're scratching our heads regarding some strange link-flapping. We have
a ~3 km run (no underground vaults before a recent accident) over which
we run 10G-LR. It has worked without problems for a couple of months. A
few weeks ago someone pushed a sharpened wooden pole down through the
plastic conduit around halfway between the end points and damaged the
fiber.

All fibers were spliced in a new vault and OTDR showed nothing wrong
afterwards. We can see the loss/reflection from the splice but it's well
within tolerances. The total link loss (according to DOM) is 0.6 dB in
one direction and 2.4 dB in the other direction. Budget should be at
least around 8 dB.

Since the fiber was repaired we're seeing quite frequent (dozens a day)
link flaps. Only one end actually registers link down, the other end
just sees the ISIS adjacency flapping. (See logs further down.) The end
that registers link down also sees a few input errors (FCS+symbol)

We tried changing the optics (third party, but we have had no problems
with them previously) and patch cords and tried using another fiber pair
in the run (in a different tube in this 96 strand run). Everything has
been clean many times over. Nothing has helped. The only thing we
haven't tried is using another physical container port since we don't
have any available at the moment. :-(

One end is Sup720 + 6708 running 12.2(33)SXJ3 AIS, the other is Sup2T +
6904 running 15.1(1)SY1. The latter, ROUTER-B in the logs, is the one
that sees link down.

We have two 8G FC links running without problems (on MDS equipment) on
other fibers in the same tube. We're contemplating swapping with one of
these to see if the problem follows the fiber or the equipment.

What could the cause be and how can we investigate? Regular OTDR shows
no problems, but can it see polarisation and/or chromatic dispersion?
And are those relevant on such a short run (3km)?

Logs follow; ISIS runs on Vlan2294 and the physical interfaces are
trunks to carry some inter-DC VLANs.

Oct  9 10:30:29.220 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet9/8, changed state to down
Oct  9 10:30:29.228 CEST: %LINK-3-UPDOWN: Interface Vlan2294, changed state to down
Oct  9 10:30:29.228 CEST: %PIM-5-NBRCHG: neighbor 192.0.2.253 DOWN on interface Vlan2294 non DR
Oct  9 10:30:29.228 CEST: %CLNS-5-ADJCHANGE: ISIS: Adjacency to ROUTER-A (Vlan2294) Down, interface deleted(non-iih)
Oct  9 10:30:29.228 CEST: %CLNS-5-ADJCHANGE: ISIS: Adjacency to ROUTER-A (Vlan2294) Down, interface deleted(non-iih)
Oct  9 10:30:29.276 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan2294, changed state to down
Oct  9 10:30:29.276 CEST: %LINK-3-UPDOWN: Interface TenGigabitEthernet9/8, changed state to down
Oct  9 10:30:29.280 CEST: %LDP-5-SP: 198.51.100.4:0: session hold up initiated
Oct  9 10:30:31.056 CEST: %LINK-3-UPDOWN: Interface TenGigabitEthernet9/8, changed state to up
Oct  9 10:30:31.060 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet9/8, changed state to up
Oct  9 10:30:32.068 CEST: %LINK-3-UPDOWN: Interface Vlan2294, changed state to up
Oct  9 10:30:32.068 CEST: %PIM-5-NBRCHG: neighbor 192.0.2.253 UP on interface Vlan2294 
Oct  9 10:30:32.072 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan2294, changed state to up
Oct  9 10:30:33.528 CEST: %PIM-5-DRCHG: DR change from neighbor 0.0.0.0 to 192.0.2.254 on interface Vlan2294
Oct  9 10:30:34.560 CEST: %LDP-5-SP: 198.51.100.4:0: session recovery succeeded
Oct  9 10:30:34.576 CEST: %CLNS-5-ADJCHANGE: ISIS: Adjacency to ROUTER-A (Vlan2294) Up, new adjacency

Oct  9 10:30:30.105 CEST: %CLNS-5-ADJCHANGE: ISIS: Adjacency to ROUTER-B (Vlan2294) Down, hold time expired
Oct  9 10:30:34.545 CEST: %CLNS-5-ADJCHANGE: ISIS: Adjacency to ROUTER-B (Vlan2294) Up, new adjacency

-- 
Peter