[c-nsp] 20 second packet delay

[Raymond Lucas]

Well, this was a new one for me.  One way packet delay of around 20 seconds
on a single link.  I had never thought it was possible, but just when you
think you've seen it all...

I have a customer with a number of sites, each with 2 x 3750 in a stack.
Each stack will typically have four ethernet hand offs to Ericsson
equipment which provides transparent circuits over microwave.  One member
of the stack will have two circuits towards one neighbouring site, the
other member will have two circuits towards a different neighbouring site.
The same microwave equipment provides all four circuits (potentially spread
across different line cards though) although with different dishes to
provide the appropriate directionality for the microwave.  Each circuit has
a single P2P VLAN configured on it.  This generally works fine.

Yesterday we had a very interesting scenario where at approximately 1200
the 3750s saw interfaces flapping on both circuits between two particular
sites.  From 1220, one of those circuits appeared to introduce delay in one
direction of 18-23 seconds.  This could be demonstrated with debugs of both
CDP and ICMP traffic.  The relevant interface at both ends were both
manually shut/no shut and also err-disabled/no shut due to UDLD detecting a
problem.

I'm pretty sure all interface up/downs were seen by the 3750s at both ends.
The microwave equipment is meant to relay up/down events that occur any
where in the path to all ethernet ports so that is expected regardless of
where the event actually occurred.  It is currently not clear if the
microwave equipment logged all those events or just the ones caused due to
UDLD or manual action.

Around 1500 we again saw interfaces flapping on both circuits between the
same two sites.  Around 1600 I checked the fault again and it had cleared.
My assumption is that it cleared after the flapping.

Ignoring the specifics of the up/down events and even if it was the Cisco
or Ericsson kit that was at fault, has anyone ever seen packets held up for
20 seconds across a link?

Cheers,
Ray