[nsp] P-bit and C-bit DS3 errors

[Mark Kent]

I've got this scenario:

                 ds3 x-conn        40 miles        OCx
 7206/PA-MC-2T3+  --------> Cerent -------> Cerent ---> cisco 12016/GRP
 12.0(25)S1                                             12.0(25)S2

I run the 7206, an NSP runs the rest of the stuff.
After an "upgrade" on the GSR, we've been experiencing hits on the
DS3, like this:

Nov 11 10:17:45: %CONTROLLER-5-UPDOWN: Controller T3 4/0, changed state to down
Nov 11 10:17:46: %CONTROLLER-5-UPDOWN: Controller T3 4/0, changed state to up
Nov 11 10:17:57: %CONTROLLER-5-UPDOWN: Controller T3 4/0, changed state to down
Nov 11 10:17:58: %CONTROLLER-5-UPDOWN: Controller T3 4/0, changed state to up
Nov 11 10:17:59: %CONTROLLER-5-UPDOWN: Controller T3 4/0, changed state to down
Nov 11 10:18:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0, changed state to down
Nov 11 10:18:00: %CONTROLLER-5-UPDOWN: Controller T3 4/0, changed state to up
Nov 11 10:18:01: %CONTROLLER-5-UPDOWN: Controller T3 4/0, changed state to down
Nov 11 10:18:02: %CONTROLLER-5-UPDOWN: Controller T3 4/0, changed state to up
Nov 11 10:18:04: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0, changed state to up

  INTERVAL      LCV   PCV   CCV   PES  PSES  SEFS   UAS   LES   CES  CSES
  10:19-10:19     0     0     0     0     0     0     0     0     0     0
  10:04-10:19     0   165   156     8     7     8     0     0     7     7
  09:49-10:04     0     0     0     0     0     0     0     0     0     0

So, we get P-bit code violations and C-bit code violations.

These happen every few hours, sometimes clustered
(three in twenty minutes is the current record).
The circuit does not pick up errors at other times,
it only gets these short-term hits.

The NSP has not made any progress on figuring out what is happening.
The circuit has been tested, but of course this is an intermittent problem
and telcos don't do well with intermittent problems.

I've swapped out everything on my end, all the way to the
patch panel hand-off from the NSP.   I previously had a PA-T3,
and swapped in the PA-MC-2T3+.

With the PA-T3 the only info I got was the circuit was going down and
the FEBE counter (far end block errors) went up during an event.
The PA-MC-2T3+ gives more info (as above).

What are the usual suspects for error patterns like this
(almost always clean, except for some short-lived error bursts)?

Thanks,
-mark