[nsp] POS OC12c NSE/PSE errors

[Clinton Work]

I'm having a problem with NSE/PSE errors on a POS OC12c local loop
in Seattle. The circuit is provisioned between two locations in Seattle
using a Telco network with SONET ADMs. The circuit runs clean except for
the NSE/PSE errors. The near end is seeing PSE errors at a rate of 200/minute
and the remote end is seeing 200 NSE errors a minute. 

Completed troubleshooting:
 - Hard loops in front of each router (runs clean)
 - SONET BERT test (runs clean)
 - Replaced router interfaces at both sides
 - Groomed fiber and checked light levels (padded where required)
 - Both ends are using "clock source line"
 - Hard loops at each site in front of the router towards the far end
   (to test the full SONET path)
  - Near end shows no errors at all
  - Far end shows both PSE/NSE errors

We suspect a problem with the Telco circuit, but I have found that
only Cisco routers report NSE/PSE errors. Has anybody found a good way
to troubleshoot this problem and prove its a Telco problem? Its been
suggested that an all-zeros SONET test or watching the stability of the
H1/H2 pointers would highlight the problem.

Cisco troubleshooting documents:
http://www.cisco.com/warp/public/127/POS/posclocking_16181.html
http://www.cisco.com/warp/public/127/nsepse_18932.html

near end config:
interface POS3/0
 ip address x.x.x.x 255.255.255.252
 no ip directed-broadcast
 ip route-cache flow sampled
 load-interval 30
 crc 32
 pos report rdool
 pos report lais
 pos report lrdi
 pos report pais
 pos report prdi
 pos report sd-ber
 pos scramble-atm
 no cdp enable
end

seattle#show controllers pos 3/0
POS3/0
SECTION
  LOF = 0          LOS    = 0                            BIP(B1) = 0
LINE
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B2) = 0
PATH
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B3) = 0
  LOP = 0          NEWPTR = 0          PSE  = 214        NSE     = 0

Active Defects: None
Active Alarms:  None
Alarm reporting enabled for: SF SD SLOS SLOF B1-TCA LAIS LRDI B2-TCA PAIS PLOP PRDI B3-TCA RDOOL

Framing: SONET
APS

  COAPS = 0          PSBF = 0
  State: PSBF_state = False
  ais_shut = FALSE
  Rx(K1/K2): 00/00  S1S0 = 00, C2 = 16
  Remote aps status (none); Reflected local aps status (none)
CLOCK RECOVERY
  RDOOL = 0
  State: RDOOL_state = False
PATH TRACE BUFFER : STABLE
  Remote hostname : somewhere
  Remote interface: POS8/0
  Remote IP addr  : x.x.x.x
  Remote Rx(K1/K2): 00/00  Tx(K1/K2): 00/00

BER thresholds:  SF = 10e-3  SD = 10e-6
TCA thresholds:  B1 = 10e-6  B2 = 10e-6  B3 = 10e-6



-- 
=========================================================================
Clinton Work                                        clinton@scripty.com
Calgary, Alberta