[cisco-voip] Clicking/popping noise on inbound side of PRI

[Mike Armstrong]

Warning: I've gone through 4 Cisco engineers at TAC, 3 Verizon technicians,
2 VWICs, and 2 SmartJacks, with no resolution to this problem yet, so this
may not be easy.  I'm looking for out-of-the-box thinking, unencumbered by
reason and conventional troubleshooting procedures.  Have a few beers before
working on this.

I've got a 3640/NM-HDV/VWIC-1MFT-T1 connected to an Adtran HTU-R (Verizon
calls it a "SmrtJack") which is the T1 (DS1) interface to an HDSL circuit
between us and the Verizon CO.  There is an intermittent loud
clicking/popping noise -- like a Geiger counter -- on the inbound side only.
It was first reported in early August around the time we upgraded the router
to IOS version 12.2(15)T.   Reverting to the previous version -- 
12.2(13)T -- did not help.

Cisco has replaced the VWIC, even though they didn't think it was the
problem.  Verizon replaced the HTU-R even though they didn't think it was
the problem.  Both Cisco and Verizon claimed it was my unshielded patch
cable between the VWIC and the HTU-R, so I replaced it with a certified,
shielded cable (10 feet long).  None of these replacements had any effect.
In all this time, looking at the diagnostic data recorded by the HTU-R, I
have seen only a few errors of any kind associated with the Telco circuit,
and most of them were apparently caused by us screwing around with the
circuit.  During the same time, I have recorded 6.7M Line Code Violations
and 800K Path Code Violations (hereafter referred to as "errors") on the
VWIC.

We all believe that layer 1 timing is the issue.  I monitored an open line
for a couple of hours and confirmed that the noise was correlated with T1
controller errors.  I've got "sho controller t1 1/0" data going back to
August 8th, with only a few gaps, in a database keyed to the
15-minute-collection-interval time, and from all this data can tell you:

o We see error rates from 0 to 109,333 per 15-minute interval.
o We can go for hours with 0 errors, then jump to tens of thousands in the
next 15-minute interval.
o Once they start, the errors tend to continue for a while, but it's not
guaranteed.
o There are more errors Saturday, Sunday, and Monday than Tue-Fri.
o There is a broad peak from 2PM-10PM.
o There is no evidence of electrical noise (all our equipment is plugged
into UPS, Telco equipment is line powered).
o There is no evidence of EMF noise in the area, measured crudely with a
broadband AM receiver.
o I can't correlate error-free or error-intensive periods with much of
anything.

We had the Telco run loopbacks and BERT tests.  Cisco has done the same.
They did it together for an hour and a half last week.  I've done the same.
None of us has yet been able to produce similar errors in any test
environment.

Relevant router configs are:

controller T1 1/0
 framing esf
 linecode b8zs
 cablelength short 133
 pri-group timeslots 1-20,24
 no yellow generation
 no yellow detection

...

interface Serial1/0:23
 no ip address
 no logging event link-status
 isdn switch-type primary-ni
 isdn incoming-voice voice
 (encapsulation HDLC by default)

Any AHA!s out there?

Mike Armstrong
University of Florida CREC
Lake Alfred, FL
863.956.5891