[c-nsp] 7200 Line protocol bouncing

Thu Aug 30 13:49:10 EDT 2012

I am seeing a weird problem on one of my aggregation routers.  This is a
7206VXR, with a PA-MC-T3 and a PA-MC-4T1 cards.  I terminate about 10
T1s on the MC-T3 card (all local), and 2 long-distance T1s on the
PA-MC-4T1 card.  All services come off the same AT&T multiplexer that's
hanging on the wall 20 feet away.  On the far end of each of the T1s is
a Cisco router (either a 1750 or 1841) with a WIC-1DSU-T1.  This setup
worked fine for several years.

Few months ago we started seeing line protocol bouncing on the 2 LD
circuits, and less frequently, on some of the T3 channels.  Most of the
time it would go down for ~10seconds and come back up.  Long enough for
users to complain.  Sometimes it would stay down for up to a minute, and
sometimes (rarely) would be accompanied by a carrier transition.  The
far sides are not recording anything that would correlate.  The drops
would not be systematic, and we could not correlate them to any other
activities or events.  Sometimes they would accrue 2 drops in a 24-hr
period, and sometimes 12 or more.  Over time, drops started happening
more frequently.  

Telco, of course, cannot find any problems, even after I made them run
an extended (1hr+) test on each circuit.  I replaced the PA-MC-4T1 card,
the patch cables, and the remote routers.  No change.

Last week I swapped out the NPE-300 for an NPE-400 (accompanied by IOS
switch from 12.4.4.T7 to 12.4.25d).  Yes, I know it's not the latest,
but I am running it successfully on several other similar routers.  The
NPE-400 was pulled from a functional router (it was, in turn, upgraded
to a NPE-G1).
Now the frequency of these drops skyrocketed.  Below is the log just
from today.

My main question is: can these issues be caused by a router component
beyond the PA card (i.e. engine or backplane)?  As far as I understand
the 7200 architecture, L1 and L2 would be terminated on the PA, and
since I see same problems on multiple PAs, it shouldn't be the router.
In which case the common point would be the AT&T multiplexer, or
something beyond it in the telco network.  Am I thinking correctly about
this?

Any thoughts or suggestions would be appreciated.

Aug 30 2012 03:13:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to down
Aug 30 2012 03:13:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to up
Aug 30 2012 03:14:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/0:0, changed state to down
Aug 30 2012 03:14:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/0:0, changed state to up
Aug 30 2012 03:15:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/10:0, changed state to down
Aug 30 2012 03:15:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/10:0, changed state to up
Aug 30 2012 03:16:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to down
Aug 30 2012 03:16:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to up
Aug 30 2012 04:36:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to down
Aug 30 2012 04:36:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to up
Aug 30 2012 07:19:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/10:0, changed state to down
Aug 30 2012 07:19:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/10:0, changed state to up
Aug 30 2012 07:46:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to down
Aug 30 2012 07:46:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to up
Aug 30 2012 07:50:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to down
Aug 30 2012 07:50:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to down
Aug 30 2012 07:50:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to up
Aug 30 2012 07:50:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to up
Aug 30 2012 07:57:39 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/0:0, changed state to down
Aug 30 2012 07:57:39 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to down
Aug 30 2012 07:57:49 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/0:0, changed state to up
Aug 30 2012 07:57:49 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to up
Aug 30 2012 08:11:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to down
Aug 30 2012 08:11:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to up
Aug 30 2012 08:13:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to down
Aug 30 2012 08:14:08 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to up
Aug 30 2012 08:37:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to down
Aug 30 2012 08:37:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to up
Aug 30 2012 08:52:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to down
Aug 30 2012 08:53:08 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to up
Aug 30 2012 09:24:09 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/0:0, changed state to down
Aug 30 2012 09:24:19 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/0:0, changed state to up
Aug 30 2012 09:25:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to down
Aug 30 2012 09:26:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to up
Aug 30 2012 09:26:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to down
Aug 30 2012 09:26:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to up
Aug 30 2012 09:37:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to down
Aug 30 2012 09:37:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to up
Aug 30 2012 09:37:39 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to down
Aug 30 2012 09:37:49 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to up
Aug 30 2012 09:38:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/10:0, changed state to down
Aug 30 2012 09:38:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to down
Aug 30 2012 09:38:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to up
Aug 30 2012 09:38:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/10:0, changed state to up
Aug 30 2012 09:41:08 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/10:0, changed state to down
Aug 30 2012 09:41:19 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to down
Aug 30 2012 09:41:49 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to up
Aug 30 2012 09:41:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/10:0, changed state to up
Aug 30 2012 09:43:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to down
Aug 30 2012 09:43:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to up
Aug 30 2012 09:45:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to down
Aug 30 2012 09:46:08 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to down
Aug 30 2012 09:46:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to up
Aug 30 2012 09:46:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to up
Aug 30 2012 09:47:09 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/0:0, changed state to down
Aug 30 2012 09:47:19 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/0:0, changed state to up
Aug 30 2012 10:11:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to down
Aug 30 2012 10:11:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to up
Aug 30 2012 10:37:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to down
Aug 30 2012 10:37:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to up
Aug 30 2012 10:48:47 CST: %CONTROLLER-5-UPDOWN: Controller T1 4/0,
changed state to down
Aug 30 2012 10:48:49 CST: %LINK-3-UPDOWN: Interface Serial4/0:0, changed
state to down
Aug 30 2012 10:48:50 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/0:0, changed state to down
Aug 30 2012 10:49:17 CST: %CONTROLLER-5-UPDOWN: Controller T1 4/0,
changed state to up
Aug 30 2012 10:49:19 CST: %LINK-3-UPDOWN: Interface Serial4/0:0, changed
state to up
Aug 30 2012 10:49:20 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/0:0, changed state to up
Aug 30 2012 11:36:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to down
Aug 30 2012 11:36:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to up
Aug 30 2012 11:37:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to down
Aug 30 2012 11:37:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to up
Aug 30 2012 11:48:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to down
Aug 30 2012 11:48:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/7:0, changed state to up
Aug 30 2012 11:50:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to down
Aug 30 2012 11:51:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to up
Aug 30 2012 11:58:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to down
Aug 30 2012 11:58:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to up
Aug 30 2012 11:59:08 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to down
Aug 30 2012 11:59:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to up
Aug 30 2012 12:00:08 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to down
Aug 30 2012 12:00:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to up
Aug 30 2012 12:01:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to down
Aug 30 2012 12:01:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to up
Aug 30 2012 12:01:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to down
Aug 30 2012 12:01:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to up
Aug 30 2012 12:02:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to down
Aug 30 2012 12:02:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to up
Aug 30 2012 12:09:08 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to down
Aug 30 2012 12:09:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to up
Aug 30 2012 12:09:39 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/0:0, changed state to down
Aug 30 2012 12:09:49 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/0:0, changed state to up
Aug 30 2012 12:09:49 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to down
Aug 30 2012 12:09:59 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to up
Aug 30 2012 12:10:19 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/0:0, changed state to down
Aug 30 2012 12:10:29 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/0:0, changed state to up
Aug 30 2012 12:20:59 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to down
Aug 30 2012 12:21:09 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to up
Aug 30 2012 12:22:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to down
Aug 30 2012 12:22:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/4:0, changed state to up
Aug 30 2012 12:36:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to down
Aug 30 2012 12:36:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to up
Aug 30 2012 12:37:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to down
Aug 30 2012 12:37:19 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to down
Aug 30 2012 12:37:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial1/0/1:0, changed state to up
Aug 30 2012 12:37:29 CST: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial4/1:0, changed state to up

Sincerely,
Michael Malitsky