[c-nsp] IMA Interface/PVC flapping issue

[Michael Nicks]

Greetings all,

We have a range of devices on our network that utilize an IMA based 
service from both SBC and ATT. Essentially its NxT1 in an IMA group that 
is a PVC delivered on an OC3 to another endpoint. (Think tail end sites 
aggregated onto an OC3) The application is 2x to 4x T1s on the tail end 
side in an IMA group.

The hardware we are using is generally the 2600 and 2800 platforms for 
the chassis, ATM AIM cards, WIC-1DSU-T1 and VWIC-2MFT-T1 cards.

We've had a general recurring problem where the underlying ATM 
interfaces (T1 interfaces/controllers) keep randomly dropping out of the 
IMA group, causing the PVC to go down. An example is as follows:

Aug 29 00:41:29 cdt: %ATM-5-UPDOWN: Interface ATM0/IMA0.72, Changing 
autovc 3/72 to PVC deactivated.
Aug 29 00:41:29 cdt: %ATM_AIM-5-ACTIVE_LINK_DOWN: Interface ATM0/1 of 
IMA Group ATM0/IMA0 is now inactive.
Aug 29 00:41:31 cdt: %LINK-3-UPDOWN: Interface ATM0/1, changed state to down
Aug 29 00:41:32 cdt: %LINEPROTO-5-UPDOWN: Line protocol on Interface 
ATM0/1, changed state to down
Aug 29 00:41:39 cdt: %ATM-5-UPDOWN: Interface ATM0/IMA0.72, Changing 
autovc 3/72 to PVC activated.
Aug 29 00:41:39 cdt: %ATM_AIM-5-ACTIVE_LINK_UP: Interface ATM0/1 of IMA 
Group ATM0/IMA0 is now active.
Aug 29 00:41:41 cdt: %LINK-3-UPDOWN: Interface ATM0/1, changed state to up
Aug 29 00:41:42 cdt: %LINEPROTO-5-UPDOWN: Line protocol on Interface 
ATM0/1, changed state to up
Aug 29 00:46:34 cdt: %ATM_AIM-5-ACTIVE_LINK_DOWN: Interface ATM0/0 of 
IMA Group ATM0/IMA0 is now inactive.
Aug 29 00:46:34 cdt: %ATM_AIM-5-ACTIVE_LINK_DOWN: Interface ATM0/1 of 
IMA Group ATM0/IMA0 is now inactive.
Aug 29 00:46:34 cdt: %LINK-3-UPDOWN: Interface ATM0/IMA0, changed state 
to down
Aug 29 00:46:35 cdt: %ATM_AIM-5-ACTIVE_LINK_UP: Interface ATM0/0 of IMA 
Group ATM0/IMA0 is now active.
Aug 29 00:46:35 cdt: %ATM_AIM-5-ACTIVE_LINK_UP: Interface ATM0/1 of IMA 
Group ATM0/IMA0 is now active.
Aug 29 00:46:37 cdt: %LINK-3-UPDOWN: Interface ATM0/IMA0, changed state 
to up

This is a Cisco 2651 running IOS 12.3(9). It has an ATM AIM card and a 
VWIC-2MFT-T1 card in it. T1 controllers have "mode atm aim 0" defined in 
controller t1 context, ATM0/0 and ATM0/1 have "ima-group 0" defined in 
interface context. PVC defined under ATM0/IMA0.72.

This is actually two separate incidents in a ten minute window -- 
however I have a feeling they are related. In the first instance ATM0/1 
goes down, causing the PVC to go down. ATM0/1 comes back up after a few 
seconds, and everything is returned to normal. In the second instance 
both ATM0/0 and ATM0/1 become inactive, and ATM0/IMA0 is changed to 
down. ATM0/0 and ATM0/1 return momentarily, and ATM0/IMA0 comes back up.

We have gone round and round with each of our telco providers regarding 
this, and none have been able to definitively find anything wrong with 
our provisioned services. (We've replaced physical wiring, numerous 
smartjacks, rebuilt PVCs with completely different VP/VCs, etc.)

What gets even better is that the T1 controllers show no line code 
violations, slip seconds, or errored seconds. Same for the actual ATM 
interface in regards to input/output/crc errors.

I did actually get a breath of fresh air yesterday when an ATT fellow 
informed me that they have had numerous run-ins with this same 
scenario/situation in the past regarding IMA flapping/dropping randomly. 
He said they (ATT with the assistance of Cisco) came to the conclusion 
that there are problems with the WIC-1DSU-T1 and WIC-2MFT-T1 cards when 
being used in an IMA scenario. This problem is further amplified by the 
fact that certain versions of IOS make this problem worse.

The only thing on CCO I was able to locate regarding IMA flapping 
randomly was related to how the T1s are clocking themselves. (We have 
ours set for the clocking on the first T1 to derive from the line, then 
clock every other T1 from the first T1.)

I am curious as to if any of you have had a similar problem, and if so, 
did you find a resolution?

Thanks for your time.
-Michael

-- 
Michael Nicks
Network Engineer
KanREN
e: mtnicks at kanren.net
o: +1-785-856-9800 x221
m: +1-913-378-6516