[j-nsp] ACX5048 issue causing vpls side affect it seems

Tue May 2 12:23:32 EDT 2017

HI all, I'm having issues with vpls's going down and coming back up and not
fully restoring to Cisco ASR9000's.  Once I bounce a portion of the Juniper
ACX5048 routing-instance for the vpls, then it restores and functions.  (I
can also clear ldp neighbor x.x.x.x on the asr9000 side and that seems to
revive the juniper acx5048 vpls functions also)

Currently when we make a change to one of our ACX5048's, vpls's will do this
aforementioned bounce. down and back up, and then I will have to delete and
set vpls routing instance.  Not one certain thing seems to be the trigger,
in other words, last week someone added a subinterface (unit) and l2circuit,
and then this bad vpls thing happened.  Then today someone added a
subinterface, a vlan, an irb, and added the irb to a vrf routing-instance,
and this same strange vpls problem occurred again.  It is reproducible, as
the other day I created a dummy subint and l2circuit and BAM, vpls issue
happened.

I do see some logs (shown below) and JTAC is working on it with me, though
we haven't made much progress yet.and so I wanted to query you all and see
what you might know.

Please know this, my early batch purchase of several acx5048's have this rev
7 fpc, but my newer batch have this rev 5 fpc.  I wonder if there is a
problem with the newer batch showing rev 5. ?  The vpls problem is happening
on the rev 5's but I don't recall seeing this issue on the rev 7.  However I
don't want to jump to conclusions just yet until jtac or someone else tells
me what is causing this.

(serial numbers intentionally changed)

.interestingly, the batch I bought first has the higher rev number (7)..

{master:0}

agould at 5048-10> show chassis hardware | grep "FPC 0|Part"

Item             Version  Part number  Serial number     Description

FPC 0            REV 07   650-060545   WB3712345679      ACX5048

.and the batch I bought a few months later, have the earlier rev number (5).

{master:0}

agould at 5048-20> show chassis hardware | grep "FPC 0|Part"

Item             Version  Part number  Serial number     Description

FPC 0            REV 05   650-066415   WB3712345678      ACX5048

.

May  2 07:48:57  sabn-h-5048 rpd[1474]: Deleting vpls intf
vpls-12-10.101.36.12-10.101.20.4-10.101.20.4, real name lsi.1049041 for
L2VPN FEC129 instance

May  2 07:48:57  sabn-h-5048 rpd[1474]: Deleting vpls intf
vpls-12-10.101.36.12-10.101.24.4-10.101.24.4, real name lsi.1049040 for
L2VPN FEC129 instance

May  2 07:48:57  sabn-h-5048 rpd[1474]: Deleting vpls intf
vpls-12-10.101.36.12-10.101.24.5-10.101.24.5, real name lsi.1049033 for
L2VPN FEC129 instance

May  2 07:48:57  sabn-h-5048 rpd[1474]: Deleting vpls intf
vpls-12-10.101.36.12-10.101.28.4-10.101.28.4, real name lsi.1049025 for
L2VPN FEC129 instance

May  2 07:48:57  sabn-h-5048 rpd[1474]: Deleting vpls intf
vpls-12-10.101.36.12-10.101.32.9-10.101.32.9, real name lsi.1049023 for
L2VPN FEC129 instance

May  2 07:48:57  sabn-h-5048 rpd[1474]: Deleting vpls intf
vpls-12-10.101.36.12-10.101.36.10-10.101.36.10, real name lsi.1049024 for
L2VPN FEC129 instance

May  2 07:48:57  sabn-h-5048 rpd[1474]: Deleting vpls intf
vpls-11-10.101.36.12-10.101.0.1-10.101.0.1, real name lsi.1049010 for L2VPN
FEC129 instance

May  2 07:48:57  sabn-h-5048 rpd[1474]: Deleting vpls intf
vpls-11-10.101.36.12-10.101.36.10-10.101.36.10, real name lsi.1049011 for
L2VPN FEC129 instance

May  2 07:49:01  sabn-h-5048 rpd[1474]: RPD_KRT_Q_RETRIES: IFL IFF Update:
Device busy

May  2 07:49:01  sabn-h-5048 last message repeated 2 times

May  2 07:49:01  sabn-h-5048 rpd[1474]: RPD_KRT_Q_RETRIES: IFL IFF Update:
No buffer space available

May  2 07:49:01  sabn-h-5048 last message repeated 21 times

May  2 07:49:01  sabn-h-5048 rpd[1474]: Error creating dynamic logical
interface from sub-unit 1049047: Device busy

May  2 07:49:01  sabn-h-5048 rpd[1474]: RPD_KRT_Q_RETRIES: IFL IFF Update:
No buffer space available

May  2 07:49:01  sabn-h-5048 rpd[1474]: Error creating dynamic logical
interface from sub-unit 1049048: Device busy

May  2 07:49:01  sabn-h-5048 rpd[1474]: RPD_KRT_Q_RETRIES: IFL IFF Update:
No buffer space available

May  2 07:49:01  sabn-h-5048 rpd[1474]: Error creating dynamic logical
interface from sub-unit 1049049: Device busy

May  2 07:49:01  sabn-h-5048 rpd[1474]: RPD_KRT_Q_RETRIES: IFL IFF Update:
No buffer space available

May  2 07:49:01  sabn-h-5048 rpd[1474]: Error creating dynamic logical
interface from sub-unit 1049047: Device busy

May  2 07:49:01  sabn-h-5048 rpd[1474]: RPD_KRT_Q_RETRIES: IFL IFF Update:
No buffer space available

May  2 07:49:01  sabn-h-5048 last message repeated 2 times

May  2 07:49:01  sabn-h-5048 rpd[1474]: RPD_KRT_Q_RETRIES: IFL IFF Update:
Device busy

May  2 07:49:01  sabn-h-5048 rpd[1474]: RPD_KRT_Q_RETRIES: IFL IFF Update:
No buffer space available

May  2 07:49:01  sabn-h-5048 last message repeated 60 times

May  2 07:49:01  sabn-h-5048 rpd[1474]: RPD_KRT_Q_RETRIES: IFL IFF Update:
Device busy

May  2 07:49:01  sabn-h-5048 rpd[1474]: Error creating dynamic logical
interface from sub-unit 1049047: Device busy

May  2 07:49:01  sabn-h-5048 rpd[1474]: RPD_KRT_Q_RETRIES: IFL IFF Update:
Device busy

.

-Aaron