[j-nsp] Ex stack of 4 switchs stops routing, switching, ...

Mon Jan 6 05:58:58 EST 2014

Humz..not sure if the VC will keep working!

In the past an AFL license was needed, somewhere in the 12.3 branch this was
changed to 'standard licensing' ..

Laurent, can you confirm which version you're running ?

Brgds,

Maarten

Van: Eugeniu Patrascu [mailto:eugen at imacandi.net] 
Verzonden: maandag 6 januari 2014 11:41
Aan: Maarten van der Hoek
CC: Laurent CARON; juniper-nsp at puck.nether.net
Onderwerp: Re: [j-nsp] Ex stack of 4 switchs stops routing, switching, ...

Juniper recommends running 12.3R3.4 on EX4200, EX2200. You seem to be
running very bleeding edge code.

Can you try downgrading and see if the problem goes away ?

On Mon, Jan 6, 2014 at 11:15 AM, Maarten van der Hoek <maarten at vanderhoek.nl
<mailto:maarten at vanderhoek.nl> > wrote:

Hi Laurent,

Had almost exactly the same this morning when I came in the office...
All network traffic was still flowing, however DHCP packet's didn't (for
machine's which were not in the office during the weekend....laptop's /
pda's  / etc....everything 'new' to the switch)

Message log showed:

Jan  6 10:16:37  swex2200vc /kernel: rt_pfe_veto: Memory over consumed. Op
2, rtsm_id 47, msg type 2
Jan  6 10:16:41  swex2200vc /kernel: kmem type session using 58778K,
exceeding limit 40960K
Jan  6 10:16:42  swex2200vc /kernel: rt_pfe_veto: Memory over consumed. Op
1, rtsm_id 47, msg type 2
Jan  6 10:16:57  swex2200vc last message repeated 3 times

Stack consists of 2x EX2200-48T running Junos 13.2X50-D15.3

What ls your Junos ?
Found anything yet ?

Brgds,

Maarten

-----Oorspronkelijk bericht-----
Van: juniper-nsp [mailto:juniper-nsp-bounces at puck.nether.net
<mailto:juniper-nsp-bounces at puck.nether.net> ] Namens Laurent
CARON
Verzonden: zondag 5 januari 2014 17:52
Aan: juniper-nsp at puck.nether.net <mailto:juniper-nsp at puck.nether.net> 
Onderwerp: [j-nsp] Ex stack of 4 switchs stops routing, switching, ...

Hi,

Running a chassis composed of 2 EX4200 and 2 EX4500.

One of the RE did reboot (by itself) on Dec 26th.

I managed to collect some logs:

Dec 25 12:21:41  swa eventd: sendto: Cannot allocate memory Dec 25 12:21:44
swa /kernel: rt_pfe_veto: Memory over consumed. Op 1, rtsm_id 47, msg type 2
Dec 25 12:21:59  swa last message repeated 3 times Dec 25 12:22:49  swa last
message repeated 10 times Dec 25 12:22:54  swa /kernel: rt_pfe_veto: Memory
over consumed. Op 1, rtsm_id 47, msg type 2 Dec 25 12:22:59  swa /kernel:
rt_pfe_veto: Memory over consumed. Op 1, rtsm_id 47, msg type 2 ....
Jan  5 12:50:37  swa /kernel: rt_pfe_veto: Memory over consumed. Op 8,
rtsm_id 0, msg type 10 Jan  5 12:50:38  swa rpd[20440]: RPD_KRT_Q_RETRIES:
Route Update: No buffer space available Jan  5 12:50:42  swa /kernel:
rt_pfe_veto: Memory over consumed. Op 8, rtsm_id ...
Jan  5 15:09:17  swa /kernel: rt_pfe_veto: Memory over consumed. Op 8,
rtsm_id 0, msg type 10 Jan  5 15:09:17  swa /kernel: rt_pfe_veto: Possible
slowest client is pfem2. States processed - 117754359. States to be
processed - 27 Jan  5 15:09:22  swa /kernel: rt_pfe_veto: Memory over
consumed. Op 8, rtsm_id 0, msg type 10 Jan  5 15:09:22  swa /kernel:
rt_pfe_veto: Possible slowest client is pfem2. States processed - 117754359.
States to be processed - 27 Jan  5 15:09:26  swa rpd[20440]:
RPD_KRT_Q_RETRIES: Route Update: No buffer space available

Today the switch would only continue switching for a while but not route
packets anymore.

The arp table was empty.

Restarting routing process only rendered the switch unresponsive so I had to
reboot it via console port.

This switch only handles 3 dozens of LACP aggregates, a few of them are
10Gb, the others Gb, a few SVI, a few pure L2 VLANs, 100 firewall rules, no
dhcp snooping. I use OSPF on ~30 interfaces

The only "fancy" features I use are:
Graceful switchover
RSTP
LLDP
LLDP-Med
NSB
Ethernet storm control

Do any of you have a clue about it ?

Thanks

Laurent

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
<mailto:juniper-nsp at puck.nether.net> 
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
<mailto:juniper-nsp at puck.nether.net> 
https://puck.nether.net/mailman/listinfo/juniper-nsp