[j-nsp] Ex stack of 4 switchs stops routing, switching, ...

[Eugeniu Patrascu]

Juniper recommends running 12.3R3.4 on EX4200, EX2200. You seem to be
running very bleeding edge code.

Can you try downgrading and see if the problem goes away ?


On Mon, Jan 6, 2014 at 11:15 AM, Maarten van der Hoek <maarten at vanderhoek.nl
> wrote:

> Hi Laurent,
>
> Had almost exactly the same this morning when I came in the office...
> All network traffic was still flowing, however DHCP packet's didn't (for
> machine's which were not in the office during the weekend....laptop's /
> pda's  / etc....everything 'new' to the switch)
>
> Message log showed:
>
> Jan  6 10:16:37  swex2200vc /kernel: rt_pfe_veto: Memory over consumed. Op
> 2, rtsm_id 47, msg type 2
> Jan  6 10:16:41  swex2200vc /kernel: kmem type session using 58778K,
> exceeding limit 40960K
> Jan  6 10:16:42  swex2200vc /kernel: rt_pfe_veto: Memory over consumed. Op
> 1, rtsm_id 47, msg type 2
> Jan  6 10:16:57  swex2200vc last message repeated 3 times
>
> Stack consists of 2x EX2200-48T running Junos 13.2X50-D15.3
>
> What ls your Junos ?
> Found anything yet ?
>
> Brgds,
>
> Maarten
>
>
> -----Oorspronkelijk bericht-----
> Van: juniper-nsp [mailto:juniper-nsp-bounces at puck.nether.net] Namens
> Laurent
> CARON
> Verzonden: zondag 5 januari 2014 17:52
> Aan: juniper-nsp at puck.nether.net
> Onderwerp: [j-nsp] Ex stack of 4 switchs stops routing, switching, ...
>
> Hi,
>
> Running a chassis composed of 2 EX4200 and 2 EX4500.
>
> One of the RE did reboot (by itself) on Dec 26th.
>
> I managed to collect some logs:
>
> Dec 25 12:21:41  swa eventd: sendto: Cannot allocate memory Dec 25 12:21:44
> swa /kernel: rt_pfe_veto: Memory over consumed. Op 1, rtsm_id 47, msg type
> 2
> Dec 25 12:21:59  swa last message repeated 3 times Dec 25 12:22:49  swa
> last
> message repeated 10 times Dec 25 12:22:54  swa /kernel: rt_pfe_veto: Memory
> over consumed. Op 1, rtsm_id 47, msg type 2 Dec 25 12:22:59  swa /kernel:
> rt_pfe_veto: Memory over consumed. Op 1, rtsm_id 47, msg type 2 ....
> Jan  5 12:50:37  swa /kernel: rt_pfe_veto: Memory over consumed. Op 8,
> rtsm_id 0, msg type 10 Jan  5 12:50:38  swa rpd[20440]: RPD_KRT_Q_RETRIES:
> Route Update: No buffer space available Jan  5 12:50:42  swa /kernel:
> rt_pfe_veto: Memory over consumed. Op 8, rtsm_id ...
> Jan  5 15:09:17  swa /kernel: rt_pfe_veto: Memory over consumed. Op 8,
> rtsm_id 0, msg type 10 Jan  5 15:09:17  swa /kernel: rt_pfe_veto: Possible
> slowest client is pfem2. States processed - 117754359. States to be
> processed - 27 Jan  5 15:09:22  swa /kernel: rt_pfe_veto: Memory over
> consumed. Op 8, rtsm_id 0, msg type 10 Jan  5 15:09:22  swa /kernel:
> rt_pfe_veto: Possible slowest client is pfem2. States processed -
> 117754359.
> States to be processed - 27 Jan  5 15:09:26  swa rpd[20440]:
> RPD_KRT_Q_RETRIES: Route Update: No buffer space available
>
> Today the switch would only continue switching for a while but not route
> packets anymore.
>
> The arp table was empty.
>
> Restarting routing process only rendered the switch unresponsive so I had
> to
> reboot it via console port.
>
> This switch only handles 3 dozens of LACP aggregates, a few of them are
> 10Gb, the others Gb, a few SVI, a few pure L2 VLANs, 100 firewall rules, no
> dhcp snooping. I use OSPF on ~30 interfaces
>
> The only "fancy" features I use are:
> Graceful switchover
> RSTP
> LLDP
> LLDP-Med
> NSB
> Ethernet storm control
>
> Do any of you have a clue about it ?
>
> Thanks
>
> Laurent
>
> _______________________________________________
> juniper-nsp mailing list juniper-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>
> _______________________________________________
> juniper-nsp mailing list juniper-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>