[rbak-nsp] SE100 - dying ppp and pppoe

Tue Mar 11 16:34:42 EDT 2014

put here:
show memory

show process ppp detail
show process pppoe detail

Marcin


On 2014-03-11 19:54, "Kuba" Dawid Chrzan wrote:
> Hi
> weird stuff...
> Today suddenly - ppp and pppoe started to die...
> Out of nowhere - around 14:00  subscribers where not able to connect.
>
> Help , anyone  - any ideas ?
>
> Redback Networks SmartEdge OS Version SEOS-12.1.1.4-Release
> Built by sysbuild at eussjlx7032.sj.us.am.ericsson.se Thu Jun 27 03:58:36 
> PDT 2013
> Copyright (C) 1998-2013, Redback Networks Inc. All rights reserved.
> System Bootstrap version is PowerPC,rev2.0.1.4
> Installed minikernel version is 2.7
> Router Up Time -   1 hour 13 minutes
>
>
> Restarting pppoe
>
> Mar 11 17:22:53: %PPPOE-7-PADX: Ininitalizind padx
> Mar 11 17:22:53: %PPPOE-7-PADX: aaad process endpoint is alive
> Mar 11 17:22:53: %PPPOE-7-CFG: Starting cfg thread
> Mar 11 17:22:53: %PPPOE-7-RCM: pppoed process is alive (demand_dying 0)
> Mar 11 17:22:53: %PPPOE-7-RCM: [pppoe_mgr_pppoe_event_handler]PPPoE 
> service stop request
> Mar 11 17:22:53: %PPPOE-7-RCM: [pppoe_mgr_start_stop_BE] stop pppoed
> Mar 11 17:22:53: %PPPOE-7-CFG: [pppoe_cfg_process_q_elem] PPPoE 
> service stop request
> Mar 11 17:23:03: %PPPOE-7-RCM: pppoed process is dead (demand_dying 1)
>
>
> Restarting ppp
>
> [pppoe]RS0t#process restart pppMar 11 17:27:13: %PM-5-GEN: restarting 
> <ppp> now
> Mar 11 17:27:13: %PPP-7-FUNC: ppp_init called
> oMar 11 17:27:14: %PPP-7-FUNC: ppp_cct_tree_init called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_intf_tree_init called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_protection_init called
> Mar 11 17:27:14: %PPP-7-PROTECTION: Initializing protection code
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_session_init called
> Mar 11 17:27:14: %PPP-7-INT: magic_init using random pool method
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_create_thread called
> Mar 11 17:27:14: %PPP-7-INFO: Starting thread pkt-out
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_pkt_out_thread_handler called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_init called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_sync_init called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_ctx_tree_init called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_clnt_tree_init called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_create_thread called
> Mar 11 17:27:14: %PPP-7-INFO: Starting thread ppp-ppa-tx_thread
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_create_thread called
> Mar 11 17:27:14: %PPP-7-INFO: Starting thread worker
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_tx_thread called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_thread_handler called
> eMar 11 17:27:14: %PPP-7-FUNC: ppp_create_thread called
> Mar 11 17:27:14: %PPP-7-INFO: Starting thread ism
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_thread_handler called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_mbe_birth_death_notify called
> Mar 11 17:27:14: %PPP-7-ISM_E: ISM MBE endpoint is ALIVE
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_sched_event called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_signal called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_q_elem called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_process_event called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_e_mbe_ep_alive called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_register_mbe called
> Mar 11 17:27:14: %PPP-7-ISM: FSM State was All-EP-down, now MBE-EP-up
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_create_thread called
> Mar 11 17:27:14: %PPP-7-INFO: Starting thread cfg
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_config_thread_handler called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_ppp_event_handler called
> Mar 11 17:27:14: %PPP-7-RCM_E: pppd process is DEAD, NOT DEMAND DIE
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_ppp_event_handler called
> Mar 11 17:27:14: %PPP-7-RCM_E: pppd process is ALIVE, NOT DEMAND DIE
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_for_global called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_send_global_option called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_config_process_q_elem called
> Mar 11 17:27:14: %PPP-7-CONFIG: Processing request type 0xe00012, 
> opcode 7
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_config_process_global_cfg called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_for_all_ccts called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_for_all_mp_options called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_for_all_ctx called
> Mar 11 17:27:14: %PPP-7-RCM_E: ppp_mgr_for_all_ctx: down loading all 
> context info to ppp
> Mar 11 17:27:14: %PPP-7-RCM_E: ppp_mgr_for_all_ctx: sending context 
> 0x40080004 interval 10 timeout 10 retries 3 data check 0
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_config_process_q_elem called
> Mar 11 17:27:14: %PPP-7-CONFIG: Processing request type 0xe00014, 
> opcode 7
> Mar 11 17:27:14: %PPP-7-CONFIG: New record for context 0x40080004
> Mar 11 17:27:14: %PPP-7-CONFIG: ppp_config_create_keepalive: added ep 
> 0x41952024, context 1074266116 operation 1 interval 10000ms timeout 
> 10000ms retries 3
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_config_process_q_elem called
> Mar 11 17:27:14: %PPP-7-CONFIG: Processing request type 0x400029, 
> opcode 7
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_cfg_eof_callback called
> Mar 11 17:27:14: %PPP-7-RCM: Sending EOF to pppd
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_config_process_q_elem called
> Mar 11 17:27:14: %PPP-7-CONFIG: Processing request type 0x400026, 
> opcode 8
> Mar 11 17:27:14: %PPP-7-CONFIG: Received MO_REQID_COMMAND 0x400026
> Mar 11 17:27:14: %PPP-7-CONFIG: Received EOF from RCM
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_birth_death_notify called
> Mar 11 17:27:14: %PPP-7-ISM_E: ISM process endpoint is ALIVE
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_sched_event called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_signal called
> Mar 11 17:27:14: %PPP-7-ISM: ICR Lib init-ed successfully!
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_q_elem called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_process_event called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_e_client_ep_alive called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_a_all_ep_up called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_register_client called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_register_client2 called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_register_client3 called
> Mar 11 17:27:14: %PPP-7-ISM: FSM State was MBE-EP-up, now All-EP-up
> Mar 11 17:27:14: %PPP-7-INT: Received ISM message 3405, length 424
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_signal called
> Mar 11 17:27:14: %PPP-7-INT: Received ISM message 3405, length 28
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_signal called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_q_elem called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_event_ipc called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_event called
> Mar 11 17:27:14: %PPP-7-ISM: Processing ISM event: CRD state; CRD create
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_recv_card_create called
> Mar 11 17:27:14: [1/255:511:63:31/1/0/1]: %PPP-7-ISM: Received card 
> create from ISM for slot:1, type:0
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_event called
> Mar 11 17:27:14: %PPP-7-ISM: Processing ISM event: CRD state; CRD up
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_event called
> Mar 11 17:27:14: %PPP-7-ISM: Processing ISM event: CRD state; CRD create
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_recv_card_create called
> Mar 11 17:27:14: [2/255:511:63:31/1/0/1]: %PPP-7-ISM: Received card 
> create from ISM for slot:2, type:81
> Mar 11 17:27:14: %PPP-7-ISM: PPA type: PPA2
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_event called
> Mar 11 17:27:14: %PPP-7-ISM: Processing ISM event: CRD state; CRD up
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_event called
> Mar 11 17:27:14: %PPP-7-ISM: Processing ISM event: CRD state; CRD attr
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_event called
> Mar 11 17:27:14: %PPP-7-ISM: Processing ISM event: CRD state; CRD 
> rate-cfg
> Mar 11 17:27:14: %PPP-7-ISM: Received EOF from ISM
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_sched_event called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_signal called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_q_elem called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_event_ipc called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_event called
> Mar 11 17:27:14: %PPP-7-ISM: Processing ISM event: HDR only; NONE
> Mar 11 17:27:14: %PPP-7-ISM: ICR Lib ISM MBE EOF received.
> Mar 11 17:27:14: %PPP-7-ISM: Received MBE ALL EOF from ISM
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_sched_event called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_signal called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_q_elem called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_process_event called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_e_eof called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_q_elem called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_process_event called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_e_mbe_all_eof called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_create_thread called
> Mar 11 17:27:14: %PPP-7-INFO: Starting thread auth
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_auth_aaad_birth_death_notify called
> Mar 11 17:27:14: %PPP-7-AUTH: AAAd process endpoint is ALIVE
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_auth_l2tp_birth_death_notify called
> Mar 11 17:27:14: %PPP-7-AUTH: L2TP process endpoint is ALIVE
> Mar 11 17:27:14: %PPP-7-CCT: ppp_cct_walk_all_cct, event: 10
> Mar 11 17:27:14: %PPP-7-CCT: ppp_cct_walk_all_cct, event: 11
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_auth_send_aaad_eof called
> Mar 11 17:27:14: %PPP-7-AUTH: Sending EOF to AAAd
> Mar 11 17:27:14: %PPP-7-CCT: ppp_cct_walk_all_cct, event: 8
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_send_eof called
> Mar 11 17:27:14: %PPP-7-ISM: Sending EOF to ISM
> Mar 11 17:27:14: %PPP-7-CCT: ppp_cct_walk_all_cct, event: 14
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_create_thread called
> Mar 11 17:27:14: %PPP-7-INFO: Starting thread ipc
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ipc_thread_handler called
> Mar 11 17:27:14: %PPP-7-IPC: Start IPC thread
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_signal called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ipc_process_q_elem called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_handle_msg called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_process_event called
> Mar 11 17:27:14: %PPP-7-PPA: Received PPP_PPA_REG from PPP SLOT 02/0
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_client_register called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_insert called
> Mar 11 17:27:14: %PPP-7-PPA: Inserted PPA client PPP SLOT 02/0 with id 
> of 20
> Mar 11 17:27:14: %PPP-7-CCT: ppp_cct_walk_all_cct, event: 13
> Mar 11 17:27:14: %PPP-7-INFO: Process pppd ready
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_main called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_pkt_init_io called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_pkt_init_ctrl_hdr called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_pkt_init_ctrl_hdr called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_pkt_open_socket called
> Mar 11 17:27:14: %PPP-7-PKT: Rcvd socket size set to 225280
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_pkt_buf_get called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_pkt_set_control_msg called
> Mar 11 17:27:14: %PPP-7-PPA: PPA transmit thread: context 0x40080004 
> ppa_op 1, interval 10000ms retries 3 resp time 10000ms
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_ctx_install called
> Mar 11 17:27:14: %PPP-7-PPA: Installing ppp context 0x40080004 on all 
> PPAs
> Mar 11 17:27:14: %PPP-7-PPA: PPA EP PPP SLOT 02/0 is ALIVE
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_proc_restart called
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_ctx_install_entry called
> Mar 11 17:27:14: %PPP-7-PPA: Installing context entry: context 
> 0x40080004 info on ppa PPP SLOT 02/0 interval 10000ms retries 3 
> timeout 10000ms flags 0x0
> Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_ctx_eof called
> Mar 11 17:27:14: %PPP-7-PPA: Installing EOF ppa PPP SLOT 02/0
>
>
> Mar 11 17:27:55: %PPP-7-FUNC: ppp_mgr_ppp_event_handler called
> Mar 11 17:27:55: %PPP-7-RCM_E: pppd process is DEAD, NOT DEMAND DIE
>