[rbak-nsp] SE100 - dying ppp and pppoe

Tue Mar 11 14:54:02 EDT 2014

Hi
weird stuff...
Today suddenly - ppp and pppoe started to die...
Out of nowhere - around 14:00  subscribers where not able to connect.

Help , anyone  - any ideas ?

Redback Networks SmartEdge OS Version SEOS-12.1.1.4-Release
Built by sysbuild at eussjlx7032.sj.us.am.ericsson.se Thu Jun 27 03:58:36 
PDT 2013
Copyright (C) 1998-2013, Redback Networks Inc. All rights reserved.
System Bootstrap version is PowerPC,rev2.0.1.4
Installed minikernel version is 2.7
Router Up Time -   1 hour 13 minutes

Restarting pppoe

Mar 11 17:22:53: %PPPOE-7-PADX: Ininitalizind padx
Mar 11 17:22:53: %PPPOE-7-PADX: aaad process endpoint is alive
Mar 11 17:22:53: %PPPOE-7-CFG: Starting cfg thread
Mar 11 17:22:53: %PPPOE-7-RCM: pppoed process is alive (demand_dying 0)
Mar 11 17:22:53: %PPPOE-7-RCM: [pppoe_mgr_pppoe_event_handler]PPPoE 
service stop request
Mar 11 17:22:53: %PPPOE-7-RCM: [pppoe_mgr_start_stop_BE] stop pppoed
Mar 11 17:22:53: %PPPOE-7-CFG: [pppoe_cfg_process_q_elem] PPPoE service 
stop request
Mar 11 17:23:03: %PPPOE-7-RCM: pppoed process is dead (demand_dying 1)

Restarting ppp

[pppoe]RS0t#process restart pppMar 11 17:27:13: %PM-5-GEN: restarting 
<ppp> now
Mar 11 17:27:13: %PPP-7-FUNC: ppp_init called
oMar 11 17:27:14: %PPP-7-FUNC: ppp_cct_tree_init called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_intf_tree_init called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_protection_init called
Mar 11 17:27:14: %PPP-7-PROTECTION: Initializing protection code
Mar 11 17:27:14: %PPP-7-FUNC: ppp_session_init called
Mar 11 17:27:14: %PPP-7-INT: magic_init using random pool method
Mar 11 17:27:14: %PPP-7-FUNC: ppp_create_thread called
Mar 11 17:27:14: %PPP-7-INFO: Starting thread pkt-out
Mar 11 17:27:14: %PPP-7-FUNC: ppp_pkt_out_thread_handler called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_init called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_sync_init called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_ctx_tree_init called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_clnt_tree_init called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_create_thread called
Mar 11 17:27:14: %PPP-7-INFO: Starting thread ppp-ppa-tx_thread
Mar 11 17:27:14: %PPP-7-FUNC: ppp_create_thread called
Mar 11 17:27:14: %PPP-7-INFO: Starting thread worker
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_tx_thread called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_thread_handler called
eMar 11 17:27:14: %PPP-7-FUNC: ppp_create_thread called
Mar 11 17:27:14: %PPP-7-INFO: Starting thread ism
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_thread_handler called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_mbe_birth_death_notify called
Mar 11 17:27:14: %PPP-7-ISM_E: ISM MBE endpoint is ALIVE
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_sched_event called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_signal called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_q_elem called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_process_event called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_e_mbe_ep_alive called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_register_mbe called
Mar 11 17:27:14: %PPP-7-ISM: FSM State was All-EP-down, now MBE-EP-up
Mar 11 17:27:14: %PPP-7-FUNC: ppp_create_thread called
Mar 11 17:27:14: %PPP-7-INFO: Starting thread cfg
Mar 11 17:27:14: %PPP-7-FUNC: ppp_config_thread_handler called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_ppp_event_handler called
Mar 11 17:27:14: %PPP-7-RCM_E: pppd process is DEAD, NOT DEMAND DIE
Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_ppp_event_handler called
Mar 11 17:27:14: %PPP-7-RCM_E: pppd process is ALIVE, NOT DEMAND DIE
Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_for_global called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_send_global_option called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_config_process_q_elem called
Mar 11 17:27:14: %PPP-7-CONFIG: Processing request type 0xe00012, 
opcode 7
Mar 11 17:27:14: %PPP-7-FUNC: ppp_config_process_global_cfg called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_for_all_ccts called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_for_all_mp_options called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_for_all_ctx called
Mar 11 17:27:14: %PPP-7-RCM_E: ppp_mgr_for_all_ctx: down loading all 
context info to ppp
Mar 11 17:27:14: %PPP-7-RCM_E: ppp_mgr_for_all_ctx: sending context 
0x40080004 interval 10 timeout 10 retries 3 data check 0
Mar 11 17:27:14: %PPP-7-FUNC: ppp_config_process_q_elem called
Mar 11 17:27:14: %PPP-7-CONFIG: Processing request type 0xe00014, 
opcode 7
Mar 11 17:27:14: %PPP-7-CONFIG: New record for context 0x40080004
Mar 11 17:27:14: %PPP-7-CONFIG: ppp_config_create_keepalive: added ep 
0x41952024, context 1074266116 operation 1 interval 10000ms timeout 
10000ms retries 3
Mar 11 17:27:14: %PPP-7-FUNC: ppp_config_process_q_elem called
Mar 11 17:27:14: %PPP-7-CONFIG: Processing request type 0x400029, 
opcode 7
Mar 11 17:27:14: %PPP-7-FUNC: ppp_mgr_cfg_eof_callback called
Mar 11 17:27:14: %PPP-7-RCM: Sending EOF to pppd
Mar 11 17:27:14: %PPP-7-FUNC: ppp_config_process_q_elem called
Mar 11 17:27:14: %PPP-7-CONFIG: Processing request type 0x400026, 
opcode 8
Mar 11 17:27:14: %PPP-7-CONFIG: Received MO_REQID_COMMAND 0x400026
Mar 11 17:27:14: %PPP-7-CONFIG: Received EOF from RCM
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_birth_death_notify called
Mar 11 17:27:14: %PPP-7-ISM_E: ISM process endpoint is ALIVE
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_sched_event called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_signal called
Mar 11 17:27:14: %PPP-7-ISM: ICR Lib init-ed successfully!
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_q_elem called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_process_event called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_e_client_ep_alive called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_a_all_ep_up called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_register_client called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_register_client2 called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_register_client3 called
Mar 11 17:27:14: %PPP-7-ISM: FSM State was MBE-EP-up, now All-EP-up
Mar 11 17:27:14: %PPP-7-INT: Received ISM message 3405, length 424
Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_signal called
Mar 11 17:27:14: %PPP-7-INT: Received ISM message 3405, length 28
Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_signal called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_q_elem called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_event_ipc called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_event called
Mar 11 17:27:14: %PPP-7-ISM: Processing ISM event: CRD state; CRD 
create
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_recv_card_create called
Mar 11 17:27:14: [1/255:511:63:31/1/0/1]: %PPP-7-ISM: Received card 
create from ISM for slot:1, type:0
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_event called
Mar 11 17:27:14: %PPP-7-ISM: Processing ISM event: CRD state; CRD up
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_event called
Mar 11 17:27:14: %PPP-7-ISM: Processing ISM event: CRD state; CRD 
create
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_recv_card_create called
Mar 11 17:27:14: [2/255:511:63:31/1/0/1]: %PPP-7-ISM: Received card 
create from ISM for slot:2, type:81
Mar 11 17:27:14: %PPP-7-ISM: PPA type: PPA2
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_event called
Mar 11 17:27:14: %PPP-7-ISM: Processing ISM event: CRD state; CRD up
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_event called
Mar 11 17:27:14: %PPP-7-ISM: Processing ISM event: CRD state; CRD attr
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_event called
Mar 11 17:27:14: %PPP-7-ISM: Processing ISM event: CRD state; CRD 
rate-cfg
Mar 11 17:27:14: %PPP-7-ISM: Received EOF from ISM
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_sched_event called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_signal called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_q_elem called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_event_ipc called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_event called
Mar 11 17:27:14: %PPP-7-ISM: Processing ISM event: HDR only; NONE
Mar 11 17:27:14: %PPP-7-ISM: ICR Lib ISM MBE EOF received.
Mar 11 17:27:14: %PPP-7-ISM: Received MBE ALL EOF from ISM
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_sched_event called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_signal called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_q_elem called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_process_event called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_e_eof called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_process_q_elem called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_process_event called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_fsm_e_mbe_all_eof called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_create_thread called
Mar 11 17:27:14: %PPP-7-INFO: Starting thread auth
Mar 11 17:27:14: %PPP-7-FUNC: ppp_auth_aaad_birth_death_notify called
Mar 11 17:27:14: %PPP-7-AUTH: AAAd process endpoint is ALIVE
Mar 11 17:27:14: %PPP-7-FUNC: ppp_auth_l2tp_birth_death_notify called
Mar 11 17:27:14: %PPP-7-AUTH: L2TP process endpoint is ALIVE
Mar 11 17:27:14: %PPP-7-CCT: ppp_cct_walk_all_cct, event: 10
Mar 11 17:27:14: %PPP-7-CCT: ppp_cct_walk_all_cct, event: 11
Mar 11 17:27:14: %PPP-7-FUNC: ppp_auth_send_aaad_eof called
Mar 11 17:27:14: %PPP-7-AUTH: Sending EOF to AAAd
Mar 11 17:27:14: %PPP-7-CCT: ppp_cct_walk_all_cct, event: 8
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ism_send_eof called
Mar 11 17:27:14: %PPP-7-ISM: Sending EOF to ISM
Mar 11 17:27:14: %PPP-7-CCT: ppp_cct_walk_all_cct, event: 14
Mar 11 17:27:14: %PPP-7-FUNC: ppp_create_thread called
Mar 11 17:27:14: %PPP-7-INFO: Starting thread ipc
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ipc_thread_handler called
Mar 11 17:27:14: %PPP-7-IPC: Start IPC thread
Mar 11 17:27:14: %PPP-7-FUNC: ppp_worker_signal called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ipc_process_q_elem called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_handle_msg called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_process_event called
Mar 11 17:27:14: %PPP-7-PPA: Received PPP_PPA_REG from PPP SLOT 02/0
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_client_register called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_insert called
Mar 11 17:27:14: %PPP-7-PPA: Inserted PPA client PPP SLOT 02/0 with id 
of 20
Mar 11 17:27:14: %PPP-7-CCT: ppp_cct_walk_all_cct, event: 13
Mar 11 17:27:14: %PPP-7-INFO: Process pppd ready
Mar 11 17:27:14: %PPP-7-FUNC: ppp_main called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_pkt_init_io called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_pkt_init_ctrl_hdr called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_pkt_init_ctrl_hdr called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_pkt_open_socket called
Mar 11 17:27:14: %PPP-7-PKT: Rcvd socket size set to 225280
Mar 11 17:27:14: %PPP-7-FUNC: ppp_pkt_buf_get called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_pkt_set_control_msg called
Mar 11 17:27:14: %PPP-7-PPA: PPA transmit thread: context 0x40080004 
ppa_op 1, interval 10000ms retries 3 resp time 10000ms
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_ctx_install called
Mar 11 17:27:14: %PPP-7-PPA: Installing ppp context 0x40080004 on all 
PPAs
Mar 11 17:27:14: %PPP-7-PPA: PPA EP PPP SLOT 02/0 is ALIVE
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_proc_restart called
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_ctx_install_entry called
Mar 11 17:27:14: %PPP-7-PPA: Installing context entry: context 
0x40080004 info on ppa PPP SLOT 02/0 interval 10000ms retries 3 timeout 
10000ms flags 0x0
Mar 11 17:27:14: %PPP-7-FUNC: ppp_ppa_ctx_eof called
Mar 11 17:27:14: %PPP-7-PPA: Installing EOF ppa PPP SLOT 02/0

Mar 11 17:27:55: %PPP-7-FUNC: ppp_mgr_ppp_event_handler called
Mar 11 17:27:55: %PPP-7-RCM_E: pppd process is DEAD, NOT DEMAND DIE

-- 
Best Regards
"Kuba" Dawid Chrzan