[cisco-voip] SCCP Gateway Not Registering Back to CM

Tue Sep 29 18:43:25 EDT 2009

Well, crap, I just grabbed the debugs, sniffer traces and cm traces while the problem was happening... then turned on debugs while bouncing the voice-port and realized two things:  first thing is that I didn't have the sniffer running and secondly AN1D7015355A000 is the other port on the FXS card that isn't even being used.. so the security error is probably due to that port not even being configured in CM.  :P

So this whole time I've been chasing the wrong problem because I wasn't paying enough attention.  Now I still see the security error when the phone is now registered since the error was related to an unconfigured port.  Sigh. I'm no closer to the solution and now don't have the problem reproduced.  Grrrrr.

I really appreciate your insight Wes as always and I'll try to see if I can reproduce the problem and take a fresh look at it.

Thanks again.

-keith

From: Wes Sisk [mailto:wsisk at cisco.com]
Sent: Tuesday, September 29, 2009 10:59 AM
To: Keith Klevenski
Cc: cisco-voip at puck.nether.net Voice
Subject: Re: [cisco-voip] SCCP Gateway Not Registering Back to CM

Security Error eh?

Normally that would be related to a device configured for security attempting to register without security.

Since you are using SCCP controlled FXS ports I would be concerned about:
CSCsv02123    Devices with no security profile cannot register to mixed mode cluster
This defect was found specifically based on VG224 SCCP virtual port usage.

Looking through other previously resolved defects this also looks interesting:
CSCsy20149    VG224: Voice-port goes to transient unregister under SRST mode.
appears resolved in 12.4(24)T01.

Along the diagnostic path:
So either CM got confused about the device configuration or the gateway got confused and sent incorrect registration sequence.  Do you have traces showing the device AN1D7015355A000 both registering properly and failing?  Would be good to see what (if any) changes happen on the CM between the working and failing cases.

Initially, based on your statement "bounced the voice port on the router and every one registered", I suspect the SCCP process on the router got confused and started sending incorrect information.  It would be great to compare successful and failed registrations at any of the available points:
1. debugs on the router
2. packet capture of traffic between CM and the router
3. CM SDI/SDL traces

If you have any of those for both working/failing cases and all times in between it will be significant step toward isolating the issue.

/Wes

On Tuesday, September 29, 2009 10:22:45 AM, Keith Klevenski <KKlevenski at cstcorp.net><mailto:KKlevenski at cstcorp.net> wrote:

Double checking to see if anyone has any clue about this one.

ty.

Ah, the joys of supporting an infrastructure that consists of only SCCP controlled FXS ports over satellite links...

Here is the issue this time.  There was some bad weather at the teleport (where the big dish is) which causes a satellite outage for a few minutes.  Each remote site has a SCCP gateway (2801 running 12.4(24)T) that is registered to CM 7.0 and is also configured with SRST so it will register with itself when (not if) the sat link eventually goes down.  This has worked fine in testing and in the few months this network has been in place.  This time however, when the sat link went down for a few minutes, lots of phones (SCCP controlled FXS ports) did not re-register to the CM when the link came back up.  It seemed to be very random.  At some sites all ports were registered, at some no ports were registered and others there was only 1 or 2 ports not registered.  I went through CM to find the ports that were not registered and simply bounced the voice-port on the router and every one registered almost immediately after doing this.

We have a test site at the office and fortunately it did not register either so I have the problem reproduced.  I see the following on the SCCP gateway doing a 'deb sccp packet' while experiencing the problem:

Sep 28 09:07:56.028 CST: sccpapp_process_socket_events: appl_type 4, soc_fd 1, soc 0, swb_soc 1
Sep 28 09:07:56.028 CST: sccpapp_process_socket_events: SWB_TCP_SOCKET_READ: appl_type 4, swb_eve 4, swb_state 1
Sep 28 09:07:56.028 CST: sccp_send_register_msg: Send register msg for appl_type 4, swb_soc 1,  device_name AN1D7015355A000, ip_addr 10.222.115.6, ipv4_scope 2, ipv6_addr ::, ipv6_scope 0, device_type 30027, protVer --> rsvp_enabled 0
Sep 28 09:07:56.032 CST: protocol_ver 0x11000102 max_streams 1, active_streams 0
Sep 28 09:07:56.032 CST: sccp_send_register_msg: st_ptr EF1A49A0
Sep 28 09:07:56.032 CST: sccp_send_register_msg: Register msg txed in hex(including header) - len 136

Sep 28 09:07:56.032 CST: sccp_print_hex_msg: Len:136 Hex:
80 00 00 00 00 00 00 00 01 00 00 00 41 4E 31 44 37 30 31 35 33 35 35 41 30 30 30 00 00 00 00 00 01 00 00 00 0A DE 73 06 4B 75 00 00 01 00 00 00 00 00 00 00 11 00 01 02 30 30 20 30 30 20 30 30 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 30 30 20 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 28 09:07:56.032 CST: sccp_transmit_msg: sending on socket 1
DGL394GW01#
Sep 28 09:07:56.032 CST: sccp_send_ip_port_msg_v1: send ip port msg with id 2, port 0
Sep 28 09:07:56.032 CST: sccp_transmit_msg: sending on socket 1
Sep 28 09:07:56.628 CST: sccpapp_process_socket_events: appl_type 4, soc_fd 1, soc 0, swb_soc 1
Sep 28 09:07:56.628 CST: sccpapp_process_socket_events: SWB_TCP_SOCKET_READ: appl_type 4, swb_eve 4, swb_state 5
Sep 28 09:07:56.628 CST: sccp_process_swb_control_message: rcvd register rej from SWB CCM, reason: Security Error

The last line stands out to me.  What type of security error would there be?  I see the same in the CM traces (taken last week):

09/25/2009 16:57:28.376 CCM|StationD:    (1605527) RegisterReject text='Security Error'.|<CLID::StandAloneCluster><NID::10.1.101.11><CT::2,100,37,1.51862723><IP::10.222.115.6><DEV::AN1D7015355A000><LVL::State Transition><MASK::0020>

This seems to be key.  I'm assuming the port will register if I bounce the voice-port on the gateway, but I wanted to keep it 'broken' to see what is going on.  The test bed and all other sites are configured with SRST as well, but many did not register even to the SRST router. Obviously this is a problem for the customer if they cannot be sure the ports will register after an outage.  I have all the traces if more will be useful.

Any assistance is appreciated as always.

Thanks!

-Keith Klevenski

________________________________

_______________________________________________

cisco-voip mailing list

cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>

https://puck.nether.net/mailman/listinfo/cisco-voip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20090929/4e30c00c/attachment.html>