[cisco-voip] Intermittent phone resets

Thu Jun 5 14:41:17 EDT 2014

I had a case where the guy had his CUCM server switch ports SPANed to some
dedicated capture gear.  He was able to get me captures from all his CUCM
servers up to a week back.  That was a pretty nice thing to have.
 Unfortunately, you're limited on the phone side in that you'll need to set
it up for quite a few phones to increase your chances of catching it.

On Thu, Jun 5, 2014 at 2:29 PM, Lelio Fulgenzi <lelio at uoguelph.ca> wrote:

> Just to add, we too are experiencing the same issue. Although I don't
> think we've had complaints of dropped calls.
>
> We first saw the issue when upgrading from v4.1 to v7.1 around two years
> ago, but because we didn't get complaints we had to prioritize. We're now
> seeing many more dropped phones. I'm not sure if the alerts were there in
> v4.1 and we just didn't have the alerts set up to come to us or not. It was
> a long time ago.
>
> We opened a TAC case, but unfortunately, we have not been able to work on
> it due to other pressing issues. I have to go back to our team to
> co-ordinate getting this going again. I'd like to resolve this. There
> should be no reason to have such a large number of de-registration events.
>
> We are primarily 7940/60 7912 phones with a number of 7911s, 7942, 7962
> interspersed.
>
> I too am of the thought that it's not CUCM version dependent, but more so
> either switch configuration (loop detection) or firewall issues.
>
> The case owner said the only real way to figure things out is to get a
> packet capture from both the callmanager and the phones when it happens.
>
> Unfortunately, it's difficult to do so, since how do you guess where the
> phones are going to un-register from? We might have to do some sort of
> multi-vlan capture, but that is gonna be tough too.
>
>
>
> ---
> Lelio Fulgenzi, B.A.
> Senior Analyst, Network Infrastructure
> Computing and Communications Services (CCS)
> University of Guelph
>
> 519‐824‐4120 Ext 56354
> lelio at uoguelph.ca
> www.uoguelph.ca/ccs
> Room 037, Animal Science and Nutrition Building
> Guelph, Ontario, N1G 2W1
>
> ------------------------------
> *From: *"Tashi Mar" <tmar at align.com>
> *To: *"Terry Oakley" <Terry.Oakley at rdc.ab.ca>, "Dennis Heim" <
> Dennis.Heim at wwt.com>, cisco-voip at puck.nether.net
> *Sent: *Thursday, June 5, 2014 2:10:17 PM
>
> *Subject: *Re: [cisco-voip] Intermittent phone resets
>
>  Hi all, appreciate the feedback.  There are some SCCP phones (along with
> SIP) that reset too; TAC could not pinpoint any phone firmware or CUCM bugs
> either.
>
>
>
> Will find out the switch OS, but environment is all stacked 3750s at the
> core and access layers in both locations.  Any chance you recall the bug ID?
>
>
>
> Definitely will check into:
>
> -          Switch bug related to spanning tree and excessive Topology
> Change Notifications (TCN).  They actually replaced some access switches,
> but that doesn’t mean the configuration was modified.
>
> -          Circular buffer packet captures
>
>
>
> We did verify that the ASAs are not doing any SIP or SCCP inspection;
> voice traffic on this network does not pass through firewalls anyway.
>
>
>
> Thanks!
>
>
>
> Tashi Mar | IP Telephony Engineer
>
>
>
> *From:* Terry Oakley [mailto:Terry.Oakley at rdc.ab.ca]
> *Sent:* Thursday, June 05, 2014 2:02 PM
> *To:* Heim, Dennis; Mar, Tashi; cisco-voip at puck.nether.net
> *Subject:* RE: Intermittent phone resets
>
>
>
> What version of switches and OS were you using?
>
>
>
> Thanks
>
>
>
> Terry
>
>
>
> *From:* cisco-voip [mailto:cisco-voip-bounces at puck.nether.net
> <cisco-voip-bounces at puck.nether.net>] *On Behalf Of *Heim, Dennis
> *Sent:* June-05-14 11:50 AM
> *To:* Mar, Tashi; cisco-voip at puck.nether.net
> *Subject:* Re: [cisco-voip] Intermittent phone resets
>
>
>
> I had this awhile back, and it ended up being a bug on the switches
> related to spanning tree and excessive Topology Change Notifications (TCN).
>
>
>
> *Dennis Heim | Collaboration Solutions Architect*
>
> World Wide Technology, Inc. | +1 314-212-1814
>
> [image: twitter] <https://twitter.com/CollabSensei>
>
> [image: chat][image: Phone] <+13142121814>[image: video]
>
>
>
>
>
> *From:* cisco-voip [mailto:cisco-voip-bounces at puck.nether.net
> <cisco-voip-bounces at puck.nether.net>] *On Behalf Of *Mar, Tashi
> *Sent:* Thursday, June 05, 2014 1:43 PM
> *To:* cisco-voip at puck.nether.net
> *Subject:* [cisco-voip] Intermittent phone resets
>
>
>
> Hi, troubleshooting an issue where phones intermittently reset, either on
> hook or off hook (drops the call in progress).  Logs and TAC claim it’s a
> network issue, but has anyone seen similar problems on Call Manager 9.1?
> Basically, we want to rule out Call Manager as the culprit.
>
>
>
> Majority of phones using SIP, models 8891 and 9951, experiencing issues
> similar to those outlined here:
> https://supportforums.cisco.com/discussion/10846666/ip-phones-randomly-rebooting
>
>
>
> Steps already taken:
>
> - upgrade to latest firmware
>
> - verify no POE errors on switch
>
> - cpu, memory on UCS servers ok
>
> - point phones to backup SUB, no change.  Phones randomly reset when homed
> to Pub or Sub (both in separate locations).
>
> - Wireshark to see KeepAlives dropping
>
>
>
> 1.            Phone logs show:
>
> SEP0DD3B4E90000.cnf.xml.sgn (HTTP)
>
>   8:05:20p TCP connection timed out
>
>   8:07:22p Falling back to different CUCM
>
>   11:16:50p TCP connection timed out
>
>   11:18:51p Falling back to different CUCM
>
>   6:52:38p TCP connection timed out
>
>   6:54:39p Falling back to different CUCM
>
>   9:55:04a TCP connection timed out
>
>   9:57:05a Falling back to different CUCM
>
>
>
> 2.            Call Manager logs show:
>
> - phone unregisters from Primary CUCM due to:
>
> [Reason=6] ConnectivityError - Network communication between the device
> and Unified CM has been interrupted.
>
> - phones fallback to secondary CUCM and then rehome to Primary CUCM:
>
> Reason code = 28, FallbackInitiated - The device has initiated a fallback
> and will automatically re-register to a higher-priority Unified CM. No
> action is necessary.
>
>
>
> 3.  Call Manager syslogs display EndPointUnregistered:
>
>
>
> Apr 01 14:34:18 %UC_CALLMANAGER-3-EndPointUnregistered:
> %[DeviceName=SEP0DD3B4E90000][IPAddress=10.1.10.211][Protocol=SIP][DeviceType=540][Description=User][Reason=6][IPAddrAttributes=0][LastSignalReceived=SIPConnControlInd][CallState=5286-call_initiated1][AppID=Cisco
> CallManager][ClusterID=NAregion][NodeID=CCMSub]: An endpoint has
> unregistered
>
>
>
> Apr 01 14:36:22 %UC_-3-LastOutOfServiceInformation:
> %[DeviceName=SEP0DD3B4E90000][DeviceIPv4Address=
> 10.1.10.211/24][IPv4DefaultGateway=192.168.51.1][DeviceIPv6Address=::][IPv6DefaultGateway=::][ModelNumber=CP-8961][NeighborIPv4Address=][NeighborIPv6Address=][NeighborDeviceID=][UNKNOWN_PARAMNAME:NeighborIPortID=][DHCPv4Status=1][DHCPv6Status=3][TFTPCfgStatus=1][DNSStatusUnifiedCM1=4][DNSStatusUnifiedCM2=4][DNSStatusUnifiedCM3=0][UNKNOWN_PARAMNAME:DNSv6StatusUnifiedCM1=0][UNKNOWN_PARAMNAME:DNSv6StatusUnifiedCM2=0][UNKNOWN_PARAMNAME:DNSv6StatusUnifiedCM3=0][VoiceVLAN=50][UnifiedCMIPAddress=10.1.10.5][LocalPort=50382][TimeStamp=1396377375][ReasonForOutOfService=18][LastProtocolEventSent=Sent:INVITE
> <http://10.1.10.211/24%5D%5BIPv4DefaultGateway=192.168.51.1%5D%5BDeviceIPv6Address=::%5D%5BIPv6DefaultGateway=::%5D%5BModelNumber=CP-8961%5D%5BNeighborIPv4Address=%5D%5BNeighborIPv6Address=%5D%5BNeighborDeviceID=%5D%5BUNKNOWN_PARAMNAME:NeighborIPortID=%5D%5BDHCPv4Status=1%5D%5BDHCPv6Status=3%5D%5BTFTPCfgStatus=1%5D%5BDNSStatusUnifiedCM1=4%5D%5BDNSStatusUnifiedCM2=4%5D%5BDNSStatusUnifiedCM3=0%5D%5BUNKNOWN_PARAMNAME:DNSv6StatusUnifiedCM1=0%5D%5BUNKNOWN_PARAMNAME:DNSv6StatusUnifiedCM2=0%5D%5BUNKNOWN_PARAMNAME:DNSv6StatusUnifiedCM3=0%5D%5BVoiceVLAN=50%5D%5BUnifiedCMIPAddress=10.1.10.5%5D%5BLocalPort=50382%5D%5BTimeStamp=1396377375%5D%5BReasonForOutOfService=18%5D%5BLastProtocolEventSent=Sent:INVITE>
> sip:915138521010 at 10.1.10.2;user=phone SIP/2.0  Cseq:101 INVITE
> CallId:b4e9b08c-a3ad023c-1fec1fa7-3b9727b1 at 10.1][LastProtocolEventReceived=Rcvd:SIP/2.0
> 202 Accepted  Cseq:104 REFER
> CallId:b4e9b08c-a3ad026a-30d892f1-02c748ba at 10.1.10.211][AppID=Cisco
> CallManager][ClusterID=NAregion][NodeID=CCMSub]:  Information related to
> the last out-of-service event
>
>
>
> Apr 01 14:36:24 %UC_CALLMANAGER-6-EndPointRegistered:
> %[DeviceName=SEP0DD3B4E90000][IPAddress=10.1.10.211][Protocol=SIP][DeviceType=540][PerfMonObjType=2][Description=User][UserID=User1][LoadID=sip8961.9-4-1-9][AssociatedDNs=5286,5285,5242,5237,5262,5217,5284,5294,5282,#86,][MACAddress=0DD3B4E90000][IPAddrAttributes=0][ActiveLoadId=sip8961.9-4-1-9.loads][InactiveLoadId=sip8961.9-3-2SR1-1.loads][AppID=Cisco
> CallManager][ClusterID=NAregion][NodeID=CCMSub]: Endpoint registered
>
>
>
> Apr 01 14:36:24 %UC_CALLMANAGER-6-DeviceDnInformation:
> %[DeviceName=SEP0DD3B4E90000][DeviceType=540][StationDesc=User][StationDn=5286,5285,5242,5237,5262,5217,5284,5294,5282,#86][AppID=Cisco
> CallManager][ClusterID=NAregion][NodeID=CCMSub]: List of directory numbers
> (DN) associated with this device
>
>
>
> 4.  Switchport configuration:
>
>
>
> interface GigabitEthernet3/0/x
>
> switchport access vlan 10
>
> switchport mode access
>
> switchport voice vlan 11
>
> switchport port-security maximum 3
>
> switchport port-security
>
> switchport port-security aging time 2
>
> switchport port-security violation restrict
>
> switchport port-security aging type inactivity
>
> load-interval 30
>
> spanning-tree portfast
>
> spanning-tree bpduguard enable
>
>
>
> Thanks, Tashi
>
>
>  ------------------------------
>
>
> The information contained in this message is confidential and is intended
> only for the use of the individual or entity named above. It may contain
> proprietary or legally privileged information. Mistransmission shall not
> constitute a waiver of any rights or privileges. If you are not the
> designated recipient of this message, you are hereby notified that any use,
> dissemination, distribution or reproduction of this message is strictly
> prohibited. If you have received this message in error, please immediately
> notify the sender.
>
> Although this e-mail and any attachments are believed to be free of any
> virus or other defect that might affect any computer system into which it
> is received and opened, it is the responsibility of the recipient to ensure
> that they are virus-free. Align Communications Inc. does not accept, and
> specifically disclaims, any liability or obligation for any loss or damage
> arising in any way from the use of this e-mail or any attachment. Thank You.
>
> ------------------------------
>
> The information contained in this message is confidential and is intended
> only for the use of the individual or entity named above. It may contain
> proprietary or legally privileged information. Mistransmission shall not
> constitute a waiver of any rights or privileges. If you are not the
> designated recipient of this message, you are hereby notified that any use,
> dissemination, distribution or reproduction of this message is strictly
> prohibited. If you have received this message in error, please immediately
> notify the sender.
>
> Although this e-mail and any attachments are believed to be free of any
> virus or other defect that might affect any computer system into which it
> is received and opened, it is the responsibility of the recipient to ensure
> that they are virus-free. Align Communications Inc. does not accept, and
> specifically disclaims, any liability or obligation for any loss or damage
> arising in any way from the use of this e-mail or any attachment. Thank You.
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20140605/86b7b05a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 3876 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20140605/86b7b05a/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 1391 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20140605/86b7b05a/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1292 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20140605/86b7b05a/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 1389 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20140605/86b7b05a/attachment-0003.png>