[cisco-voip] 11.5.1SU2 Can't Add Node

Sun Aug 20 22:42:28 EDT 2017

Just following up on this to close my loop .... My issue is now resolved, and happened to be a more rare case I would suspect. My customer in this case had a Juniper network infrastructure that I did not have access to; one of these deals where you have to ask someone to do or see anything in their network. Anyway and to make a long story short; the customer had jumbo frame support enabled in the network path and didn't realize it.

It was difficult for me to spot solo because I could only see the PCAPs from the CUCM pub and didn't have good 3-way handshakes to analyze (not sure how I would spot outside of a 3-way handshake looking at the SYN packets where you could see the inbound packet on the receiving end). The client ultimately opened a case with Juniper support and when the topic was mentioned ... didn't take long for them to spot in the customer's network.

What brought me to this suspicion aside from the customer stating no filtering/inspection was happening over the WAN was in the PCAPs; I could see successful communication back and forth until a 'certain point'; which in hindsight is where the packets exceed the default 1500 byte MTU CUCM/ESXi was using and unexpected fragmentation occurs which looks like packet loss to the TCP/IP stack on the servers. Then just a ton of TCP retransmissions because that is the nature of TCP, to retransmit failed packets. That isn't behavior consistent with ACLs or filtering/inspection because that behavior would typically be reflected immediately (when looking at the timeline of a packet capture sample set).

So don't rule out MTU! Now if you have control of the network end-to-end I'd guess you'd know if you had a fragmentation problem, but when you are blind-folded from the network, it can be like trying to herd a litter of feral cats.

-Ryan

________________________________
From: Matthew Loraditch <MLoraditch at heliontechnologies.com>
Sent: Thursday, August 10, 2017 7:52 AM
To: Anthony Holloway; Dana Tong; Ryan Huff; Dave Goodwin; Brian Meade
Cc: cisco-voip at puck.nether.net
Subject: RE: [cisco-voip] 11.5.1SU2 Can't Add Node

Well my password is 13 characters, but at least I’m not crazy. I’ll start from Scratch.

For this sort of thing, I generally wouldn’t open a case unless it failed.

As to opening cases, it’s not legit, but as long as you have access to any active CUCM contract, they really don’t care what install you are working on, we have similar issues where purchasing hasn’t caught up to the install cycle and we just open the case under another customer. Granted, we are an MSP, every contract is under our BID and the customers don’t see this in our setup.

Matthew G. Loraditch – CCNP-Voice, CCNA-R&S, CCDA
Network Engineer
Direct Voice: 443.541.1518

Facebook<https://www.facebook.com/heliontech?ref=hl> | Twitter<https://twitter.com/HelionTech> | LinkedIn<https://www.linkedin.com/company/helion-technologies?trk=top_nav_home> | G+<https://plus.google.com/+Heliontechnologies/posts>

From: cisco-voip [mailto:cisco-voip-bounces at puck.nether.net] On Behalf Of Anthony Holloway
Sent: Thursday, August 10, 2017 12:44 AM
To: Dana Tong <dana.tong at yellit.com.au>; Ryan Huff <ryanhuff at outlook.com>; Dave Goodwin <dave.goodwin at december.net>; Brian Meade <bmeade90 at vt.edu>
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] 11.5.1SU2 Can't Add Node

I've been noticing a higher rate of fresh installs failing in about the last 18 months or so.  I don't feel like opening a TAC case every time, because most of the time, I get the servers before the distributor even sends the smartnet stuff over, and CIN is pretty hard core about not letting me open CUCM cases with the server serial number.  I wish there was an easier way to report issues for partners who run into all kinds of issues we just don't have the time to report.

On Wed, Aug 9, 2017 at 8:29 PM Dana Tong <dana.tong at yellit.com.au<mailto:dana.tong at yellit.com.au>> wrote:

The most recent password was 12 characters, a mix of lower, upper, numbers, and special characters.

The other job was 11, and much the same.

I almost think it was a fresh Publisher, and using the OVA Subscriber. Then another fresh subscriber, for a 3 node cluster.

Cheers

From: Ryan Huff <ryanhuff at outlook.com<mailto:ryanhuff at outlook.com>>
Date: Thursday, 10 August 2017 at 11:26 am
To: Dana Tong <dana.tong at yellit.com.au<mailto:dana.tong at yellit.com.au>>, Dave Goodwin <dave.goodwin at december.net<mailto:dave.goodwin at december.net>>, Brian Meade <bmeade90 at vt.edu<mailto:bmeade90 at vt.edu>>

Cc: "cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>" <cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>>
Subject: Re: [cisco-voip] 11.5.1SU2 Can't Add Node

I just hit this issue tonight myself (11.5(SU1)); my case doesn't seem to fit CSCvb00248 as I am using 9 character passwords; I'm not using any special characters either. I've tried all the normal reboots, delete / re-added from the processNode table ... etc, NTP and DNS are solid. I opened a TAC case with the DB installer Service logs and Packet Captures from the pub. I'll update the thread if they turn anything up for me.

In parallel, I think I'm going to try Dana's fix and scrap the pub and start over.

-Ryan

________________________________

From: cisco-voip <cisco-voip-bounces at puck.nether.net<mailto:cisco-voip-bounces at puck.nether.net>> on behalf of Dana Tong <dana.tong at yellit.com.au<mailto:dana.tong at yellit.com.au>>
Sent: Wednesday, August 9, 2017 8:24 PM
To: Dave Goodwin; Brian Meade
Cc: cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] 11.5.1SU2 Can't Add Node

Yes I have had a similar error with the pre-installed OVA on two occasions. Mine was failing on the checking comms with the first node.

DNS / A and Reverse PTR was correct

Security password correct

Time / NTP was all working

Server nodes were added to Publisher as FQDN

My only solution was to build a fresh Publisher (or was it a fresh Subscriber?!?).

Hope this helps.

From: cisco-voip <cisco-voip-bounces at puck.nether.net<mailto:cisco-voip-bounces at puck.nether.net>> on behalf of Dave Goodwin <dave.goodwin at december.net<mailto:dave.goodwin at december.net>>
Date: Thursday, 10 August 2017 at 7:32 am
To: Brian Meade <bmeade90 at vt.edu<mailto:bmeade90 at vt.edu>>
Cc: "cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>" <cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>>
Subject: Re: [cisco-voip] 11.5.1SU2 Can't Add Node

A few months ago back on SU1, I was using a 16 character password for the platform on a brand new cluster and the subscriber installs would always fail. At first I thought it was a problem with PCD (which I was using to install the cluster) but turns out it happened even when doing the whole process by hand. The workaround was to use a 15 character password during the install. I haven't tried a fresh install of a cluster with SU2 yet to see if it's really fixed - but the affected versions field does list the SU2 build numbers.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvb00248/

Here is what any sub node looked like when it failed for this reason (assuming the screen shot attaches for me properly)...

On Wed, Aug 9, 2017 at 5:09 PM, Brian Meade <bmeade90 at vt.edu<mailto:bmeade90 at vt.edu>> wrote:

Found a bug related to custom banner size but not much else- https://bst.cloudapps.cisco.com/bugsearch/bug/CSCub14050

I was using the same exact version for CUCM and CUC with no issues adding nodes.

On Wed, Aug 9, 2017 at 5:07 PM, Brian Meade <bmeade90 at vt.edu<mailto:bmeade90 at vt.edu>> wrote:

I just did this successfully a couple weeks ago.  Where does it break?

On Wed, Aug 9, 2017 at 5:04 PM, Matthew Loraditch <MLoraditch at heliontechnologies.com<mailto:MLoraditch at heliontechnologies.com>> wrote:

I’ve set, reset, typed different versions of the security password and can’t get nodes added via the factory prebuilt VMs on brand new BE7Ks

Is there a known bug or am I crazy?

Both CUC and CUCM, albeit I doubt that matters since that part of the code is shared.

Matthew G. Loraditch – CCNP-Voice, CCNA-R&S, CCDA
Network Engineer
Direct Voice: 443.541.1518<tel:(443)%20541-1518>

Facebook<https://www.facebook.com/heliontech?ref=hl> | Twitter<https://twitter.com/HelionTech> | LinkedIn<https://www.linkedin.com/company/helion-technologies?trk=top_nav_home> | G+<https://plus.google.com/+Heliontechnologies/posts>

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20170821/0330ac8e/attachment.html>