[cisco-voip] [External] Re: MRA failover doesn't work, Cisco TAC agrees, says it's a documentation defect

Tue Jun 21 16:55:06 EDT 2022

Yes, they want 6 boxes for redundancy. Two for expressway-e, two for CUCM, two for expressway-c. /Boggle

Even then, that doesn't provide 100% redundancy. We want to place our CUCM and expressways at different datacenters connected by a 10GB wan. If we were to loose the WAN, we would still fail with MRA since we would lose both ESXi hosts.

-----Original Message-----
From: Lelio Fulgenzi <lelio at uoguelph.ca> 
Sent: Tuesday, June 21, 2022 4:30 PM
To: Matthew Huff <mhuff at ox.com>; Adam Pawlowski <ajp26 at buffalo.edu>; cisco-voip voyp list <cisco-voip at puck.nether.net>
Subject: RE: [cisco-voip] [External] Re: MRA failover doesn't work, Cisco TAC agrees, says it's a documentation defect

I was miffed when they said no real MRA redundancy until you upgrade to v14. But now, hearing this, man, what a disappointment. 

I had a similar discussion with the Expressway folks and ESXi compatibility/testing and they were like, yeah, you should probably have separate UCS boxes for Expressways different than your CUCMs. 

And I was all, "wait, what?" 

They want us to run a completely separate ESXi box with only an E or a C on it to get full MRA redundancy? 

What a let down. ☹

-----Original Message-----
From: cisco-voip <cisco-voip-bounces at puck.nether.net> On Behalf Of Matthew Huff
Sent: Tuesday, June 21, 2022 1:37 PM
To: Adam Pawlowski <ajp26 at buffalo.edu>; cisco-voip voyp list <cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] [External] Re: MRA failover doesn't work, Cisco TAC agrees, says it's a documentation defect

CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp at uoguelph.ca

Yes, that sounds almost exactly what we are experiencing. I think it's a design defect with the MRA architecture with the end device not downloading/retrying with the full environment.

It's the same issue we have with partial registrations. We have a number of shared SIP lines (think SALES line) that can silently fail on phone. It will try for a few minutes, but give up after that. The user doesn't know that the shared SIP line is disconnected, they just don't get calls on it. We had to add a complex SNMP monitoring so that we can be alerted when this happen and remotely reset the phones. Cisco TAC is aware of this issue and also told us it's "working as intended". We had a sales trader lose about $10k of commission because he missed a call, and he was not a happy camper.

-----Original Message-----
From: cisco-voip <cisco-voip-bounces at puck.nether.net> On Behalf Of Adam Pawlowski
Sent: Tuesday, June 21, 2022 1:04 PM
To: cisco-voip voyp list <cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] [External] Re: MRA failover doesn't work, Cisco TAC agrees, says it's a documentation defect

I'm a bit miffed on the need for the extra expressway C. We have very few MRA phones, but hadn't had this type of problem, an expressway is somehow busted and not accepting registrations - did they offer any explanation as to why that piece is needed?

The only thing I'd go to look up is how the CM list is being populated, if changing the CM group to bump the shut subscriber down (assuming reg order is sub -> pub), just since that'd come up before. Expressway doesn't seem to be configured to be aware that a UCM has gone away, despite the zone going down, at least for UDS and discovery.  I'm sure TAC looked at that though. I have a conversation going with them about this and Jabber SSO for a similar reason, that the device's configuration isn't dynamic to represent the state of the infrastructure, and sometimes they get stuck trying something that won't work and fail despite other components being available to serve them. That probably doesn't help with anything other than to say we're in a similar boat, just with Jabber and MRA.

Adam Pawlowski
Network Engineer | Network and Communication Services University at Buffalo Information Technology (UBIT)
243 Computing Center, Buffalo, NY 14260 

> -----Original Message-----
> From: cisco-voip <cisco-voip-bounces at puck.nether.net> On Behalf Of 
> Matthew Huff
> Sent: Tuesday, June 21, 2022 12:54 PM
> To: Hunter Fuller <hf0002 at uah.edu>
> Cc: cisco-voip voyp list <cisco-voip at puck.nether.net>
> Subject: Re: [cisco-voip] [External] Re: MRA failover doesn't work, 
> Cisco TAC agrees, says it's a documentation defect
> 
> We have no interest in setting up a jabber environment in order to 
> debug ciscos's issue.
> 
> Yes, every expressway-e knows about all expressway-c, all expressway-c 
> know about CUCM. Cisco TAC has verified the configuration, logs, and 
> diagnostic. I've been working with them for 2 months and it's been 
> escalated to backline-engineering. They looked at the Cisco Phone PRT 
> logs and confirmed that it's a known limitation, and there is no solution.
> 
> Maybe it's an issue with later versions of CUCM and/or expressway? We 
> are running the latest including latest phone firmware.
> 
> Failover works great except in one scenario where both the CUCM 
> subscriber and the expressway-c that reside on the same machine are both shut down.
> Brining either one up, and the phone registers.
> 
> 
> -----Original Message-----
> From: Hunter Fuller <hf0002 at uah.edu>
> Sent: Tuesday, June 21, 2022 12:41 PM
> To: Matthew Huff <mhuff at ox.com>
> Cc: Kent Roberts <dvxkid at gmail.com>; cisco-voip voyp list <cisco- 
> voip at puck.nether.net>
> Subject: Re: [External] Re: [cisco-voip] MRA failover doesn't work, 
> Cisco TAC agrees, says it's a documentation defect
> 
> It might be worth setting up a Jabber test endpoint just to see.
> 
> Some questions though:
>  - Does every Expressway-E know about every Expressway-C?
>  - Does every Expressway-C know about every CUCM?
> 
> I'm trying to figure out what the desired architecture is, and/or how 
> this problem would happen.
> In our environment, the above are both true. So the loss of any number 
> of anything, should not result in failover issues - and that is the 
> behavior we have seen (we have shut down entire sites due to 
> maintenance, power failure, etc. and failover worked).
> In fact, we have found MRA phones to be great at failover in this way 
> (our MRA phones are all 8851s). Jabber has been the problem child.
> 
> --
> Hunter Fuller (they)
> Router Jockey
> VBH M-1C
> +1 256 824 5331
> 
> Office of Information Technology
> The University of Alabama in Huntsville Network Engineering
> 
> On Tue, Jun 21, 2022 at 9:13 AM Matthew Huff <mhuff at ox.com> wrote:
> >
> > We don’t use Jabber nor Webex.
> >
> >
> >
> > Cisco TAC has been escalated and they have been working on this for 
> > over
> 2 months. I have sent repeated expressway and PRT logs from the phone.
> After working with Cisco engineering, the claim it is “working as intended”
> and plan on updating the documentation to reflect the limitation that 
> if you loose both the subscriber and redundant expressway-C server, 
> failover won’t happen.
> >
> >
> >
> > I’d love to be proven wrong since we may have to completely replace 
> > our
> solution.
> >
> >
> >
> >
> >
> > From: Kent Roberts <dvxkid at gmail.com>
> > Sent: Tuesday, June 21, 2022 10:09 AM
> > To: Matthew Huff <mhuff at ox.com>
> > Cc: cisco-voip voyp list <cisco-voip at puck.nether.net>
> > Subject: Re: [cisco-voip] MRA failover doesn't work, Cisco TAC 
> > agrees, says it's a documentation defect
> >
> >
> >
> > This sound more like a config issue…
> >
> >
> >
> > Have run into issues where expressways go stupid when boxes go 
> > offline
> >
> > As for it being the phones 88xx. Does the same happen with jabber or
> webex?    If it does i’d  requeue the case….
> >
> >
> >
> > Kent
> >
> >
> >
> > On Jun 21, 2022, at 07:47, Matthew Huff <mhuff at ox.com> wrote:
> >
> > 
> >
> > We have a fairly common and standard deployment for our MRA solution.
> > All are running CUCM 14+, latest Expressway, etc…
> >
> >
> >
> > Vmware server 1 (jn DMZ)
> >
> >   ExpressWay-E-1
> >
> >
> >
> > Vmware server 2 (in DMZ)
> >
> >    ExpressWay-E-2
> >
> >
> >
> > Vmware Server 3 (In Core)
> >
> >   CUCM Publisher
> >
> >    Expressway-C-1
> >
> >
> >
> > VMWare Server 4( In Core)
> >
> >    CUCM Subscriber
> >
> >    Expressway-C-2
> >
> >
> >
> >
> >
> > If ether Expreway-E VMs fail, redundancy works fine If either CUCM 
> > fails, redundancy works fine If either Expressway-C VMs fail, 
> > redundancy works fine If VMWare Server 4 fails (say during patching, 
> > hardware maintenance or hardware failure), redundancy fails. Remote
> phones un-register and never register no matter what is done. If 
> either CUCM Subscriber or Expressway-C-2 is brought back online, phones register.
> >
> >
> >
> > Cisco TAC claims that this is a limitation of our Cisco 88xx SIP MRA 
> > phones
> and is not solvable unless we purchase two new vmware servers and 
> split the CUCM and Expressway-C into separate servers so they both 
> won’t go down at once. Sinc VMWare Server 3 & 4 are at different 
> locations, vMotion isn’t an option since there is no shared storage.
> >
> >
> >
> > Anyone run into this or have any suggestions? We have engaged our 
> > VAR
> and cisco rep and may have to replace our phone system since we are 
> all working from home and MRA support including redundancy is critical to us.
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > cisco-voip mailing list
> > cisco-voip at puck.nether.net
> >
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck
> > .nether.net%2Fmailman%2Flistinfo%2Fcisco-
> voip&data=05%7C01%7Cajp26
> >
> %40buffalo.edu%7C719bda6c11134986ee9d08da53a6afaa%7C96464a8af8ed4
> 0b199
> >
> e25f6b50a20250%7C0%7C0%7C637914272634938090%7CUnknown%7CTWFp
> bGZsb3d8ey
> >
> JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> 7C300
> >
> 0%7C%7C%7C&sdata=rBPHyezmLEzor0096y7a9yyEuMv83QEn2tRd9a%2
> BI%2FsU%3
> > D&reserved=0
> >
> > _______________________________________________
> > cisco-voip mailing list
> > cisco-voip at puck.nether.net
> >
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck
> > .nether.net%2Fmailman%2Flistinfo%2Fcisco-
> voip&data=05%7C01%7Cajp26
> >
> %40buffalo.edu%7C719bda6c11134986ee9d08da53a6afaa%7C96464a8af8ed4
> 0b199
> >
> e25f6b50a20250%7C0%7C0%7C637914272634938090%7CUnknown%7CTWFp
> bGZsb3d8ey
> >
> JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> 7C300
> >
> 0%7C%7C%7C&sdata=rBPHyezmLEzor0096y7a9yyEuMv83QEn2tRd9a%2
> BI%2FsU%3
> > D&reserved=0
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck
> .nether.net%2Fmailman%2Flistinfo%2Fcisco-
> voip&data=05%7C01%7Cajp26%40buffalo.edu%7C719bda6c11134986ee
> 9d08da53a6afaa%7C96464a8af8ed40b199e25f6b50a20250%7C0%7C0%7C6379
> 14272634938090%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&a
> mp;sdata=rBPHyezmLEzor0096y7a9yyEuMv83QEn2tRd9a%2BI%2FsU%3D&a
> mp;reserved=0
_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip
_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip