[VoiceOps] Ideas for Building Inbound Redundancy

[Markus]

Am 02.02.2017 um 16:30 schrieb Voip Jacob:
> The case that we're trying to protect against would be if both PBXs at
> both data centers were unreachable for our SIP provider (DNS issues,
> internal network routing issues, routing issues between SIP provider &
> datacenters, etc.). [...]

I can't help exactly with the call queues/groups thingie, but here's 
what I did recently for my inbound infrastructure where DIDs get routed 
to me from several carriers and routed from me to my customers, via SIP. 
I believe I have covered all possible outage scenarios with this setup:

There are 2 data centers, geographically diverse. I colocate servers + 
routers in each. Let's call that a "site". Each site has two redundant 
(with VRRP) routers that speak BGP with the upstream routers.

Per site, there are 2 physical servers with CentOS + KVM, and each 
server hosts 2 VMs:
VM 1) Asterisk for SIP/RTP + Galera Cluster for database
VM 2) quagga for BGP + kamailio as SIP proxy for load balancing/SIP failover

A total of 8 VMs. (To start with)

Network:

I took a spare /24 IPv4 netblock I had lying around, and quagga running 
on the 4 quagga/kamailio VMs is announcing this prefix via BGP to the 4 
internet-facing routers. Each quagga connects to both the active and 
standby router local to this site. That means a total of 8 BGP sessions, 
4 per site, 2 per router.

Announcing this prefix at multiple sites at the same time, where each 
site uses different upstream providers, results in that IPv4 prefix 
becoming "anycast'ed", meaning it is visible in the global routing table 
via multiple paths and the decision at which site IP packets for this 
prefix ends up on is made by the BGP algorithm (and by the provider 
where the traffic originates).

A single IP address out of that /24 is up on all 4 quagga VMs as an 
alias address. Yes, the same IP address, four times. You may think this 
might be broken and cause problems if there is the same IP up in the 
same VLAN, but, the BGP algorithm on our internet-facing routers will 
choose one of the VMs and decide where to send all traffic to, the other 
VM will act as standby. There is no LAN traffic to/from this particular 
IP, so it just works. Initially I messed around with "keepalived" and 
some other tools, but it didn't work out, and running rather less 
userland daemons (which can crash, too) is better. :)

Now, we tell the providers that we buy DIDs from: Hey, route all the SIP 
packets for us to this particular IP only. A single IP is all they need 
and get from us! No more "Please add our new IP address ...".

Database:

I use both Asterisk' dialplan and also A2Billing (a "VoIP Softswitch 
Solution", open source and free of charge) to route DIDs to customers. 
Because of A2B and because of CDRs we need a MySQL database. This is 
what Galera Cluster is for, a multi-master active-active replacement for 
MySQL. There are 4 Asterisk/database VMs, so there are 4 instances 
running which synchronize each other all the time, thus it does not 
matter to which instance you are sending your write requests. I simply 
use the local node for read + write and let Galera take care of the 
internals. There is also a 9th VM in a country far, far way which only 
runs Galera arbiter, does not store any MySQL data and simply acts like 
a decision-making component which is there to prevent split-brain 
situations because of the even node count. It's good that it's far away 
so it is aware when a whole site is down due to network issues.

SIP + media:

kamailio running on the quagga VM is the entry point for all inbound SIP 
traffic. A simple configuration which basically just says: There are 2 
Asterisk servers to distribute the calls to. Check if they're both up. 
If one of them is down, send all traffic to the remaining server. If 
both are up, distribute evenly 50/50 so we get load balancing and all of 
our servers will actually process calls and not just sit idle until 
disaster comes. We let Asterisk handle all RTP and don't worry about an 
RTP proxy.

Each Asterisk VM has an public IP address that is local to this site and 
unique. So I tell my customers: Please allow inbound calls from these IPs:

5.5.5.5 (site 1, server 1, VM 1)
5.5.5.6 (site 1, server 2, VM 1)
90.90.90.90 (site 2, server 1, VM 1)
90.90.90.91 (site 2, server 2, VM 1)

Now, let's go through the possible disasters and see how this whole 
thing will react:

- Data center lights up in a big ball of fire/upstreams go down/fiber 
cut etc.: site 1 is down, thanks to BGP anycast all traffic will 
instantly and without manual intervention go to site 2, and vice versa.

- Router dies: Remaining router takes over, BGP sessions to both quaggas 
on each, BGP sessions to upstreams on each, VRRP between them. Instantly 
+ no manual intervention.

- Physical server dies: quagga VM goes down, BGP session disappears, BGP 
session to quagga VM on remaining physical server takes priority on 
router. (quagga + kamailio are active all the time on both VMs, on 
"standby" VM just sit idle until the other quagga disappears)

- quagga VM down/reboot: BGP session disappears, remaining VM gets priority.

- Asterisk crashes: kamailio detects this and sends all calls to 
remaining Asterisk.

All MySQL data is always everywhere, always write-able, regardless of 
site blowup, server failure or VM crash. We don't worry about harddrive 
faults or filesystem redundancy (GlusterFS, ...). Keep it simple. If a 
server dies, we replace it. (We still use RAID, of course)

I make all configuration changes on a single node. A simple script syncs 
the configuration with the other Asterisk'es and reloads them. Same for 
web access to A2B, a single node is designated for that, but you could 
easily make that redundant as well if web access is crucial, again 
thanks to BGP anycast.

The cool thing is that it scales for both load and redundancy count, 
just add new servers as you please on any site and add them to Galera, 
kamailio, even BGP if you wish. You could even add more sites in other 
cities, countries, continents.

If you have read until this point you have possibly figured out that if 
kamailio crashes on the quagga VM that has currently priority, calls 
will go to a black hole. I was too lazy to setup kamailio-failover...yet :)

Also, since there are multiple carriers that deliver DIDs, spread over 
the world and using different upstreams, anycast really does its job and 
some traffic arrives at site 1, other traffic at site 2. By luck, it's 
currently close to a 50/50 distribution.

Since I took the couple of days to implement that, I sleep so well again. :)

Regards
Markus

PS: BGP anycast is awesome.