<span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px"><div>Hi Markus, I'm curious. How are you handling reinvite traffic (And/or changes in SDP) from different sources when using Anycast?</div>
<div> </div>
<div>I'll give you an example. I have a provider I use for outbound LD. This provider takes my traffic, LCR's it and then steps out of the media path. Meaning, I send SIP to $SIPPROVIDER, They respond with SDP turning up media directly from lets say Level 3 to me. I exchange RTP directly with Level 3.</div>
<div> </div>
<div>Would there not be a potential issue here with a change in source of RTP? In that, The call could originate from $SITE1, Level3 being closer to $SITE2 Anycasts the inbound RTP traffic to a switch/site that has no active call for the incoming RTP?</div>
<div> </div>
<div>Now, I've never seen this on inbound calls. But see it all the time on outbound. Perhaps that's the make/break here? Thoughts?</div>
<div> </div>
<div>
<div><strong>Nick Olsen</strong></div>
<div>Sr. Network Engineer</div>
<div>Florida High Speed Internet</div>
<div>(321) 205-1100 x106</div>
<div><a href="http://twitter.com/flhsi"><img alt="" height="64" src="http://www.flhsi.com/files/emaillogo.jpg" width="188" /></a></div>
<div> </div>
<div><a href="https://twitter.com/flhsi"><img alt="" height="36" src="http://flhsi.com/files/twitter.png" width="36" /></a><a href="https://facebook.com/flhighspeed"><img alt="" height="36" src="http://flhsi.com/files/facebook.png" width="36" /></a></div>
<div> </div>
<div> </div>
</div>
<div> </div>
<hr align="center" size="2" width="100%" />
<div><span style="font-family: tahoma,arial,sans-serif; font-size: 10pt;"><b>From</b>: "Markus" <universe@truemetal.org><br />
<b>Sent</b>: Friday, February 03, 2017 3:02 PM<br />
<b>To</b>: voiceops@voiceops.org<br />
<b>Subject</b>: Re: [VoiceOps] Ideas for Building Inbound Redundancy</span>
<div> </div>
Am 02.02.2017 um 16:30 schrieb Voip Jacob:<br />
> The case that we're trying to protect against would be if both PBXs at<br />
> both data centers were unreachable for our SIP provider (DNS issues,<br />
> internal network routing issues, routing issues between SIP provider &<br />
> datacenters, etc.). [...]<br />
<br />
I can't help exactly with the call queues/groups thingie, but here's<br />
what I did recently for my inbound infrastructure where DIDs get routed<br />
to me from several carriers and routed from me to my customers, via SIP.<br />
I believe I have covered all possible outage scenarios with this setup:<br />
<br />
There are 2 data centers, geographically diverse. I colocate servers +<br />
routers in each. Let's call that a "site". Each site has two redundant<br />
(with VRRP) routers that speak BGP with the upstream routers.<br />
<br />
Per site, there are 2 physical servers with CentOS + KVM, and each<br />
server hosts 2 VMs:<br />
VM 1) Asterisk for SIP/RTP + Galera Cluster for database<br />
VM 2) quagga for BGP + kamailio as SIP proxy for load balancing/SIP failover<br />
<br />
A total of 8 VMs. (To start with)<br />
<br />
Network:<br />
<br />
I took a spare /24 IPv4 netblock I had lying around, and quagga running<br />
on the 4 quagga/kamailio VMs is announcing this prefix via BGP to the 4<br />
internet-facing routers. Each quagga connects to both the active and<br />
standby router local to this site. That means a total of 8 BGP sessions,<br />
4 per site, 2 per router.<br />
<br />
Announcing this prefix at multiple sites at the same time, where each<br />
site uses different upstream providers, results in that IPv4 prefix<br />
becoming "anycast'ed", meaning it is visible in the global routing table<br />
via multiple paths and the decision at which site IP packets for this<br />
prefix ends up on is made by the BGP algorithm (and by the provider<br />
where the traffic originates).<br />
<br />
A single IP address out of that /24 is up on all 4 quagga VMs as an<br />
alias address. Yes, the same IP address, four times. You may think this<br />
might be broken and cause problems if there is the same IP up in the<br />
same VLAN, but, the BGP algorithm on our internet-facing routers will<br />
choose one of the VMs and decide where to send all traffic to, the other<br />
VM will act as standby. There is no LAN traffic to/from this particular<br />
IP, so it just works. Initially I messed around with "keepalived" and<br />
some other tools, but it didn't work out, and running rather less<br />
userland daemons (which can crash, too) is better. :)<br />
<br />
Now, we tell the providers that we buy DIDs from: Hey, route all the SIP<br />
packets for us to this particular IP only. A single IP is all they need<br />
and get from us! No more "Please add our new IP address ...".<br />
<br />
Database:<br />
<br />
I use both Asterisk' dialplan and also A2Billing (a "VoIP Softswitch<br />
Solution", open source and free of charge) to route DIDs to customers.<br />
Because of A2B and because of CDRs we need a MySQL database. This is<br />
what Galera Cluster is for, a multi-master active-active replacement for<br />
MySQL. There are 4 Asterisk/database VMs, so there are 4 instances<br />
running which synchronize each other all the time, thus it does not<br />
matter to which instance you are sending your write requests. I simply<br />
use the local node for read + write and let Galera take care of the<br />
internals. There is also a 9th VM in a country far, far way which only<br />
runs Galera arbiter, does not store any MySQL data and simply acts like<br />
a decision-making component which is there to prevent split-brain<br />
situations because of the even node count. It's good that it's far away<br />
so it is aware when a whole site is down due to network issues.<br />
<br />
SIP + media:<br />
<br />
kamailio running on the quagga VM is the entry point for all inbound SIP<br />
traffic. A simple configuration which basically just says: There are 2<br />
Asterisk servers to distribute the calls to. Check if they're both up.<br />
If one of them is down, send all traffic to the remaining server. If<br />
both are up, distribute evenly 50/50 so we get load balancing and all of<br />
our servers will actually process calls and not just sit idle until<br />
disaster comes. We let Asterisk handle all RTP and don't worry about an<br />
RTP proxy.<br />
<br />
Each Asterisk VM has an public IP address that is local to this site and<br />
unique. So I tell my customers: Please allow inbound calls from these IPs:<br />
<br />
5.5.5.5 (site 1, server 1, VM 1)<br />
5.5.5.6 (site 1, server 2, VM 1)<br />
90.90.90.90 (site 2, server 1, VM 1)<br />
90.90.90.91 (site 2, server 2, VM 1)<br />
<br />
Now, let's go through the possible disasters and see how this whole<br />
thing will react:<br />
<br />
- Data center lights up in a big ball of fire/upstreams go down/fiber<br />
cut etc.: site 1 is down, thanks to BGP anycast all traffic will<br />
instantly and without manual intervention go to site 2, and vice versa.<br />
<br />
- Router dies: Remaining router takes over, BGP sessions to both quaggas<br />
on each, BGP sessions to upstreams on each, VRRP between them. Instantly<br />
+ no manual intervention.<br />
<br />
- Physical server dies: quagga VM goes down, BGP session disappears, BGP<br />
session to quagga VM on remaining physical server takes priority on<br />
router. (quagga + kamailio are active all the time on both VMs, on<br />
"standby" VM just sit idle until the other quagga disappears)<br />
<br />
- quagga VM down/reboot: BGP session disappears, remaining VM gets priority.<br />
<br />
- Asterisk crashes: kamailio detects this and sends all calls to<br />
remaining Asterisk.<br />
<br />
All MySQL data is always everywhere, always write-able, regardless of<br />
site blowup, server failure or VM crash. We don't worry about harddrive<br />
faults or filesystem redundancy (GlusterFS, ...). Keep it simple. If a<br />
server dies, we replace it. (We still use RAID, of course)<br />
<br />
I make all configuration changes on a single node. A simple script syncs<br />
the configuration with the other Asterisk'es and reloads them. Same for<br />
web access to A2B, a single node is designated for that, but you could<br />
easily make that redundant as well if web access is crucial, again<br />
thanks to BGP anycast.<br />
<br />
The cool thing is that it scales for both load and redundancy count,<br />
just add new servers as you please on any site and add them to Galera,<br />
kamailio, even BGP if you wish. You could even add more sites in other<br />
cities, countries, continents.<br />
<br />
If you have read until this point you have possibly figured out that if<br />
kamailio crashes on the quagga VM that has currently priority, calls<br />
will go to a black hole. I was too lazy to setup kamailio-failover...yet :)<br />
<br />
Also, since there are multiple carriers that deliver DIDs, spread over<br />
the world and using different upstreams, anycast really does its job and<br />
some traffic arrives at site 1, other traffic at site 2. By luck, it's<br />
currently close to a 50/50 distribution.<br />
<br />
Since I took the couple of days to implement that, I sleep so well again. :)<br />
<br />
Regards<br />
Markus<br />
<br />
PS: BGP anycast is awesome.<br />
<br />
_______________________________________________<br />
VoiceOps mailing list<br />
VoiceOps@voiceops.org<br />
https://puck.nether.net/mailman/listinfo/voiceops<br />
</div></span>