[VoiceOps] VoIP Innovations reliability

Thu Mar 17 22:51:36 EDT 2016

Being directly connected is indeed better -- faster with fewer points of failure -- but is not infallible. It just means you're fewer AS hops away. If you use a carrier and they are attacked, chances are you're going to be impacted in some way, regardless if you are directly connected. Even the big guys have been victims of attack. Just a few short years ago (2011?) Verizon was attacked. AT&T was attacked (2012). UltraDNS/NeuStar was attacked (2014). It happens.

How you respond, both technically and customer-service-wise, makes all the difference. I think that's a hard lesson VI is learning from this. And for some on the list (I presume) are learning to diversify their access.

I agree with Alex that it's time the PSTN grows up and (we all) begin to architect the next generation PSTN to eliminate single route points of failure.

-Pete

On Mar 17, 2016, at 21:17, Alexander Lopez <alex.lopez at opsys.com> wrote:

I think that most on this list understand the inherent risks in using the Internet As A Cross Connect  (IaaCC, I want credit for that term BTW :-) ). 

However, we cannot just accept that this will be common place in the future. Dedicated links are becoming less and less prevalent as most move to the Internet to provide that last mile as well as back haul.

The Internet was built with resilience in its design, it is time that the PSTN steps up to the plate and breaks free from the single route architecture and provide multiple paths to connect a call. 

When my customers complain about a call not being perfect, I remind them that they are paying cents for what used to cost dollars. Sure it's cheaper, but 99.99999 percent of the time it's just fine.

 Since VI has most of the small to medium ITSP market, most of our competition is having the same issues, we are better off than most because we have multiple ingress points and don't have all eggs in the VI basket.

I wholeheartedly agree with what you said, you get what you pay for.....

-------- Original message --------
From: Ivan Kovacevic <ivan.kovacevic at startelecom.ca> 
Date: 3/17/2016 8:55 PM (GMT-05:00) 
To: voiceops at voiceops.org 
Subject: Re: [VoiceOps] VoIP Innovations reliability 

Ahem… <stands on the soap box>

At a risk of this not being a popular point of view… I view issues with VoIP Innovations as being largely self-inflicted.

Not only on part of VoIP Innovations (no relationship) who, as it appears, could have designed and communicated the DR better; but also on part of people complaining about the outage. You are using a non-guaranteed service. No SLAs, no promises, no commitments. If stuff like this happens, it is only because it is to be expected. No one can guarantee the quality of Internet between your IP and most other IPs on the Internet. That’s the reality. And if you are relying on a non-guaranteed service, you should at least have contingencies to move traffic quickly (CIC TFNs) or reroute (outbound).

And where that’s not possible (DIDs), having a dedicated data link covered by SLAs may be more appropriate.

And if you are porting your services to another provider with whom you will also interconnect over the Internet, if I was your client, I would be looking elsewhere. Unless your clients are not looking for a guaranteed 100% (or 99% or whatever) uptime. In which case, staying with VoIP Innovations is just fine…

And I know the response will likely be that it isn’t economically feasible to get dedicated connectivity, but by making that argument you are just saying that your business (and your clients’ business) doesn’t require a guaranteed level of uptime.

And either your clients have made this decision knowingly, in exchange for cost savings (in which case you are still delivering what you promised) or you made  this decision on behalf of your clients unbeknownst to them, in which case you are up a creek… because this is either going to happen again and you will have to justify yet again why their services are down, or you’ll have to pony up the cost of a dedicated connection without being to recover it from your’ clients’ service fees. Either way, it wasn’t a good idea in the first place and it isn’t VoIP Innovation’s fault, unless they mislead you by saying Internet communication is infallible, and you believed it… I have a bridge to sell you.

<ducks>

Best Regards,

Ivan Kovacevic
Vice President, Client Services
Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers

From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Anthony Orlando via VoiceOps
Sent: Thursday, March 17, 2016 8:09 PM
To: Paul Timmins <paul at timmins.net>
Cc: voiceops at voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

I get it Paul. It's just a shame the terrorists win. Too bad we can't put a bounty on their sorry asses. 

Sent from my iPhone

On Mar 17, 2016, at 6:55 PM, Paul Timmins <paul at timmins.net> wrote:

Sadly, assassins win if i stop breathing, so if someone starts to choke me, I fight it the best I can and run away at my earliest opportunity.

On 03/17/2016 07:38 PM, Anthony Orlando via VoiceOps wrote:
It's a shame we can't support them. This could be anyone of us. Hackers win if you port away. 

Sent from my iPhone

On Mar 17, 2016, at 5:43 PM, Robert Johnson <fasterfourier at gmail.com> wrote:

Their primary SBC and DR IP are both in the same IP netblock, so whenever the DDOS hits, both IPs are affected. The past few outages have involved 80% packet loss or so to both hosts, so some calls do make their way through, and plenty of wierdness ensues when an INVITE makes its way through but not the OK on the way back.

Can't wait to get our numbers ported out.

On Thu, Mar 17, 2016 at 6:27 PM, Nate Burke <nate at blastcomm.com> wrote:
Annnd they're down again.

On 3/17/2016 5:14 PM, Nate Burke wrote:
6 calls from 4 different CID numbers.  All within 3 minutes. 

On 3/17/2016 4:58 PM, Nathan Anderson wrote:
Yesterday I definitely saw calls coming from the DR IP in my logs, but I had not yet added that IP as a peerin our SBC.  I'll have to comb through logs today to see if we got any.

Are you saying that the multiple calls you saw coming to your desk were all from the same number?  If I had to guess, their side probably sent continuous INVITEs to you when it failed to get back an OK for any of them (not that you weren't sending back OK, but that their either didn't reach their SBC or did not reach it in a timely manner).

-- Nathan

From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Jeff Waddell
Sent: Thursday, March 17, 2016 2:53 PM
To: Nate Burke
Cc: voiceops at voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

That is the issue a lot of our customers are reporting - where multiple calls are sent.... 

On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <nate at blastcomm.com> wrote:
I didn't see any traffic increment on the DR IP Address in my firewall rules, but this was odd.  During the 15 minute period, I had probably 5 or 6 simultaneous calls ring into my desk.  I normally only take a handfull of calls a day.

On 3/17/2016 4:39 PM, Jeff Waddell wrote:
We implemented it too - I haven't checked to see if any traffic was sent across it

On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <nate at blastcomm.com> wrote:
Only 15 Minutes this time though.  I had implemented the Disaster Recover Trunk as mentioned previously, but I didn't seem to be getting any calls completed through it.

On 3/17/2016 4:16 PM, Shripal Daphtary wrote:
Down again! 

Thanks, 

Shripal

On Mar 16, 2016, at 9:50 AM, Nate Burke <nate at blastcomm.com> wrote:

Looks like it just came back up for me.  Just over 30 min.

Nate

On 3/16/2016 8:45 AM, Shripal Daphtary wrote:
We are experiencing an outage as well. 

Thanks, 

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com> wrote:

Problems again this morning?  Looks to be acting the same as it has been. 

On 3/11/2016 6:00 PM, Alexander Lopez wrote:
I added them to our monitoring platform, stated getting alarms this past hour or so.
Up and down.

-------- Original message --------
From: Nathan Anderson <nathana at fsr.com> 
Date: 3/11/2016 6:31 PM (GMT-05:00) 
To: 'Nate Burke' <nate at blastcomm.com>, voiceops at voiceops.org 
Subject: Re: [VoiceOps] VoIP Innovations reliability

...aaaaaand we're back.

-- Nathan

-----Original Message-----
From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson
Sent: Friday, March 11, 2016 3:30 PM
To: 'Nate Burke'; voiceops at voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.

-- Nathan

-----Original Message-----
From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson
Sent: Friday, March 11, 2016 3:28 PM
To: 'Nate Burke'; voiceops at voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-----Original Message-----
From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nate Burke
Sent: Friday, March 11, 2016 3:26 PM
To: voiceops at voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now?  My traceroutes aren't even 
leaving Chicago.  Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:
> Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?
>
> -- Nathan
>
> -----Original Message-----
> From: frnkblk at iname.com [mailto:frnkblk at iname.com]
> Sent: Sunday, March 06, 2016 4:43 PM
> To: Nathan Anderson; voiceops at voiceops.org
> Subject: RE: VoIP Innovations reliability
>
> More here: http://blog.voipinnovations.com/blog
>
> Frank
>
> -----Original Message-----
> From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan
> Anderson
> Sent: Tuesday, March 01, 2016 8:27 PM
> To: voiceops at voiceops.org
> Subject: [VoiceOps] VoIP Innovations reliability
>
> Holy schlamoly.  Anybody else use them here and being handed outage after
> outage over the last 2 days?  Seriously thinking at this point about doing
> something else.  This is ridiculous.
>
> I desperately need sleep and if my cell goes off one more time...
>
> -- Nathan
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/voiceops/attachments/20160317/84a94ebd/attachment-0001.html>