[VoiceOps] Looking for a good defense for a bad VoIP provider

[Rafael Possamai]

I learned this the hard way, but now I only continue to service customers
if they actually understand the value brought by a secure and reliable
service. As soon as they want to cut corners, for one reason or another,
and that becomes a huge issue (like in your case), I make use of a
"termination for convenience" clause that I have in all my contracts and
within 90 days I no longer have to service this customer, they can find
some other provider to make their lives hell.

No more servicing every customer with insane expectations of cost vs
benefit.

*Rafael Possamai*
Founder & CEO at E2W Solutions
*office:* (414) 269-6000
*e-mail:* rafael at e2wsolutions.com


On Fri, Mar 25, 2016 at 6:55 PM, Aaron C. de Bruyn <aaron at heyaaron.com>
wrote:

> I'll try to make this short:  I am an IT contractor for "Company X" that
> has ~26 offices around the western US.  We are paid a flat fee to manage
> every office, keep things secure, train and assist users, etc...
>
> About two weeks ago, offices suddenly started going offline.  After 15-20
> minutes of frantically digging, we got a call from a VoIP provider who was
> apparently told by Company X "we want to use your VoIP service, go get it
> installed at all our offices".
>
> This VoIP provider walked in to several off the offices and just started
> yanking out switches that had various VLANs running on them and replacing
> them with their own Netgear PoE switches with no config and default
> passwords.  They took down tons of virtual servers and SANs.
>
> We spent the better part of a week ripping their stuff out and putting the
> network back the way it was.
>
> We had a brief meeting with the VoIP provider and told them things need to
> be planned in advance and we would be happy to work with them to get things
> going.
>
> We set up a VLAN for VoIP traffic and told them how to cross-connect their
> switch to ours, what IP ranges to use, and even set up a VPN connection at
> each office so they could access the equipment remotely.
>
> They they scheduled 6 installs in 3 days and with no testing or
> communication they came in and hooked things up.  I received repeated phone
> calls in the evenings, mornings, and weekends that they needed huge swaths
> of ports forwarded so they could remotely program phones and the phone
> server.
>
> And of course it was all an emergency.  As in "we ported the numbers over
> already and the office is opening in 15 minutes, just forward the damn
> ports".
>
> We were even told the 6 installs could not be stopped or re-scheduled
> because the VoIP provider 'went out' on the equipment and really needs to
> finish the install so they can recoup their money.
>
> The disaster should be coming to a close this weekend, and I'm trying to
> clean up and gather information for a report to the CEO.  My main concerns
> are:
>
> * We set up VPN connections.  The VoIP guys aren't using them.  They don't
> have time to test and/or troubleshoot any issues they are complaining about.
>
> * The devices all have static IPs instead of using DHCP.  The phones
> appear to get a DHCP address off VLAN 100 properly, but when it's time for
> a renewal they drop the VLAN tag, get switched to the wrong network, and
> lose communication.
>
> * We were told to set up port forwards to every phone's *HTTP* interface
> as well as a forward to the phone server HTTPS interface.
>
> * Most passwords appear to be factory defaults
>
> * The CEO of Company X was told the port forward must remain in place and
> they can not be disabled for 'security reasons' because VoIP phone are not
> a computer and therefore can't be hacked.  (Seriously?!?!! Who cares about
> hacking when your equipment has default passwords...)
>
> * They skimped on proper wiring in a lot of places and have computers
> jumpered through the phones
>
> * Because of that, the phones are self-tagging packets with VLAN 100 and
> the jumpered workstations are un-tagged which required us to accept
> un-tagged packets on to the network containing patient data.
>
> * If the phones or phone server gets compromised, it seems like it would
> be real easy to simply drop the VLAN tag and have access to a network
> containing patient data.
>
> * A quick sniff at our WAN interface shows all the calls and communication
> are happening with a server over HTTP.  I was able to capture voice data in
> the clear containing patient information, credit card details, etc...
>
> I have worked with professional VoIP companies before.  When they do it
> right, the networks are isolated, phones have their own network drops, no
> ports are exposed to the internet, etc...
>
> The CEO of Company X appears to only have been informed that the offices
> were 'switching to VoIP to save costs' and nothing more.  He is a very
> data-oriented guy and loves technical documentation.  When we make our case
> to him, I'd like to back it up with as much 'best practice references' as
> possible.
>
> Are there any best-practice documents out there I can provide to the CEO?
>
> I know what this provider is doing is horribly wrong and insecure, but the
> CEO is the kind of person who wants documentation from lots of sources to
> back it up.
>
> Basically the phone guys are blaming us for all the problems, and we are
> blaming them for causing several thousand dollars in after-hours emergency
> site visits and remote work because of poor planning, scheduling, and
> simply ripping out equipment they know nothing about.  (In addition to
> making the network insecure as hell and not doing their due diligence.)
>
> Thanks for listening. :)
>
> -A
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/voiceops/attachments/20160328/77edefd5/attachment.html>