[VoiceOps] Looking for a good defense for a bad VoIP provider

Aaron C. de Bruyn aaron at heyaaron.com
Sat Mar 26 00:30:51 EDT 2016

Wow--thank you all for your on- and off-list links and advice.  I
appreciate it.

Just to answer a few questions/statements floating around:

I have worked with a lot of 'cowboy' VoIP providers in the last 10 years.
They were all terrible.  But their prices were really good, which is why I
think most decision makers choose them.  They don't have the experience or
knowledge to understand *why* they are so cheap.

The phones aren't configured by DHCP.  DHCP only gives them an IP, Subnet,
Gateway, DNS, and NTP server.

For the curious people who were wondering about the network config:

The network is identical at all the offices.  A decent internet connection
connected to a pfSense router we manage.
The router has a link to the switch that carries tagged VLANs 'office',
'guest-wap', 'staff-wap', and 'VoIP'.

When they installed phones, they brought in a Netgear PoE web-managed
switch and jumpered it over to our switch.  We pass the office VLAN and the
VoIP VLAN to the switch (both are tagged).  On their switch we set the
ports to accept untagged office VLAN and tagged VoIP VLAN for the phone
pass-through connections.  Unfortunately as long as they decide to not wire
up direct runs to the phones, I don't see an easy way to separate them.  If
I force traffic into the VoIP VLAN, the phones can't break out, but the
computers would be on the wrong network.  Meh.

Because the VoIP VLAN requires tagged packets, they set the firmware to tag
the packets.  We have a DHCP server listening in the VoIP VLAN that hands
out addresses, but no configuration information.  They get an IP, Gateway,
DNS, and an NTP server.

When the phones come up, sniffing on the connection between the firewall
and the switch shows incoming DHCP packets with the VoIP VLAN tag.  Once
their lease is about half-over the phone requests an updated lease.  When
it does so, it has the Office VLAN tag (probably because the device dropped
the VoIP VLAN tag, and their switch automatically tags un-tagged packets as

We allow all TCP/UDP traffic outbound from the VoIP network.

According to the provider, we have to forward a bunch of TCP and UDP ports
inbound to the phone server or it won't work.  I tested.  It doesn't work
if ports like 5060 aren't allowed in.  (I thought the 'NAT issue' was
solved a long time ago.  I played with Asterisk years ago and never had
problems with SIP trunk responses getting back through the firewall.)

The VPN we provided them dumps them on to a 192.168.x.x/24 block.  The
third octet is different for each office.  They can then traverse the
router out to the VoIP network with no restrictions.  They claimed the VPN
didn't work because they couldn't talk to the phones.  Turns out the phones
wouldn't renew their DHCP lease and would drop off the VoIP network to the
Office network, and then they would have no contact with the phones.

At some of the offices the phones do stay on the correct VLAN and renew
their leases.  But they aren't able to communicate with the phones through
the VPN or port forwarding.  After about an hour of playing around, I
turned on masquerading.  Turns out the phones had no default gateway or it
was set wrong.  They seem unable to fix the issue and want us to leave the
more-complicated masquerading config in our firewall.  *sigh*

I personally find this whole comedy entertaining and profitable, but I'm
slightly irked that my 'perfect' network setup that was identical at every
site and a snap to manage is now dented and tarnished.  Oh well.

Now I'm going to waste my weekend context switching between writing a
detailed report to the CEO, preparing one more office they decided to
switch tomorrow morning (and told me about ~90 minutes ago), a nice beer,
and a few rounds of Halo.  ;)

Thanks again for all your responses and input.


On Fri, Mar 25, 2016 at 4:55 PM, Aaron C. de Bruyn <aaron at heyaaron.com>

> I'll try to make this short:  I am an IT contractor for "Company X" that
> has ~26 offices around the western US.  We are paid a flat fee to manage
> every office, keep things secure, train and assist users, etc...
> About two weeks ago, offices suddenly started going offline.  After 15-20
> minutes of frantically digging, we got a call from a VoIP provider who was
> apparently told by Company X "we want to use your VoIP service, go get it
> installed at all our offices".
> This VoIP provider walked in to several off the offices and just started
> yanking out switches that had various VLANs running on them and replacing
> them with their own Netgear PoE switches with no config and default
> passwords.  They took down tons of virtual servers and SANs.
> We spent the better part of a week ripping their stuff out and putting the
> network back the way it was.
> We had a brief meeting with the VoIP provider and told them things need to
> be planned in advance and we would be happy to work with them to get things
> going.
> We set up a VLAN for VoIP traffic and told them how to cross-connect their
> switch to ours, what IP ranges to use, and even set up a VPN connection at
> each office so they could access the equipment remotely.
> They they scheduled 6 installs in 3 days and with no testing or
> communication they came in and hooked things up.  I received repeated phone
> calls in the evenings, mornings, and weekends that they needed huge swaths
> of ports forwarded so they could remotely program phones and the phone
> server.
> And of course it was all an emergency.  As in "we ported the numbers over
> already and the office is opening in 15 minutes, just forward the damn
> ports".
> We were even told the 6 installs could not be stopped or re-scheduled
> because the VoIP provider 'went out' on the equipment and really needs to
> finish the install so they can recoup their money.
> The disaster should be coming to a close this weekend, and I'm trying to
> clean up and gather information for a report to the CEO.  My main concerns
> are:
> * We set up VPN connections.  The VoIP guys aren't using them.  They don't
> have time to test and/or troubleshoot any issues they are complaining about.
> * The devices all have static IPs instead of using DHCP.  The phones
> appear to get a DHCP address off VLAN 100 properly, but when it's time for
> a renewal they drop the VLAN tag, get switched to the wrong network, and
> lose communication.
> * We were told to set up port forwards to every phone's *HTTP* interface
> as well as a forward to the phone server HTTPS interface.
> * Most passwords appear to be factory defaults
> * The CEO of Company X was told the port forward must remain in place and
> they can not be disabled for 'security reasons' because VoIP phone are not
> a computer and therefore can't be hacked.  (Seriously?!?!! Who cares about
> hacking when your equipment has default passwords...)
> * They skimped on proper wiring in a lot of places and have computers
> jumpered through the phones
> * Because of that, the phones are self-tagging packets with VLAN 100 and
> the jumpered workstations are un-tagged which required us to accept
> un-tagged packets on to the network containing patient data.
> * If the phones or phone server gets compromised, it seems like it would
> be real easy to simply drop the VLAN tag and have access to a network
> containing patient data.
> * A quick sniff at our WAN interface shows all the calls and communication
> are happening with a server over HTTP.  I was able to capture voice data in
> the clear containing patient information, credit card details, etc...
> I have worked with professional VoIP companies before.  When they do it
> right, the networks are isolated, phones have their own network drops, no
> ports are exposed to the internet, etc...
> The CEO of Company X appears to only have been informed that the offices
> were 'switching to VoIP to save costs' and nothing more.  He is a very
> data-oriented guy and loves technical documentation.  When we make our case
> to him, I'd like to back it up with as much 'best practice references' as
> possible.
> Are there any best-practice documents out there I can provide to the CEO?
> I know what this provider is doing is horribly wrong and insecure, but the
> CEO is the kind of person who wants documentation from lots of sources to
> back it up.
> Basically the phone guys are blaming us for all the problems, and we are
> blaming them for causing several thousand dollars in after-hours emergency
> site visits and remote work because of poor planning, scheduling, and
> simply ripping out equipment they know nothing about.  (In addition to
> making the network insecure as hell and not doing their due diligence.)
> Thanks for listening. :)
> -A
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/voiceops/attachments/20160325/29cdeffc/attachment.html>

More information about the VoiceOps mailing list