[VoiceOps] Looking for a good defense for a bad VoIP provider

[Adrian]

Ouch... Sad as it seems, this is more or less standard for a certain segment 
of the VoIP provider market, where they practice a "drop and run" (out the 
door mostly) type of install. At $DAYJOB I use to be primary (now tertiary) 
for a ~1K deployed VoIP extensions. We typically provide data and phone with 
managed routers onsite, but in cases where there is an existing IT department 
we will work with them well ahead of time to: 1) plan a roll-out schedule, and 
2) establish a separate voice VLAN and external IP (preferably with our router 
inbetween) that is separate from the rest of their data. It's rare to find in 
house talent that can properly configure separate DHCP scopes/VLANs and handle 
VoIP prioritization.

Due to the diversity of ways different manufacturers config their phones, 
building a standard install tool can be a real pain as well. We have had to do 
a whole lot of work to build custom config tools to do pre-install 
(firmware/update/initial config sources) and post-install (user config) mostly 
automagically. Still, some things require manually hitting the phone's web 
interface. Then comes the maintenance of the config files/firmware and whether 
those a only saved on the phone locally or on a remote server. I'll respond to 
a couple things below as to my experience... take from it what you will.

On Friday 25 March 2016 16:55:58 Aaron C. de Bruyn wrote:
... 
> This VoIP provider walked in to several off the offices and just started
> yanking out switches that had various VLANs running on them and replacing
> them with their own Netgear PoE switches with no config and default
> passwords.  

Typical for a certain segment of the market... We've had existing customers 
who decided to switch to someone else without notice either: port out the 
numbers in the middle of the business day when phones are not even installed 
or their backend configured to accept the calls or, as you had, show up and 
start swapping stuff without any planning.


> * We set up VPN connections.  The VoIP guys aren't using them.  They don't
> have time to test and/or troubleshoot any issues they are complaining
>  about.

*If* you are dropping them straight into the voice VLAN with direct 
communication with the phones, with a standard codec (PPTP/IPSEC/etc), yeah 
this should be a red flag they are not prepared.


> * The devices all have static IPs instead of using DHCP.  The phones appear
> to get a DHCP address off VLAN 100 properly, but when it's time for a
> renewal they drop the VLAN tag, get switched to the wrong network, and lose
> communication.

This can be a real pain... In some phones it can be real hard to determine if 
the phone is using network setting received from DHCP, from the phone's config 
file, or the phones web/keypad interface, particullarly if you are having the 
phone DHCP from an untagged network, get a VLAN option and jump to that, then 
DHCP again for the local network. Some phones have non-standard DHCP VLAN 
options that need to be given in the first and/or second stage. Additionally, 
some manufacturers have specific DHCP provisioning options that must be given 
after switching to the VLAN.

The above can get even more complicated if the phone is also doing 
tagging/untagging/pass-thru of data to a computer. Here is a big gotcha.... If 
you switch the "network" (to wall jack) and "computer" (pass-thru to computer) 
ports on the phone, a bunch of different models will DHCP/option correctly but 
then fail to register or fail to connect later. Yet the phone/computer will 
think the network connection is fine.

If the provider can not explicitly give you the required DHCP options at each 
stage without hesitation, I would try to force the issue with the provider. 
Either have them hard code the VLAN or move all phones to untagged switch 
ports on a separate VLAN segment.


> * We were told to set up port forwards to every phone's *HTTP* interface as
> well as a forward to the phone server HTTPS interface.

As above... this is sadly true for a lot of phones, though I suspect it is so 
they can remotely full-configure the phone in this case, not make minor 
extension specific changes. In our setup, we use dynamically-created, time-
limited, port forwards and filter rules that allow our only our management 
address to communicate with specific phones. (I.e. A tech hits the web tool to 
create a link and connect to phone 3, port 8000 is dynamically forwarded to 
phone 3 port 80 in the router, then after 60min the forward expires, multiple 
forward are created sequentially.) But that requires that one of our 
management routers is between the world and the voice VLAN.


> * Most passwords appear to be factory defaults

Not that unusual...<pet-peave>I don;t care what security best practice says, 
having non-default passwords on fixed internal equipment is a problem. If you 
leave it defaulted you can always update/change configs as needed, the internal 
network should always be isolated and never be exposed to external sources 
anyways. If you set all the phones (across many sites) to your "secret" 
password, it quickly becomes well known anyways. If you set good passwords on 
every phone individually, the passwords will be lost and you will have to wipe 
everything and work from scratch. Additionally, depending on the sales model, 
the customer may "own" the phone, so having the third party lock the customer 
out may be a problem.

The phones should never be directly exposed in the first place, only allowed to 
communicate with the PBX and a management addresses as needed. There are 
already so many phone exploits... If an attacker is on your internal network 
with the phones, or the phone is exposed to the outside, a config password 
isn't going to stop them.</pet-peave>


> * They skimped on proper wiring in a lot of places and have computers
> jumpered through the phones
>
> * Because of that, the phones are self-tagging packets with VLAN 100 and
> the jumpered workstations are un-tagged which required us to accept
> un-tagged packets on to the network containing patient data.
...
> * If the phones or phone server gets compromised, it seems like it would be
> real easy to simply drop the VLAN tag and have access to a network
> containing patient data.

Yep... That is why we always specify separate drops (or we are directly 
managing/monitoring) for phone/data. In the cases where you have to use pass-
thru (lack of infrastructure/port availability), there is not a lot that can 
be done. Locking phone VLANs and then setting port MAC security on your 
switches may be the only option. A portion of why third-party VoIP is less 
expensive is because the customer is expected to handle/reuse existing 
infrastructure wiring.


> * A quick sniff at our WAN interface shows all the calls and communication
> are happening with a server over HTTP.  I was able to capture voice data in
> the clear containing patient information, credit card details, etc...

Yeah, but you could do the same on the PSTN as well, which the same calls will 
travel. 


> Basically the phone guys are blaming us for all the problems, and we are
> blaming them for causing several thousand dollars in after-hours emergency
> site visits and remote work because of poor planning, scheduling, and
> simply ripping out equipment they know nothing about.  (In addition to
> making the network insecure as hell and not doing their due diligence.)

Keep blaming them. Their poor planning and setup is the cause of the downtime. 
If it comes to working but poor-voice, well then you my have to start looking 
internally. Properly handling/queuing voice traffic in parallel with data can be 
a real challenge.


Adrian