[VoiceOps] Looking for a good defense for a bad VoIP provider

[Aaron C. de Bruyn]

I'll try to make this short:  I am an IT contractor for "Company X" that
has ~26 offices around the western US.  We are paid a flat fee to manage
every office, keep things secure, train and assist users, etc...

About two weeks ago, offices suddenly started going offline.  After 15-20
minutes of frantically digging, we got a call from a VoIP provider who was
apparently told by Company X "we want to use your VoIP service, go get it
installed at all our offices".

This VoIP provider walked in to several off the offices and just started
yanking out switches that had various VLANs running on them and replacing
them with their own Netgear PoE switches with no config and default
passwords.  They took down tons of virtual servers and SANs.

We spent the better part of a week ripping their stuff out and putting the
network back the way it was.

We had a brief meeting with the VoIP provider and told them things need to
be planned in advance and we would be happy to work with them to get things
going.

We set up a VLAN for VoIP traffic and told them how to cross-connect their
switch to ours, what IP ranges to use, and even set up a VPN connection at
each office so they could access the equipment remotely.

They they scheduled 6 installs in 3 days and with no testing or
communication they came in and hooked things up.  I received repeated phone
calls in the evenings, mornings, and weekends that they needed huge swaths
of ports forwarded so they could remotely program phones and the phone
server.

And of course it was all an emergency.  As in "we ported the numbers over
already and the office is opening in 15 minutes, just forward the damn
ports".

We were even told the 6 installs could not be stopped or re-scheduled
because the VoIP provider 'went out' on the equipment and really needs to
finish the install so they can recoup their money.

The disaster should be coming to a close this weekend, and I'm trying to
clean up and gather information for a report to the CEO.  My main concerns
are:

* We set up VPN connections.  The VoIP guys aren't using them.  They don't
have time to test and/or troubleshoot any issues they are complaining about.

* The devices all have static IPs instead of using DHCP.  The phones appear
to get a DHCP address off VLAN 100 properly, but when it's time for a
renewal they drop the VLAN tag, get switched to the wrong network, and lose
communication.

* We were told to set up port forwards to every phone's *HTTP* interface as
well as a forward to the phone server HTTPS interface.

* Most passwords appear to be factory defaults

* The CEO of Company X was told the port forward must remain in place and
they can not be disabled for 'security reasons' because VoIP phone are not
a computer and therefore can't be hacked.  (Seriously?!?!! Who cares about
hacking when your equipment has default passwords...)

* They skimped on proper wiring in a lot of places and have computers
jumpered through the phones

* Because of that, the phones are self-tagging packets with VLAN 100 and
the jumpered workstations are un-tagged which required us to accept
un-tagged packets on to the network containing patient data.

* If the phones or phone server gets compromised, it seems like it would be
real easy to simply drop the VLAN tag and have access to a network
containing patient data.

* A quick sniff at our WAN interface shows all the calls and communication
are happening with a server over HTTP.  I was able to capture voice data in
the clear containing patient information, credit card details, etc...

I have worked with professional VoIP companies before.  When they do it
right, the networks are isolated, phones have their own network drops, no
ports are exposed to the internet, etc...

The CEO of Company X appears to only have been informed that the offices
were 'switching to VoIP to save costs' and nothing more.  He is a very
data-oriented guy and loves technical documentation.  When we make our case
to him, I'd like to back it up with as much 'best practice references' as
possible.

Are there any best-practice documents out there I can provide to the CEO?

I know what this provider is doing is horribly wrong and insecure, but the
CEO is the kind of person who wants documentation from lots of sources to
back it up.

Basically the phone guys are blaming us for all the problems, and we are
blaming them for causing several thousand dollars in after-hours emergency
site visits and remote work because of poor planning, scheduling, and
simply ripping out equipment they know nothing about.  (In addition to
making the network insecure as hell and not doing their due diligence.)

Thanks for listening. :)

-A
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/voiceops/attachments/20160325/8ee81bbd/attachment.html>