[VoiceOps] Looking for a good defense for a bad VoIP provider

Fri Mar 25 20:25:34 EDT 2016

We had a customer who removed their router and connected all the phones directly. Their ISP was happy to hand out individual public IP addresses to each phone. Immediately people started attacking the phones, reprogramming them, and attempting to make phone expensive phone calls. We have blocking for anything really expensive so they didn't get anywhere but it was still a real mess for the customer.

Sent from my iPhone

> On Mar 25, 2016, at 5:15 PM, Peter E <peeip989 at gmail.com> wrote:
> 
> I'm not telling you anything you don't already know, but if the web interfaces are exposed, especially with default passwords, they are going to get hacked, likely this weekend, so you might want to prepare the CEO now for the "savings" he's going to see on his first bill.
> 
> 
> 
> On Mar 25, 2016, at 20:10, Peter E <peeip989 at gmail.com> wrote:
> 
> Ouch, that was painful to read. I can't imagine having lived it. 
> 
> When in doubt, dig for Cisco documents because CxO's all know and trust the brand. I'll forward whatever I can find.
> 
> On Mar 25, 2016, at 19:55, Aaron C. de Bruyn <aaron at heyaaron.com> wrote:
> 
> I'll try to make this short:  I am an IT contractor for "Company X" that has ~26 offices around the western US.  We are paid a flat fee to manage every office, keep things secure, train and assist users, etc...
> 
> About two weeks ago, offices suddenly started going offline.  After 15-20 minutes of frantically digging, we got a call from a VoIP provider who was apparently told by Company X "we want to use your VoIP service, go get it installed at all our offices".
> 
> This VoIP provider walked in to several off the offices and just started yanking out switches that had various VLANs running on them and replacing them with their own Netgear PoE switches with no config and default passwords.  They took down tons of virtual servers and SANs.
> 
> We spent the better part of a week ripping their stuff out and putting the network back the way it was.
> 
> We had a brief meeting with the VoIP provider and told them things need to be planned in advance and we would be happy to work with them to get things going.
> 
> We set up a VLAN for VoIP traffic and told them how to cross-connect their switch to ours, what IP ranges to use, and even set up a VPN connection at each office so they could access the equipment remotely.
> 
> They they scheduled 6 installs in 3 days and with no testing or communication they came in and hooked things up.  I received repeated phone calls in the evenings, mornings, and weekends that they needed huge swaths of ports forwarded so they could remotely program phones and the phone server.
> 
> And of course it was all an emergency.  As in "we ported the numbers over already and the office is opening in 15 minutes, just forward the damn ports".
> 
> We were even told the 6 installs could not be stopped or re-scheduled because the VoIP provider 'went out' on the equipment and really needs to finish the install so they can recoup their money.
> 
> The disaster should be coming to a close this weekend, and I'm trying to clean up and gather information for a report to the CEO.  My main concerns are:
> 
> * We set up VPN connections.  The VoIP guys aren't using them.  They don't have time to test and/or troubleshoot any issues they are complaining about.
> 
> * The devices all have static IPs instead of using DHCP.  The phones appear to get a DHCP address off VLAN 100 properly, but when it's time for a renewal they drop the VLAN tag, get switched to the wrong network, and lose communication.
> 
> * We were told to set up port forwards to every phone's *HTTP* interface as well as a forward to the phone server HTTPS interface.
> 
> * Most passwords appear to be factory defaults
> 
> * The CEO of Company X was told the port forward must remain in place and they can not be disabled for 'security reasons' because VoIP phone are not a computer and therefore can't be hacked.  (Seriously?!?!! Who cares about hacking when your equipment has default passwords...)
> 
> * They skimped on proper wiring in a lot of places and have computers jumpered through the phones
> 
> * Because of that, the phones are self-tagging packets with VLAN 100 and the jumpered workstations are un-tagged which required us to accept un-tagged packets on to the network containing patient data.
> 
> * If the phones or phone server gets compromised, it seems like it would be real easy to simply drop the VLAN tag and have access to a network containing patient data.
> 
> * A quick sniff at our WAN interface shows all the calls and communication are happening with a server over HTTP.  I was able to capture voice data in the clear containing patient information, credit card details, etc...
> 
> I have worked with professional VoIP companies before.  When they do it right, the networks are isolated, phones have their own network drops, no ports are exposed to the internet, etc...
> 
> The CEO of Company X appears to only have been informed that the offices were 'switching to VoIP to save costs' and nothing more.  He is a very data-oriented guy and loves technical documentation.  When we make our case to him, I'd like to back it up with as much 'best practice references' as possible.
> 
> Are there any best-practice documents out there I can provide to the CEO?  
> 
> I know what this provider is doing is horribly wrong and insecure, but the CEO is the kind of person who wants documentation from lots of sources to back it up.
> 
> Basically the phone guys are blaming us for all the problems, and we are blaming them for causing several thousand dollars in after-hours emergency site visits and remote work because of poor planning, scheduling, and simply ripping out equipment they know nothing about.  (In addition to making the network insecure as hell and not doing their due diligence.)
> 
> Thanks for listening. :)
> 
> -A
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops