[Outages-discussion] [outages] Call Centric Sip outage

[Mitch]

Latest update:

*Oct 2012 DDoS attack status:*
*Updated 10/7*

As of 12AM EST, Sunday October 7th 2012, we have returned our systems to
99.9% availability. Calls should work normally and overall call quality
should be back to normal.

The second attack, which occurred on October 6th 2012, has been fully
suppressed and the first has now been contained. Due to the nature,
aggressiveness, and complexity of the original attack, we are still now and
will continue to thoroughly investigate this matter. Upon the completion of
the updates to our full farm of registrar servers additional resources will
be deployed to assist in the aforementioned investigation.

New code is being written in real-time to further strengthen our network
and these updates are being applied throughout the day as soon as they
satisfy our internal tests. As such, as these servers are restarted some
users may notice call drops in the middle of a call or when ringing (these
types of occurrences should be rare), BUT NOT FOR OUR ENTIRE NETWORK.
Please note that these issues WILL NOT BE DUE TO A NEW ATTACK but to our
engineers rolling out updates as they are applied.

Please keep in mind that this is being done to our registrar servers and
not to our databases, web servers, internet or PSTN interconnects; as these
systems were entirely unaffected.

As we mentioned in previous updates, the intent of this attack was to cause
a severe service disruption. With that said, again we want to stress that
ALL of your information is encrypted, safe and secure; and NO customer data
was stolen NOR destroyed.

The new registration domains will remain in our network as new registrar
servers which can be used by any user. The differences are as follows:

sip.callcentric.com: For software/hardware that *ONLY* supports DNS A record
srv.callcentric.com: For software/hardware that supports DNS SRV records

In addition to the real-time security audit being done, a system wide
security audit will be performed to further strengthen our registrar
servers which were attacked.

Our main priorities at Callcentric remain unchanged; security, performance,
stability, and delivering the best service experience possible for our
end-users.

Needless to say this attack has been humbling and we want to ensure you
that we will learn and grow from this experience and we will use this as an
impetus to further enhance our service, support response, and the overall
security of our network.

We sincerely apologize for any inconvenience experienced as a result of
these attacks and we remain committed to employ all of our available
resources to assist in answering any questions/concerns and getting all of
our customer's services fully restored to the high level that both they/we
expect.

Where necessary, further relevant updates will continue to be posted both
within the My Callcentric Portal and on our Twitter Page (
https://twitter.com/Callcentric). Additionally, we will provide further
information in a full email and notice to our customers early in the
upcoming week.

On Sun, Oct 7, 2012 at 6:00 PM, Mitch <mitpatterson at gmail.com> wrote:

> Doesn't look like it, the time frames don't line up, and the PowerPhone
> one says it was brief, call centric has been on going.
>
>
> On Sun, Oct 7, 2012 at 1:19 PM, Frank Bulk <frnkblk at iname.com> wrote:
>
>> Is is PhonePower using CallCentric?****
>>
>> http://www.newsforshoppers.com/phone-power-experiencing-outage/36710225/*
>> ***
>>
>> ** **
>>
>> Frank****
>>
>> ** **
>>
>> *From:* outages-bounces at outages.org [mailto:outages-bounces at outages.org]
>> *On Behalf Of *Mitch
>> *Sent:* Saturday, October 06, 2012 8:37 PM
>> *To:* Micah Brandon
>> *Cc:* outages at outages.org
>> *Subject:* Re: [outages] Call Centric Sip outage****
>>
>> ** **
>>
>> Latest update(I didn't bother pasting the whole thing):****
>>
>> ** **
>>
>> *Investigation into current problems:*
>> Hello,
>>
>> *UPDATED 10/06 4:00 PM EST*
>>
>> The second attack against our new servers has been suppressed.
>>
>> sip.callcentric.com and srv.callcentric.com should provide better
>> quality and functionality and they are being continually monitored. We are
>> still committed to answering questions and will continue to provide as
>> timely/detailed support as possible.
>>
>> With this in mind, we recommend subscribing/following our official
>> Twitter page (http://www.twitter.com/Callcentric) as we will be posting
>> updates and additional information as available.
>>
>> The original attack is still ongoing and we are returning to this issue
>> in order to attempt to restore normal service across the board. This
>> investigation process involves deep packet inspection and analysis to
>> properly diagnose and prevent any other damage.
>>
>> We sincerely appreciate your patience with us and again apologize for the
>> inconvenience.****
>>
>> ** **
>>
>> On Sat, Oct 6, 2012 at 6:38 PM, Micah Brandon <brandon at netsville.com>
>> wrote:****
>>
>> Well, that's just weak.  Let's publish information for some of our
>> clients.  Yeah, the Internets will never find out...  However, they do
>> appear to have a handle on things right now.  But the botnet is probably at
>> reduced strength over the weekend.  They are going to have a serious case
>> of the Mondays if they don't dial it up and get ahead of this thing.****
>>
>>
>> On 10/06/2012 12:25 PM, Mitch wrote:
>> > Well I guess their work around didn't work(probably didn't help that I
>> was one of the people to publish the new info):
>> >****
>>
>> > *Investigation into current problems:*
>> > Hello,
>> >
>> > *UPDATE*
>> >
>> > Unfortunately, the attacks against us have been modified and we are now
>> experiencing two different types of DDoS attacks. The new servers (
>> sip.callcentric.com <http://sip.callcentric.com> and srv.callcentric.com<
>> http://srv.callcentric.com>) were meant to assist in reducing the load
>> and allowing service to operate for some clients. Unfortunately due to this
>> new information being publicly released, outside of Callcentric, our new
>> servers have also been affected. As such, the work-around that we developed
>> and the information provided yesterday will no longer work.
>> >****
>>
>> ** **
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20121007/19d78788/attachment-0001.html>