[outages] Vitelity / Voyant Colorado Outage

[Peter Beckman]

START: 02-09-2020 20:31:39 UTC (~3:31pm EST)
END:   02-09-2020 23:40:47 UTC (~6:41pm EST)

Thanks Mel -- we're waiting on an RFO from Vitelity/Voyant.

Services recovered approximately 3 hours and 10 minutes after we initially
detected the network issues. Round Trip Times dropped from 50% packet loss
and 375ms RTT back down to 68ms RTT with 0% packet loss.

I'll be curious if the CenturyLink issue is related, though our route is
via Zayo or Above.net, and I don't know how they might be related.

Cheers, fellow Beckman :-)

On Mon, 10 Feb 2020, Mel Beckman wrote:

> This may not be related, but on CenturyLink our gigabit fiber BPG peering
> point router is experiencing random high CPU surges that we’ve isolated
> to traffic arriving on this link. We haven’t identified the specific
> traffic yet because the surges are intermittent, lasting just a minute or
> tow. This began at 7:30AM PT and we’ve had about a dozen events in the
> last nine hours.
>
> -mel beckman
>
>> On Feb 9, 2020, at 1:50 PM, Peter Beckman via Outages <outages at outages.org> wrote:
>>
>> START: 02-09-2020 20:31:39 UTC
>> END: ongoing
>>
>> Vitelity is experiencing some sort of network issue. While the Voyant NOC
>> says they are aware, their portal is unstable due to the 40-50% packet loss
>> and jump from 40-50ms to 375ms round trip times.
>>
>> This is affecting all services AFAICT: API endpoints, Portal, incoming and
>> outgoing calls.
>>
>> We have moved most of our traffic away from Vitelity, except for phone
>> numbers they host, as there is no ETA and 375ms plus 40-50% packet loss
>> means pretty terrible call quality.
>>
>> I'll update as I learn more.

---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
beckman at angryox.com                                 http://www.angryox.com/
---------------------------------------------------------------------------