[VoiceOps] Vitelity Inbound SMS Outage March 27, 2022 0150 to 1720 UTC

[Ross Tajvar]

Sounds like you need to find another provider...?

On Sun, Mar 27, 2022, 7:10 PM Peter Beckman <beckman at angryox.com> wrote:

> Vitelity hasn't posted anything about this, so I will for those that care.
>
> Short Version: Inbound SMS to all Vitelity DIDs failed between 0150 UTC and
> 1720 UTC on March 27, 2022
>
> Long Version:
>
> At 0150 UTC our Inbound SMS Monitoring detected that one of our DIDs did
> not receive a sent SMS.
>
> This continued until 0230 UTC where our system automatically cut a ticket
> (via email) to Vitelity that we were seeing 30+ minutes of zero inbound SMS
> messages to a random sample of our DIDs.
>
> At 0305 UTC I opened an emergency ticket about the issue.
>
> At 0341 UTC The ticket was responded to.
>
> At 0503 UTC I was able to talk to someone at Vitelity who confirmed that
> they also had sent SMS messages to their inbound DIDs and the messages were
> not received, and they were escalating. During that, I also provided DLRs
> for all the sent messages, showing that they were received by Onvoy,
> Iristel, Neutral Tandem, etc.
>
> Vitelity confirmed that they were esclating and "sounding the alarm."
>
> At 0810 UTC Vitelity updated the ticket and said they were still
> escalating. No change, 100% of SMS messages sent to Vitelity DIDs failed to
> reach Vitelity or us.
>
> At 1634 UTC I re-re-re-escalated the issue to the Inteliquent NOC and my
> Vitelity contact. I was informed that a tech had found that "a certificate
> had expired" and was causing all inbound SMS delivery to fail.
>
> Incidentally, this ALSO prevented an emails to Vitelity Support to ALSO not
> be delivered to the portal, and thus not open tickets.
>
> At 1720 UTC a test finally succeeded and we received our first SMS in over
> 15h 30m. Several tests after that across another sample of Vitelity DIDs
> were successful.
>
> At no point was status.vitelity.com actually updated with any sort of
> incident, and despite it being acknowledged to me, has still not posted any
> sign of the outage.
>
>
> Why I'm Posting
>
> This is the 48th SMS outage since August 2015 that Vitelity has
> experienced, and in most cases, I have been the first person to alert them
> to a problem. They do not monitor inbound SMS, they just wait for a
> customer complaint, and even then, I've been told "nobody else has opened a
> ticket," as if multiple customers need to complain before they take an
> outage seriously.
>
> In this case, it took 2 hours from opening an emergency ticket for Vitelity
> to actually test inbound SMS and confirm an issue, followed by another 12
> hours to actually find the root cause and fix it.
>
> Beckman
> ---------------------------------------------------------------------------
> Peter Beckman                                                  Internet Guy
> beckman at angryox.com
> https://www.angryox.com/
> ---------------------------------------------------------------------------
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/voiceops/attachments/20220327/e632f144/attachment.htm>