<div dir="auto">Sounds like you need to find another provider...?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Mar 27, 2022, 7:10 PM Peter Beckman <<a href="mailto:beckman@angryox.com">beckman@angryox.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Vitelity hasn't posted anything about this, so I will for those that care.<br>
<br>
Short Version: Inbound SMS to all Vitelity DIDs failed between 0150 UTC and<br>
1720 UTC on March 27, 2022<br>
<br>
Long Version:<br>
<br>
At 0150 UTC our Inbound SMS Monitoring detected that one of our DIDs did<br>
not receive a sent SMS.<br>
<br>
This continued until 0230 UTC where our system automatically cut a ticket<br>
(via email) to Vitelity that we were seeing 30+ minutes of zero inbound SMS<br>
messages to a random sample of our DIDs.<br>
<br>
At 0305 UTC I opened an emergency ticket about the issue.<br>
<br>
At 0341 UTC The ticket was responded to.<br>
<br>
At 0503 UTC I was able to talk to someone at Vitelity who confirmed that<br>
they also had sent SMS messages to their inbound DIDs and the messages were<br>
not received, and they were escalating. During that, I also provided DLRs<br>
for all the sent messages, showing that they were received by Onvoy,<br>
Iristel, Neutral Tandem, etc.<br>
<br>
Vitelity confirmed that they were esclating and "sounding the alarm."<br>
<br>
At 0810 UTC Vitelity updated the ticket and said they were still<br>
escalating. No change, 100% of SMS messages sent to Vitelity DIDs failed to<br>
reach Vitelity or us.<br>
<br>
At 1634 UTC I re-re-re-escalated the issue to the Inteliquent NOC and my<br>
Vitelity contact. I was informed that a tech had found that "a certificate<br>
had expired" and was causing all inbound SMS delivery to fail.<br>
<br>
Incidentally, this ALSO prevented an emails to Vitelity Support to ALSO not<br>
be delivered to the portal, and thus not open tickets.<br>
<br>
At 1720 UTC a test finally succeeded and we received our first SMS in over<br>
15h 30m. Several tests after that across another sample of Vitelity DIDs<br>
were successful.<br>
<br>
At no point was <a href="http://status.vitelity.com" rel="noreferrer noreferrer" target="_blank">status.vitelity.com</a> actually updated with any sort of<br>
incident, and despite it being acknowledged to me, has still not posted any<br>
sign of the outage.<br>
<br>
<br>
Why I'm Posting<br>
<br>
This is the 48th SMS outage since August 2015 that Vitelity has<br>
experienced, and in most cases, I have been the first person to alert them<br>
to a problem. They do not monitor inbound SMS, they just wait for a<br>
customer complaint, and even then, I've been told "nobody else has opened a<br>
ticket," as if multiple customers need to complain before they take an<br>
outage seriously.<br>
<br>
In this case, it took 2 hours from opening an emergency ticket for Vitelity<br>
to actually test inbound SMS and confirm an issue, followed by another 12<br>
hours to actually find the root cause and fix it.<br>
<br>
Beckman<br>
---------------------------------------------------------------------------<br>
Peter Beckman Internet Guy<br>
<a href="mailto:beckman@angryox.com" target="_blank" rel="noreferrer">beckman@angryox.com</a> <a href="https://www.angryox.com/" rel="noreferrer noreferrer" target="_blank">https://www.angryox.com/</a><br>
---------------------------------------------------------------------------<br>
_______________________________________________<br>
VoiceOps mailing list<br>
<a href="mailto:VoiceOps@voiceops.org" target="_blank" rel="noreferrer">VoiceOps@voiceops.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/voiceops" rel="noreferrer noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/voiceops</a><br>
</blockquote></div>