[Outages-discussion] [outages-discussion] Microsoft Teams outage?

Tue Sep 12 18:24:51 EDT 2023

Quelle surprise.

Microsoft makes unannounced change, change has major impacts, Microsoft
quietly rolls it back and goes "huh? No, we've changed nothing here, look,
it's all good! Must be you!".

SOP for them. It's almost like they're effectively a monopoly for a
significant percentage of business use, and don't care about their
customers.

On Wed, 13 Sept 2023 at 07:18, Hunt, Fred A - DOA via Outages-discussion <
outages-discussion at outages.org> wrote:

> It took nearly two weeks to squeeze it out of Microsoft:
>
> “•           ‘Yes’ they had deployed the SIP proxy version 2023.8.28.1 on
> 30th August between 10:30 - 17:15 UTC.
>
> •             For Service health PG confirmed that if there wasn't message
> until now, then there won't be.”
>
>
>
> *From:* Hunt, Fred A - DOA <FredA.Hunt at wisconsin.gov>
> *Sent:* Wednesday, August 30, 2023 4:12 PM
> *To:* Hunt, Fred A - DOA <FredA.Hunt at wisconsin.gov>; 'Pete Eisengrein' <
> peeip989 at gmail.com>
> *Cc:* 'outages-discussion at outages.org' <outages-discussion at outages.org>
> *Subject:* RE: [outages-discussion] Microsoft Teams outage?
>
>
>
> The irony
>
>
>
> *From:* Outages-discussion <outages-discussion-bounces at outages.org> *On
> Behalf Of *Hunt, Fred A - DOA via Outages-discussion
> *Sent:* Wednesday, August 30, 2023 3:21 PM
> *To:* 'Pete Eisengrein' <peeip989 at gmail.com>
> *Cc:* 'outages-discussion at outages.org' <outages-discussion at outages.org>
> *Subject:* Re: [Outages-discussion] [outages-discussion] Microsoft Teams
> outage?
>
>
>
> *CAUTION: This email originated from outside the organization. *
> *Do not click links or open attachments unless you recognize the sender
> and know the content is safe.*
>
>
>
> Nice observation. That’s the same as what I see here, after further
> review. I reached out to a contact at Microsoft earlier, but have not heard
> back. Am going to pass this along and press for some ownership of the cause.
>
>
>
> *From:* Pete Eisengrein <peeip989 at gmail.com>
> *Sent:* Wednesday, August 30, 2023 2:55 PM
> *To:* Hunt, Fred A - DOA <FredA.Hunt at wisconsin.gov>
> *Cc:* outages at voiceops.org
> *Subject:* Re: [outages] Microsoft Teams outage?
>
>
>
> *CAUTION: This email originated from outside the organization. *
> *Do not click links or open attachments unless you recognize the sender
> and know the content is safe.*
>
>
>
> I don't believe we have gotten any sort of RCA from them yet but we noted
> that the User-Agent, while calls were failing, had "v.2023.08.28.1" in it
> and after the problem cleared the User-Agent had "v.2023.06.29.3.i" in it,
> so it smells like a rollback to me.
>
>
>
> On Wed, Aug 30, 2023 at 1:11 PM Pete Eisengrein <peeip989 at gmail.com>
> wrote:
>
> It is Direct Routing, but we weren't getting the 400 Bad Request, and for
> us the BYE was originating from the PSTN towards Microsoft due to no audio.
> This magically began to clear around noon eastern.
>
>
>
> On Wed, Aug 30, 2023 at 12:37 PM Hunt, Fred A - DOA <
> FredA.Hunt at wisconsin.gov> wrote:
>
> Are you using Teams Direct Routing? This morning it does appear that Teams
> is not cleanly terminating calls. Here’s an example response from Teams to
> a SIP BYE message sent
>
>
>
> 148249222: Aug 30 15:09:39.986:
> //12929228/0B60EED0B3ED/CUBE_VT/SIP/Msg/ccsipDisplayMsg:
>
> Received: SIP TLS message from 52.114.132.46:5061
> <http://secure-web.cisco.com/1kABKIdweERcFsWcUUBGyvAmisgiygsYuRC0nyv-GTfYKBW3pxm949BdAiAN3C_8yUAx-QnIz7eb_mjLOOVx8PB3hfdi45FUXQAxFlnTjX-YKUgyoGGr0LOF0ctOjawA15Yzkynjm4iLYdTxtd2743gKvLkgI05zkJNEyfJ_lbuseCBcDCRJFoDObUUxqM9h1IjiNPBrzt9L_ZlJkKbdpgI0H8IeAHiAcjmt_EQR3eItzV0m4jbg9k_5gwMlPG2uynes4MAhTp4B_7ko7umxwZPRnSIpLTRQ0Kzp2VSNMaEEKRoF8no6cfRJo_H2OVLxtKcbuTGuChX8VN5Pd98IgIJyDPpvq2AjzBOpSkItfdxo/http%3A%2F%2F52.114.132.46%3A5061>
> to [REDACTED]
>
> SIP/2.0 400 Bad Request
>
> FROM: "[REDACTED]"sip:+[REDACTED]@[REDACTED];tag=59E32C35-1282
>
> TO:
> sip:+[REDACTED]@sip.pstnhub.microsoft.com;tag=42b61721ea1d4686bf22e4b2f03384d0
>
> CSEQ: 102 BYE
>
> CALL-ID: B629C8A-467E11EE-B3F38F31-9397CD96@[REDACTED]
> <B629C8A-467E11EE-B3F38F31-9397CD96 at TeamsMADSBC.wisconsin.gov>
>
> VIA: SIP/2.0/TLS [REDACTED];branch=z9hG4bK3B67395A8
>
> REASON: Q.850;cause=95;text="9e76dc15-53f0-4eb9-82b0-215b7e029e87;Expected
> a Protocol of Reason header as a token, got
> '\5cx17\5cx03\5cx03\5cx03n\5cx8BC^YB C;8t\5cx96C^]BYE
> sip:api-du-a-usea.pstnhub.microsoft.com:443;x-i=9e76dc15-53f0-4eb9-82b0-215b7e029e87;x-c=4c4164749a1c57e5958505fe4de5d0cb/s/1/dd4c
> "
>
> CONTENT-LENGTH: 0
>
> ALLOW: INVITE,ACK,OPTIONS,CANCEL,BYE,NOTIFY
>
> SERVER: Microsoft.PSTNHub.SIPProxy v.2023.8.28.1 i.USEA.10
>
>
>
> Nothing has changed with what we are sending to MS in the SIP Reason
> header and I also see plenty of calls today that do terminate cleanly.
>
>
>
> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Pete
> Eisengrein via Outages
> *Sent:* Wednesday, August 30, 2023 10:32 AM
> *To:* outages at voiceops.org
> *Subject:* [outages] Microsoft Teams outage?
>
>
>
> *CAUTION: This email originated from outside the organization. *
> *Do not click links or open attachments unless you recognize the sender
> and know the content is safe.*
>
>
>
> We are experiencing an issue with Microsoft this morning, where they
> initially connect but when we send a re-INVITE, Microsoft responds with a
> 100 Trying but then never updates the session and the call is left with
> dead air and eventually the caller hangs up.
>
>
>
> Waiting on Microsoft support.....
>
>
>
> Thanks
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>

-- 
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20230913/ddb4d328/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 21931 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20230913/ddb4d328/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 21931 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20230913/ddb4d328/attachment-0003.png>