[Outages-discussion] [EXTERNAL] Re: [outages-discussion] Microsoft Teams outage?

[Christopher Conley]

I’ll take “Desperately trying to avoid a rush of people filing SLA credit claims” for $500, Alex

From: Outages-discussion <outages-discussion-bounces at outages.org> On Behalf Of Hunt, Fred A - DOA via Outages-discussion
Sent: Tuesday, September 12, 2023 5:17 PM
To: 'Pete Eisengrein' <peeip989 at gmail.com>
Cc: 'outages-discussion at outages.org' <outages-discussion at outages.org>
Subject: [EXTERNAL] Re: [Outages-discussion] [outages-discussion] Microsoft Teams outage?

It took nearly two weeks to squeeze it out of Microsoft:

“•           ‘Yes’ they had deployed the SIP proxy version 2023.8.28.1 on 30th August between 10:30 - 17:15 UTC.
•             For Service health PG confirmed that if there wasn't message until now, then there won't be.”

From: Hunt, Fred A - DOA <FredA.Hunt at wisconsin.gov<mailto:FredA.Hunt at wisconsin.gov>>
Sent: Wednesday, August 30, 2023 4:12 PM
To: Hunt, Fred A - DOA <FredA.Hunt at wisconsin.gov<mailto:FredA.Hunt at wisconsin.gov>>; 'Pete Eisengrein' <peeip989 at gmail.com<mailto:peeip989 at gmail.com>>
Cc: 'outages-discussion at outages.org' <outages-discussion at outages.org<mailto:outages-discussion at outages.org>>
Subject: RE: [outages-discussion] Microsoft Teams outage?

The irony
[cid:image001.png at 01D9E59E.FAED9D10]

From: Outages-discussion <outages-discussion-bounces at outages.org<mailto:outages-discussion-bounces at outages.org>> On Behalf Of Hunt, Fred A - DOA via Outages-discussion
Sent: Wednesday, August 30, 2023 3:21 PM
To: 'Pete Eisengrein' <peeip989 at gmail.com<mailto:peeip989 at gmail.com>>
Cc: 'outages-discussion at outages.org' <outages-discussion at outages.org<mailto:outages-discussion at outages.org>>
Subject: Re: [Outages-discussion] [outages-discussion] Microsoft Teams outage?


CAUTION: This email originated from outside the organization.
Do not click links or open attachments unless you recognize the sender and know the content is safe.


Nice observation. That’s the same as what I see here, after further review. I reached out to a contact at Microsoft earlier, but have not heard back. Am going to pass this along and press for some ownership of the cause.

From: Pete Eisengrein <peeip989 at gmail.com<mailto:peeip989 at gmail.com>>
Sent: Wednesday, August 30, 2023 2:55 PM
To: Hunt, Fred A - DOA <FredA.Hunt at wisconsin.gov<mailto:FredA.Hunt at wisconsin.gov>>
Cc: outages at voiceops.org<mailto:outages at voiceops.org>
Subject: Re: [outages] Microsoft Teams outage?


CAUTION: This email originated from outside the organization.
Do not click links or open attachments unless you recognize the sender and know the content is safe.


I don't believe we have gotten any sort of RCA from them yet but we noted that the User-Agent, while calls were failing, had "v.2023.08.28.1" in it and after the problem cleared the User-Agent had "v.2023.06.29.3.i" in it, so it smells like a rollback to me.

On Wed, Aug 30, 2023 at 1:11 PM Pete Eisengrein <peeip989 at gmail.com<mailto:peeip989 at gmail.com>> wrote:
It is Direct Routing, but we weren't getting the 400 Bad Request, and for us the BYE was originating from the PSTN towards Microsoft due to no audio. This magically began to clear around noon eastern.

On Wed, Aug 30, 2023 at 12:37 PM Hunt, Fred A - DOA <FredA.Hunt at wisconsin.gov<mailto:FredA.Hunt at wisconsin.gov>> wrote:
Are you using Teams Direct Routing? This morning it does appear that Teams is not cleanly terminating calls. Here’s an example response from Teams to a SIP BYE message sent

148249222: Aug 30 15:09:39.986: //12929228/0B60EED0B3ED/CUBE_VT/SIP/Msg/ccsipDisplayMsg:
Received: SIP TLS message from 52.114.132.46:5061<http://secure-web.cisco.com/1kABKIdweERcFsWcUUBGyvAmisgiygsYuRC0nyv-GTfYKBW3pxm949BdAiAN3C_8yUAx-QnIz7eb_mjLOOVx8PB3hfdi45FUXQAxFlnTjX-YKUgyoGGr0LOF0ctOjawA15Yzkynjm4iLYdTxtd2743gKvLkgI05zkJNEyfJ_lbuseCBcDCRJFoDObUUxqM9h1IjiNPBrzt9L_ZlJkKbdpgI0H8IeAHiAcjmt_EQR3eItzV0m4jbg9k_5gwMlPG2uynes4MAhTp4B_7ko7umxwZPRnSIpLTRQ0Kzp2VSNMaEEKRoF8no6cfRJo_H2OVLxtKcbuTGuChX8VN5Pd98IgIJyDPpvq2AjzBOpSkItfdxo/http%3A%2F%2F52.114.132.46%3A5061> to [REDACTED]
SIP/2.0 400 Bad Request
FROM: "[REDACTED]"sip:+[REDACTED]@[REDACTED];tag=59E32C35-1282
TO: sip:+[REDACTED]@sip.pstnhub.microsoft.com;tag=42b61721ea1d4686bf22e4b2f03384d0
CSEQ: 102 BYE
CALL-ID: B629C8A-467E11EE-B3F38F31-9397CD96@[REDACTED]<mailto:B629C8A-467E11EE-B3F38F31-9397CD96 at TeamsMADSBC.wisconsin.gov>
VIA: SIP/2.0/TLS [REDACTED];branch=z9hG4bK3B67395A8
REASON: Q.850;cause=95;text="9e76dc15-53f0-4eb9-82b0-215b7e029e87;Expected a Protocol of Reason header as a token, got '\5cx17\5cx03\5cx03\5cx03n\5cx8BC^YB C;8t\5cx96C^]BYE sip:api-du-a-usea.pstnhub.microsoft.com:443;x-i=9e76dc15-53f0-4eb9-82b0-215b7e029e87;x-c=4c4164749a1c57e5958505fe4de5d0cb/s/1/dd4c"
CONTENT-LENGTH: 0
ALLOW: INVITE,ACK,OPTIONS,CANCEL,BYE,NOTIFY
SERVER: Microsoft.PSTNHub.SIPProxy v.2023.8.28.1 i.USEA.10

Nothing has changed with what we are sending to MS in the SIP Reason header and I also see plenty of calls today that do terminate cleanly.

From: Outages <outages-bounces at outages.org<mailto:outages-bounces at outages.org>> On Behalf Of Pete Eisengrein via Outages
Sent: Wednesday, August 30, 2023 10:32 AM
To: outages at voiceops.org<mailto:outages at voiceops.org>
Subject: [outages] Microsoft Teams outage?


CAUTION: This email originated from outside the organization.
Do not click links or open attachments unless you recognize the sender and know the content is safe.


We are experiencing an issue with Microsoft this morning, where they initially connect but when we send a re-INVITE, Microsoft responds with a 100 Trying but then never updates the session and the call is left with dead air and eventually the caller hangs up.

Waiting on Microsoft support.....

Thanks

The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20230912/65b65a5b/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 21931 bytes
Desc: image001.png
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20230912/65b65a5b/attachment-0001.png>