<div dir="ltr"><div>Quelle surprise.</div><div><br></div><div>Microsoft makes unannounced change, change has major impacts, Microsoft quietly rolls it back and goes "huh? No, we've changed nothing here, look, it's all good! Must be you!".</div><div><br></div><div>SOP for them. It's almost like they're effectively a monopoly for a significant percentage of business use, and don't care about their customers.<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 13 Sept 2023 at 07:18, Hunt, Fred A - DOA via Outages-discussion <<a href="mailto:outages-discussion@outages.org">outages-discussion@outages.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg7206636586068682306">
<u></u><u></u>
<div style="overflow-wrap: break-word;" lang="EN-US">
<div class="m_7206636586068682306WordSection1">
<p class="MsoNormal">It took nearly two weeks to squeeze it out of Microsoft:<br>
<br>
“• ‘Yes’ they had deployed the SIP proxy version 2023.8.28.1 on 30th August between 10:30 - 17:15 UTC.<u></u><u></u></p>
<p class="MsoNormal">• For Service health PG confirmed that if there wasn't message until now, then there won't be.”<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Hunt, Fred A - DOA <<a href="mailto:FredA.Hunt@wisconsin.gov" target="_blank">FredA.Hunt@wisconsin.gov</a>> <br>
<b>Sent:</b> Wednesday, August 30, 2023 4:12 PM<br>
<b>To:</b> Hunt, Fred A - DOA <<a href="mailto:FredA.Hunt@wisconsin.gov" target="_blank">FredA.Hunt@wisconsin.gov</a>>; 'Pete Eisengrein' <<a href="mailto:peeip989@gmail.com" target="_blank">peeip989@gmail.com</a>><br>
<b>Cc:</b> '<a href="mailto:outages-discussion@outages.org" target="_blank">outages-discussion@outages.org</a>' <<a href="mailto:outages-discussion@outages.org" target="_blank">outages-discussion@outages.org</a>><br>
<b>Subject:</b> RE: [outages-discussion] Microsoft Teams outage?<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">The irony<u></u><u></u></p>
<p class="MsoNormal"><img style="width: 3.9416in; height: 1.6083in;" id="m_7206636586068682306Picture_x0020_1" src="cid:image001.png@01D9E594.983C27E0" width="378" height="154"><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Outages-discussion <<a href="mailto:outages-discussion-bounces@outages.org" target="_blank">outages-discussion-bounces@outages.org</a>>
<b>On Behalf Of </b>Hunt, Fred A - DOA via Outages-discussion<br>
<b>Sent:</b> Wednesday, August 30, 2023 3:21 PM<br>
<b>To:</b> 'Pete Eisengrein' <<a href="mailto:peeip989@gmail.com" target="_blank">peeip989@gmail.com</a>><br>
<b>Cc:</b> '<a href="mailto:outages-discussion@outages.org" target="_blank">outages-discussion@outages.org</a>' <<a href="mailto:outages-discussion@outages.org" target="_blank">outages-discussion@outages.org</a>><br>
<b>Subject:</b> Re: [Outages-discussion] [outages-discussion] Microsoft Teams outage?<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p><strong><span style="font-size:10pt;font-family:"Arial",sans-serif;color:white;background:rgb(255,102,0) none repeat scroll 0% 0%">CAUTION: This email originated from outside the organization.
</span></strong><br>
<strong><span style="font-size:10pt;font-family:"Arial",sans-serif;color:white;background:rgb(255,102,0) none repeat scroll 0% 0%">Do not click links or open attachments unless you recognize the sender and know the content is safe.</span></strong><u></u><u></u></p>
<p><u></u> <u></u></p>
<p class="MsoNormal">Nice observation. That’s the same as what I see here, after further review. I reached out to a contact at Microsoft earlier, but have not heard back. Am going to pass this along and press for some ownership of the cause.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Pete Eisengrein <<a href="mailto:peeip989@gmail.com" target="_blank">peeip989@gmail.com</a>>
<br>
<b>Sent:</b> Wednesday, August 30, 2023 2:55 PM<br>
<b>To:</b> Hunt, Fred A - DOA <<a href="mailto:FredA.Hunt@wisconsin.gov" target="_blank">FredA.Hunt@wisconsin.gov</a>><br>
<b>Cc:</b> <a href="mailto:outages@voiceops.org" target="_blank">outages@voiceops.org</a><br>
<b>Subject:</b> Re: [outages] Microsoft Teams outage?<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p><strong><span style="font-size:10pt;font-family:"Arial",sans-serif;color:white;background:rgb(255,102,0) none repeat scroll 0% 0%">CAUTION: This email originated from outside the organization.
</span></strong><br>
<strong><span style="font-size:10pt;font-family:"Arial",sans-serif;color:white;background:rgb(255,102,0) none repeat scroll 0% 0%">Do not click links or open attachments unless you recognize the sender and know the content is safe.</span></strong><u></u><u></u></p>
<p><u></u> <u></u></p>
<div>
<p class="MsoNormal">I don't believe we have gotten any sort of RCA from them yet but we noted that the User-Agent, while calls were failing, had "v.2023.08.28.1" in it and after the problem cleared the User-Agent had "v.2023.06.29.3.i" in it, so it smells
like a rollback to me.<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, Aug 30, 2023 at 1:11 PM Pete Eisengrein <<a href="mailto:peeip989@gmail.com" target="_blank">peeip989@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<div>
<p class="MsoNormal">It is Direct Routing, but we weren't getting the 400 Bad Request, and for us the BYE was originating from the PSTN towards Microsoft due to no audio. This magically began to clear around noon eastern.<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, Aug 30, 2023 at 12:37 PM Hunt, Fred A - DOA <<a href="mailto:FredA.Hunt@wisconsin.gov" target="_blank">FredA.Hunt@wisconsin.gov</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<div>
<div>
<div>
<p class="MsoNormal">Are you using Teams Direct Routing? This morning it does appear that Teams is not cleanly terminating calls. Here’s an example response from Teams to a SIP BYE message sent<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:9pt;font-family:"Courier New";color:black">148249222: Aug 30 15:09:39.986: //12929228/0B60EED0B3ED/CUBE_VT/SIP/Msg/ccsipDisplayMsg:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:9pt;font-family:"Courier New";color:black">Received: SIP TLS message from
<a href="http://secure-web.cisco.com/1kABKIdweERcFsWcUUBGyvAmisgiygsYuRC0nyv-GTfYKBW3pxm949BdAiAN3C_8yUAx-QnIz7eb_mjLOOVx8PB3hfdi45FUXQAxFlnTjX-YKUgyoGGr0LOF0ctOjawA15Yzkynjm4iLYdTxtd2743gKvLkgI05zkJNEyfJ_lbuseCBcDCRJFoDObUUxqM9h1IjiNPBrzt9L_ZlJkKbdpgI0H8IeAHiAcjmt_EQR3eItzV0m4jbg9k_5gwMlPG2uynes4MAhTp4B_7ko7umxwZPRnSIpLTRQ0Kzp2VSNMaEEKRoF8no6cfRJo_H2OVLxtKcbuTGuChX8VN5Pd98IgIJyDPpvq2AjzBOpSkItfdxo/http%3A%2F%2F52.114.132.46%3A5061" target="_blank">
52.114.132.46:5061</a> to [REDACTED] </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:9pt;font-family:"Courier New";color:black;background:yellow none repeat scroll 0% 0%">SIP/2.0 400 Bad Request</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:9pt;font-family:"Courier New";color:black">FROM: "[REDACTED]"sip:+[REDACTED]@[REDACTED];tag=59E32C35-1282</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:9pt;font-family:"Courier New";color:black">TO:
<a>
sip:+[REDACTED]@sip.pstnhub.microsoft.com;tag=42b61721ea1d4686bf22e4b2f03384d0</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:9pt;font-family:"Courier New";color:black">CSEQ: 102 BYE</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:9pt;font-family:"Courier New";color:black">CALL-ID:
<a href="mailto:B629C8A-467E11EE-B3F38F31-9397CD96@TeamsMADSBC.wisconsin.gov" target="_blank">
B629C8A-467E11EE-B3F38F31-9397CD96@[REDACTED]</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:9pt;font-family:"Courier New";color:black">VIA: SIP/2.0/TLS [REDACTED];branch=z9hG4bK3B67395A8</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:9pt;font-family:"Courier New";color:black;background:yellow none repeat scroll 0% 0%">REASON: Q.850;cause=95;text="9e76dc15-53f0-4eb9-82b0-215b7e029e87;Expected a Protocol of Reason
header as a token, got '\5cx17\5cx03\5cx03\5cx03n\5cx8BC^YB C;8t\5cx96C^]BYE <a>
sip:api-du-a-usea.pstnhub.microsoft.com:443;x-i=9e76dc15-53f0-4eb9-82b0-215b7e029e87;x-c=4c4164749a1c57e5958505fe4de5d0cb/s/1/dd4c</a>"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:9pt;font-family:"Courier New";color:black">CONTENT-LENGTH: 0</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:9pt;font-family:"Courier New";color:black">ALLOW: INVITE,ACK,OPTIONS,CANCEL,BYE,NOTIFY</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:9pt;font-family:"Courier New";color:black">SERVER: Microsoft.PSTNHub.SIPProxy v.2023.8.28.1 i.USEA.10</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Nothing has changed with what we are sending to MS in the SIP Reason header and I also see plenty of calls today that do terminate cleanly.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Outages <<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>>
<b>On Behalf Of </b>Pete Eisengrein via Outages<br>
<b>Sent:</b> Wednesday, August 30, 2023 10:32 AM<br>
<b>To:</b> <a href="mailto:outages@voiceops.org" target="_blank">outages@voiceops.org</a><br>
<b>Subject:</b> [outages] Microsoft Teams outage?<u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p><strong><span style="font-size:10pt;font-family:"Arial",sans-serif;color:white;background:rgb(255,102,0) none repeat scroll 0% 0%">CAUTION: This email originated from outside the organization.
</span></strong><br>
<strong><span style="font-size:10pt;font-family:"Arial",sans-serif;color:white;background:rgb(255,102,0) none repeat scroll 0% 0%">Do not click links or open attachments unless you recognize the sender and know the content is safe.</span></strong><u></u><u></u></p>
<p> <u></u><u></u></p>
<div>
<p class="MsoNormal">We are experiencing an issue with Microsoft this morning, where they initially connect but when we send a re-INVITE, Microsoft responds with a 100 Trying but then never updates
the session and the call is left with dead air and eventually the caller hangs up.<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Waiting on Microsoft support.....<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
_______________________________________________<br>
Outages-discussion mailing list<br>
<a href="mailto:Outages-discussion@outages.org" target="_blank">Outages-discussion@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages-discussion" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages-discussion</a><br>
</div></blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">veg·e·tar·i·an:<br>Ancient tribal slang for the village idiot who can't hunt, fish or ride</div>