<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">I’ll take “Desperately trying to avoid a rush of people filing SLA credit claims” for $500, Alex
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Outages-discussion <outages-discussion-bounces@outages.org>
<b>On Behalf Of </b>Hunt, Fred A - DOA via Outages-discussion<br>
<b>Sent:</b> Tuesday, September 12, 2023 5:17 PM<br>
<b>To:</b> 'Pete Eisengrein' <peeip989@gmail.com><br>
<b>Cc:</b> 'outages-discussion@outages.org' <outages-discussion@outages.org><br>
<b>Subject:</b> [EXTERNAL] Re: [Outages-discussion] [outages-discussion] Microsoft Teams outage?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It took nearly two weeks to squeeze it out of Microsoft:<br>
<br>
“• ‘Yes’ they had deployed the SIP proxy version 2023.8.28.1 on 30th August between 10:30 - 17:15 UTC.<o:p></o:p></p>
<p class="MsoNormal">• For Service health PG confirmed that if there wasn't message until now, then there won't be.”<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Hunt, Fred A - DOA <<a href="mailto:FredA.Hunt@wisconsin.gov">FredA.Hunt@wisconsin.gov</a>>
<br>
<b>Sent:</b> Wednesday, August 30, 2023 4:12 PM<br>
<b>To:</b> Hunt, Fred A - DOA <<a href="mailto:FredA.Hunt@wisconsin.gov">FredA.Hunt@wisconsin.gov</a>>; 'Pete Eisengrein' <<a href="mailto:peeip989@gmail.com">peeip989@gmail.com</a>><br>
<b>Cc:</b> 'outages-discussion@outages.org' <<a href="mailto:outages-discussion@outages.org">outages-discussion@outages.org</a>><br>
<b>Subject:</b> RE: [outages-discussion] Microsoft Teams outage?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The irony<o:p></o:p></p>
<p class="MsoNormal"><img border="0" width="378" height="154" style="width:3.9375in;height:1.6041in" id="Picture_x0020_1" src="cid:image001.png@01D9E59E.FAED9D10"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Outages-discussion <<a href="mailto:outages-discussion-bounces@outages.org">outages-discussion-bounces@outages.org</a>>
<b>On Behalf Of </b>Hunt, Fred A - DOA via Outages-discussion<br>
<b>Sent:</b> Wednesday, August 30, 2023 3:21 PM<br>
<b>To:</b> 'Pete Eisengrein' <<a href="mailto:peeip989@gmail.com">peeip989@gmail.com</a>><br>
<b>Cc:</b> 'outages-discussion@outages.org' <<a href="mailto:outages-discussion@outages.org">outages-discussion@outages.org</a>><br>
<b>Subject:</b> Re: [Outages-discussion] [outages-discussion] Microsoft Teams outage?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><strong><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white;background:#FF6600">CAUTION: This email originated from outside the organization.
</span></strong><br>
<strong><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white;background:#FF6600">Do not click links or open attachments unless you recognize the sender and know the content is safe.</span></strong><o:p></o:p></p>
<p><o:p> </o:p></p>
<p class="MsoNormal">Nice observation. That’s the same as what I see here, after further review. I reached out to a contact at Microsoft earlier, but have not heard back. Am going to pass this along and press for some ownership of the cause.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Pete Eisengrein <<a href="mailto:peeip989@gmail.com">peeip989@gmail.com</a>>
<br>
<b>Sent:</b> Wednesday, August 30, 2023 2:55 PM<br>
<b>To:</b> Hunt, Fred A - DOA <<a href="mailto:FredA.Hunt@wisconsin.gov">FredA.Hunt@wisconsin.gov</a>><br>
<b>Cc:</b> <a href="mailto:outages@voiceops.org">outages@voiceops.org</a><br>
<b>Subject:</b> Re: [outages] Microsoft Teams outage?<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><strong><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white;background:#FF6600">CAUTION: This email originated from outside the organization.
</span></strong><br>
<strong><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white;background:#FF6600">Do not click links or open attachments unless you recognize the sender and know the content is safe.</span></strong><o:p></o:p></p>
<p><o:p> </o:p></p>
<div>
<p class="MsoNormal">I don't believe we have gotten any sort of RCA from them yet but we noted that the User-Agent, while calls were failing, had "v.2023.08.28.1" in it and after the problem cleared the User-Agent had "v.2023.06.29.3.i" in it, so it smells
like a rollback to me.<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Aug 30, 2023 at 1:11 PM Pete Eisengrein <<a href="mailto:peeip989@gmail.com">peeip989@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">It is Direct Routing, but we weren't getting the 400 Bad Request, and for us the BYE was originating from the PSTN towards Microsoft due to no audio. This magically began to clear around noon eastern.<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Aug 30, 2023 at 12:37 PM Hunt, Fred A - DOA <<a href="mailto:FredA.Hunt@wisconsin.gov" target="_blank">FredA.Hunt@wisconsin.gov</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Are you using Teams Direct Routing? This morning it does appear that Teams is not cleanly terminating calls. Here’s an example response from Teams to a SIP BYE message sent<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Courier New";color:black">148249222: Aug 30 15:09:39.986: //12929228/0B60EED0B3ED/CUBE_VT/SIP/Msg/ccsipDisplayMsg:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Courier New";color:black">Received: SIP TLS message from
<a href="http://secure-web.cisco.com/1kABKIdweERcFsWcUUBGyvAmisgiygsYuRC0nyv-GTfYKBW3pxm949BdAiAN3C_8yUAx-QnIz7eb_mjLOOVx8PB3hfdi45FUXQAxFlnTjX-YKUgyoGGr0LOF0ctOjawA15Yzkynjm4iLYdTxtd2743gKvLkgI05zkJNEyfJ_lbuseCBcDCRJFoDObUUxqM9h1IjiNPBrzt9L_ZlJkKbdpgI0H8IeAHiAcjmt_EQR3eItzV0m4jbg9k_5gwMlPG2uynes4MAhTp4B_7ko7umxwZPRnSIpLTRQ0Kzp2VSNMaEEKRoF8no6cfRJo_H2OVLxtKcbuTGuChX8VN5Pd98IgIJyDPpvq2AjzBOpSkItfdxo/http%3A%2F%2F52.114.132.46%3A5061" target="_blank">
52.114.132.46:5061</a> to [REDACTED] </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Courier New";color:black;background:yellow">SIP/2.0 400 Bad Request</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Courier New";color:black">FROM: "[REDACTED]"sip:+[REDACTED]@[REDACTED];tag=59E32C35-1282</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Courier New";color:black">TO:
<a href="sip:+[REDACTED]@sip.pstnhub.microsoft.com;tag=42b61721ea1d4686bf22e4b2f03384d0">
sip:+[REDACTED]@sip.pstnhub.microsoft.com;tag=42b61721ea1d4686bf22e4b2f03384d0</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Courier New";color:black">CSEQ: 102 BYE</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Courier New";color:black">CALL-ID:
<a href="mailto:B629C8A-467E11EE-B3F38F31-9397CD96@TeamsMADSBC.wisconsin.gov" target="_blank">
B629C8A-467E11EE-B3F38F31-9397CD96@[REDACTED]</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Courier New";color:black">VIA: SIP/2.0/TLS [REDACTED];branch=z9hG4bK3B67395A8</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Courier New";color:black;background:yellow">REASON: Q.850;cause=95;text="9e76dc15-53f0-4eb9-82b0-215b7e029e87;Expected a Protocol of Reason
header as a token, got '\5cx17\5cx03\5cx03\5cx03n\5cx8BC^YB C;8t\5cx96C^]BYE <a href="sip:api-du-a-usea.pstnhub.microsoft.com:443;x-i=9e76dc15-53f0-4eb9-82b0-215b7e029e87;x-c=4c4164749a1c57e5958505fe4de5d0cb/s/1/dd4c">
sip:api-du-a-usea.pstnhub.microsoft.com:443;x-i=9e76dc15-53f0-4eb9-82b0-215b7e029e87;x-c=4c4164749a1c57e5958505fe4de5d0cb/s/1/dd4c</a>"</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Courier New";color:black">CONTENT-LENGTH: 0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Courier New";color:black">ALLOW: INVITE,ACK,OPTIONS,CANCEL,BYE,NOTIFY</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Courier New";color:black">SERVER: Microsoft.PSTNHub.SIPProxy v.2023.8.28.1 i.USEA.10</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Nothing has changed with what we are sending to MS in the SIP Reason header and I also see plenty of calls today that do terminate cleanly.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b> Outages <<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>>
<b>On Behalf Of </b>Pete Eisengrein via Outages<br>
<b>Sent:</b> Wednesday, August 30, 2023 10:32 AM<br>
<b>To:</b> <a href="mailto:outages@voiceops.org" target="_blank">outages@voiceops.org</a><br>
<b>Subject:</b> [outages] Microsoft Teams outage?<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p><strong><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white;background:#FF6600">CAUTION: This email originated from outside the organization.
</span></strong><br>
<strong><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white;background:#FF6600">Do not click links or open attachments unless you recognize the sender and know the content is safe.</span></strong><o:p></o:p></p>
<p> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We are experiencing an issue with Microsoft this morning, where they initially connect but when we send a re-INVITE, Microsoft responds with a 100 Trying but then never updates
the session and the call is left with dead air and eventually the caller hangs up.<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Waiting on Microsoft support.....<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
<p><span style="font-size:7.0pt;line-height:107%;font-family:
"Arial",sans-serif">The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party,
without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.</span></p>
</body>
</html>