<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:inherit;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma",sans-serif;}
span.textexposedshow
{mso-style-name:text_exposed_show;}
span.highlight
{mso-style-name:highlight;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#44546A;}
span.EmailStyle26
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body bgcolor=white lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Yes, more here:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Amazon Web Services: “All but a small single-digit percentage of instances across the Amazon EC2 fleet are already protected. The remaining ones will be completed in the next several hours.”<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Microsoft: “The majority of Azure infrastructure has already been updated to address this vulnerability. Some aspects of Azure are still being updated and require a reboot of customer VMs for the security update to take effect. Many of you have received notification in recent weeks of a planned maintenance on Azure and have already rebooted your VMs to apply the fix, and no further action by you is required. With the public disclosure of the security vulnerability today, we are accelerating the planned maintenance timing and will begin automatically rebooting the remaining impacted VMs starting at 3:30pm PST on January 3, 2018.”<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><a href="https://www.geekwire.com/2018/intel-cloud-vendors-shed-light-two-major-chip-security-issues-emerged-week/">https://www.geekwire.com/2018/intel-cloud-vendors-shed-light-two-major-chip-security-issues-emerged-week/</a><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Frank<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> Joseph Jackson [mailto:jjackson@aninetworks.net] <br><b>Sent:</b> Thursday, January 04, 2018 10:51 AM<br><b>To:</b> Frank Bulk <frnkblk@iname.com>; outages-discussion@outages.org<br><b>Subject:</b> RE: [Outages-discussion] Baicells Outage?<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='color:#44546A'>AWS moved up their reboots as well. Was scheduled for the 6<sup>th</sup> but I woke up this morning to all our EC2 instances being rebooted. <o:p></o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#44546A'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma",sans-serif'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma",sans-serif'> Outages-discussion [<a href="mailto:outages-discussion-bounces@outages.org">mailto:outages-discussion-bounces@outages.org</a>] <b>On Behalf Of </b>Frank Bulk<br><b>Sent:</b> Thursday, January 04, 2018 10:46 AM<br><b>To:</b> <a href="mailto:outages-discussion@outages.org">outages-discussion@outages.org</a><br><b>Subject:</b> Re: [Outages-discussion] Baicells Outage?<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Moving over to -discussion.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The Intel flaw is real.<o:p></o:p></p><p class=MsoNormal>There’s nothing on Azure Status history indicating that there was a problem: <a href="https://azure.microsoft.com/en-us/status/history/">https://azure.microsoft.com/en-us/status/history/</a><o:p></o:p></p><p class=MsoNormal>There were a few Azure issues overnight, but not a lot: <a href="http://downdetector.com/status/windows-azure">http://downdetector.com/status/windows-azure</a>. Here are two note there:<o:p></o:p></p><p class=MsoNormal style='text-indent:.5in'><i>Due to the CPU vulnerability issue Azure VMs are being restarted one after the other. But all of our Azure VMs are very slow during and after the reboot. Some of them cannot be accessed after reboot (only black screen through MSTSC). (Central Europe)<o:p></o:p></i></p><p class=MsoNormal style='text-indent:.5in'><i>Same problem here. VMs are slow booting and are very slow in general. Diskspd tests gave me very low results (sometimes les than 100 iops)<o:p></o:p></i></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>So it’s not clear to me why BaiCells was so affected while it barely created a ripple for others. The group (<a href="https://www.facebook.com/groups/baicellsoperatorsupportgroup/?ref=br_rs">https://www.facebook.com/groups/baicellsoperatorsupportgroup/?ref=br_rs</a>) is closed, so I can’t see what’s being discussed there.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Frank<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> Outages [<a href="mailto:outages-bounces@outages.org">mailto:outages-bounces@outages.org</a>] <b>On Behalf Of </b>Zak Rupas via Outages<br><b>Sent:</b> Thursday, January 04, 2018 10:34 AM<br><b>To:</b> <a href="mailto:Outages@outages.org">Outages@outages.org</a><br><b>Subject:</b> [outages] Baicells Outage?<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><p><span style='color:black'>Good Morning Outages-<o:p></o:p></span></p><p><span style='color:black'><o:p> </o:p></span></p><p><span style='color:black'>Is anyone feeling this issue currently? Any truth to the story?<o:p></o:p></span></p><p><span style='color:black'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Segoe UI",sans-serif;color:#212121;background:white'>From Baicells facebook group:</span><span style='font-size:11.5pt;font-family:"Segoe UI",sans-serif;color:#212121'><br><br><span style='background:white'>So this morning I feel a bit like the parent who woke up having discovered his child injured someone in a car crash. I'm not directly responsible, but I've some culpability -- or a lot of it -- due to the choices I've made as parent.</span></span><span style='font-size:12.0pt;color:black'> <o:p></o:p></span></p><p style='mso-margin-top-alt:4.5pt;margin-right:0in;margin-bottom:4.5pt;margin-left:0in;background:white'><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#1D2129'>Yes, it is true the discovery of the Intel x86 security flaw set the tech industry ablaze last night. Among the reactions Microsoft abruptly shutdown it's Azure cloud servers with no advance notice. Yes, that specific action was beyond our contro</span><span class=textexposedshow><span style='font-size:10.5pt;font-family:"inherit",serif;color:#1D2129'>l. But, Amazon is dancing, while people who rely upon Azure -- all of us -- are justifiably frustrated and angry.</span></span><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#1D2129'><o:p></o:p></span></p><div><p style='margin-bottom:4.5pt;background:white'><span style='font-size:10.5pt;font-family:"inherit",serif;color:#1D2129'>On this occasion, redundancy across Azure servers AND across various LTE functions, like HSS and MME, did not help us.<o:p></o:p></span></p><p style='mso-margin-top-alt:4.5pt;margin-right:0in;margin-bottom:4.5pt;margin-left:0in;background:white'><span style='font-size:10.5pt;font-family:"inherit",serif;color:#1D2129'>Yes, we are going to investigate adding redundancy across multiple cloud providers such as Amazon....but let's be blunt...who's to say that step still will be enough. What needs completion without excuse or qualification are the other options we've discussed ad nauseum, the local EPC and Halo B.<o:p></o:p></span></p><p style='mso-margin-top-alt:4.5pt;margin-right:0in;margin-bottom:4.5pt;margin-left:0in;background:white'><span style='font-size:10.5pt;font-family:"inherit",serif;color:#1D2129'>Apologies from us aren't gonna cut it, nor will excuses, so I'm not going to waste your time offering them. I'm already this AM discussing with our executive team pushing as top priority the other EPC options. Here's what I will tell you.<o:p></o:p></span></p><p style='mso-margin-top-alt:4.5pt;margin-right:0in;margin-bottom:4.5pt;margin-left:0in;background:white'><span style='font-size:10.5pt;font-family:"inherit",serif;color:#1D2129'>This past Tuesday a new hire started at Baicells North America. Ronald Mao is ex-Huawei and ex-Motorola and has lived in the US since 1987. His entire career has been centered on product line management. He is our new PLM for, shall we say, major things. I've already asked Jesse to brief Ronald on the ongoing cloud issues, as well as local where we are on the local EPC and Halo B. I will ask Ronald to send me an update each week, and I'll pass this one to the group via Facebook AND an email UNTIL THIS IS DONE.<o:p></o:p></span></p><p style='mso-margin-top-alt:4.5pt;margin-right:0in;margin-bottom:4.5pt;margin-left:0in;background:white'><span style='font-size:10.5pt;font-family:"inherit",serif;color:#1D2129'>One P.S. note, thank you Cameron and Rick. While it's small consolation to our customers, Rick and Cam have been up all night working with the team overseas, reporting to you, and in general trying to manage what's been frankly out of their hands.<o:p></o:p></span></p><p style='mso-margin-top-alt:4.5pt;margin-right:0in;margin-bottom:4.5pt;margin-left:0in;background:white'><span style='font-size:10.5pt;font-family:"inherit",serif;color:#1D2129'><a href="https://www.facebook.com/jesse.raasch?fref=gs&dti=1588455311448839&hc_location=group" target="_blank"><span style='color:#365899'>Jesse Raasch</span></a> <a href="https://www.facebook.com/ciaworks?fref=gs&dti=1588455311448839&hc_location=group" target="_blank"><span style='color:#365899'>Cameron Kilton</span></a> <a href="https://www.facebook.com/rick.harnish.10?fref=gs&dti=1588455311448839&hc_location=group" target="_blank"><span style='color:#365899'>Rick Harnish</span></a> <a href="https://www.facebook.com/savannah.lancaster?fref=gs&dti=1588455311448839&hc_location=group" target="_blank"><span style='color:#365899'>Savannah Lancaster</span></a> Ronald Mao <a href="https://www.facebook.com/minchul.ho?fref=gs&dti=1588455311448839&hc_location=group" target="_blank"><span style='color:#365899'>Minchul Ho</span></a> <a href="https://www.facebook.com/sonny.may.94?fref=gs&dti=1588455311448839&hc_location=group" target="_blank"><span style='color:#365899'>Sonny May</span></a> <a href="https://www.facebook.com/nitisha.potti?fref=gs&dti=1588455311448839&hc_location=group" target="_blank"><span style='color:#365899'>Nitisha Potti</span></a> <a href="https://www.facebook.com/bsenekham?fref=gs&dti=1588455311448839&hc_location=group" target="_blank"><span style='color:#365899'>Boun Senekham</span></a><o:p></o:p></span></p><p style='mso-margin-top-alt:4.5pt;margin-right:0in;margin-bottom:4.5pt;margin-left:0in;background:white'><span style='font-size:10.5pt;font-family:"inherit",serif;color:#1D2129'>Update: I spoke with Ronald this morning (he is in CA). He has his marching orders. I'll post updates from him until we close on the local EPC and Halo B.<o:p></o:p></span></p></div><p style='margin-bottom:4.5pt;background:white'><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#1D2129'>So Cameron has been trying to post this, but it's getting rejected:<o:p></o:p></span></p><p style='margin-top:4.5pt;background:white'><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#1D2129'>For those still having CPE attach issues. Please instruct your customers to have CPE powered off for at least 5 minutes.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Segoe UI",sans-serif;color:#212121;background:white'> </span><span style='font-size:12.0pt;color:black'> <o:p></o:p></span></p><p style='margin-top:4.5pt;background:white'><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#1D2129'><o:p> </o:p></span></p><p style='margin-top:4.5pt;background:white'><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#1D2129'><o:p> </o:p></span></p><p style='margin-top:4.5pt;background:white'><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#1D2129'>Micah Deshotel That didn't work for me. The down ones stayed down. Power<o:p></o:p></span></p><p style='margin-bottom:4.5pt;background:white'><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#1D2129'>System Alert: OMC is currently reporting offline. We are investigating as of 10:12PM EST. UPDATE: 11:07pm EST Azure is rebooting servers to apply a major patch. Most of our instances are back online. OMC should be restored shortly.<o:p></o:p></span></p><p style='mso-margin-top-alt:4.5pt;margin-right:0in;margin-bottom:4.5pt;margin-left:0in;background:white'><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#1D2129'>UPDATE 2: 1:20am EST. MME VMs also fell victim to the critical issue with Azure. VMs have since been restored. More information about the issue an be found here: <a href="https://l.facebook.com/l.php?u=https%3A%2F%2Fwww.geekwire.com%2F2018%2Fcloud-vendors-secretly-scramble-patch-critical-flaw-intel-chips-performance-hits-expected%2F&h=ATNFZQ-do0svtoa_e5hnQZl0gvZbgu6awtfb_Gk4bHVCd87C26PuC1WMVDtgDyT8qK3fFfPSQ7b34-F2tgFtUfQHStT7qfLlstCNSV7V6lexFW_hL3D1t6kkGbeOa7beRxl6DrJuMP_oeinlJRpZ3wFWRWUumOt1zsrGTV5Hv9Hzc_Br_dwBI54WDVkxt3_FW3zyGrAVA8u2dM_lnJetjxJVrD8N7PCXs3a6glQv0d_nWDmsRvN3MmiTIZV3gBHTITs7lk2JgmLJyKs_p0yfVvwJ3TkLPouV_wTQ" target="_blank"><span style='font-family:"inherit",serif;color:#365899'>https://www.geekwire.com/…/cloud-vendors-secretly-scramble…/</span></a><o:p></o:p></span></p><p style='margin-top:4.5pt;background:white'><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#1D2129'>We will continue to review our cloud infrastructure and investigate cross platform redundancy.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Segoe UI",sans-serif;color:#212121;background:white'> </span><span style='font-size:12.0pt;color:black'> <o:p></o:p></span></p><p style='margin-top:4.5pt;background:white'><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#1D2129'><o:p> </o:p></span></p><p style='margin-top:4.5pt;background:white'><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#1D2129'><o:p> </o:p></span></p><p style='margin-top:4.5pt;background:white'><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#1D2129'>Cycling the UEs hasnt worked yet either......<o:p></o:p></span></p><p><span style='color:black'><o:p> </o:p></span></p><p><span style='color:black'><o:p> </o:p></span></p><div id=Signature><div name=divtagdefaultwrapper><div><p class=MsoNormal><span class=highlight><span style='color:#1F497D;background:white'>Thanks</span></span><span style='font-size:11.5pt;font-family:"Segoe UI",sans-serif;color:#212121'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='color:#1F497D;background:white'>Zak Rupas</span><span style='font-size:12.0pt;color:black'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='color:#1F497D;background:white'>Forethought.net</span><span style='font-size:11.5pt;font-family:"Segoe UI",sans-serif;color:#212121'><o:p></o:p></span></p></div></div></div></div></body></html>