<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal00, li.msonormal00, div.msonormal00
{mso-style-name:msonormal0;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.emailquote, li.emailquote, div.emailquote
{mso-style-name:emailquote;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:1.0pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msochpdefault, li.msochpdefault, div.msochpdefault
{mso-style-name:msochpdefault;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
span.htmlpreformattedchar0
{mso-style-name:htmlpreformattedchar;
font-family:Consolas;}
span.emailstyle21
{mso-style-name:emailstyle21;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.emailstyle22
{mso-style-name:emailstyle22;
font-family:"Calibri",sans-serif;}
span.emailstyle23
{mso-style-name:emailstyle23;
font-family:"Calibri",sans-serif;}
span.EmailStyle28
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Post-mortem on Crown Castle / Lightower issue from 10/9… I would have sent to “outages-discussion” list but I was never added to the list.<o:p></o:p></span></p>
<p style="margin-bottom:0in;margin-bottom:.0001pt"><span style="color:#1F497D">“</span>On October 9, 2018, Crown Castle Fiber experienced two significant network events on our core backbone that impacted IP and Ethernet services. The root cause of the issue
was a hardware issue on a Crown Castle Fiber backbone switch in the New York Metro region. As a result of the hardware issue, Crown Castle Fiber had (3) backbone trunks down. In an effort to restore full capacity, Crown Castle Fiber Network Engineering worked
with the hardware vendor support team and it was recommended to reset the faulty hardware. Services began to restore; however, the hardware did not fully recover resulting in unidirectional routing issues. Crown Castle Fiber then disabled the hardware, forcing
traffic to alternate paths to restore connectivity. <o:p></o:p></p>
<p style="margin-bottom:0in;margin-bottom:.0001pt">At approximately 2:30pm eastern time, the faulty hardware was inadvertently re-enabled, causing a repeat of the morning’s issue. The hardware was subsequently disabled and routing was stabilized. Steps have
since been taken to prevent any opportunity for reoccurrence. <o:p></o:p></p>
<p style="margin-bottom:0in;margin-bottom:.0001pt">Any further action taken on this issue will be done in a controlled environment with no impact to customer service.<o:p></o:p></p>
<p style="margin-bottom:0in;margin-bottom:.0001pt">A complete trouble resolution report will be sent out once both Crown Castle and our hardware vendor have an opportunity to complete a full investigation into the issue.<o:p></o:p></p>
<p style="margin-bottom:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p class="MsoNormal">Crown Castle is committed to improving network performance and operational processes to reduce downtime. We apologize for the adverse impact this issue may have caused. If additional testing and verification is required, please contact
our Network Operations Center at (855) 93-FIBER.”<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Mike Castoro | Director of IT<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Adrianna Papell Group | 646-723-3885<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Mike Castoro
<br>
<b>Sent:</b> Tuesday, October 09, 2018 9:22 AM<br>
<b>To:</b> Sulaman, Muhammad <msulaman@institute.org>; Brian Weir <whickey@mac.com>; Neil Hanlon <nhanlon@kayak.com>; Daniel Hoye <dhoye@breckinridge.com><br>
<b>Cc:</b> outages@outages.org<br>
<b>Subject:</b> Re: [outages] [External] Re: CrownCastle / Lightower issue in NY/NJ area<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black">Still having issues. Going up and down in NYC. Kearny NJ is up.
<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black">Mike Castoro<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black">Director of IT<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black">(646) 723-3885<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black"><a href="mailto:mcastoro@adriannapapell.com">mcastoro@adriannapapell.com</a><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<hr size="2" width="98%" align="center">
</span></div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Outages <<a href="mailto:outages-bounces@outages.org">outages-bounces@outages.org</a>>
on behalf of Daniel Hoye via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>><br>
<b>Sent:</b> Tuesday, October 9, 2018 7:44:48 AM<br>
<b>To:</b> Sulaman, Muhammad; Brian Weir; Neil Hanlon<br>
<b>Cc:</b> <a href="mailto:outages@outages.org">outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] [External] Re: CrownCastle / Lightower issue in NY/NJ area</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Calibri",sans-serif;color:red">External Email!</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Can confirm issues at PHL colo for Lightower/Crown Castle. Back up now though.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Outages <<a href="mailto:outages-bounces@outages.org">outages-bounces@outages.org</a>>
<b>On Behalf Of </b>Sulaman, Muhammad via Outages<br>
<b>Sent:</b> Tuesday, October 9, 2018 7:41 AM<br>
<b>To:</b> Brian Weir <<a href="mailto:whickey@mac.com">whickey@mac.com</a>>; Neil Hanlon <<a href="mailto:nhanlon@kayak.com">nhanlon@kayak.com</a>><br>
<b>Cc:</b> <a href="mailto:outages@outages.org">outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] [External] Re: CrownCastle / Lightower issue in NY/NJ area</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="MsoNormal" align="center" style="text-align:center;background:#006EC2">
<span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white;border:solid windowtext 1.0pt;padding:0in"><img border="0" width="10" height="10" style="width:.1041in;height:.1041in" id="_x0000_i1026" src="cid:image001.jpg@01D460BD.C5177000" alt="Image removed by sender."></span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white">This
Message was Encrypted. </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">We had issues this morning connecting to our VPN concentrator located in upstate NY. After reading this last message I tried again and was able to connect.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Cheers,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">M</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Brian Weir via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>>
<br>
<b>Sent:</b> Tuesday, October 9, 2018 7:37 AM<br>
<b>To:</b> Neil Hanlon <<a href="mailto:nhanlon@kayak.com">nhanlon@kayak.com</a>><br>
<b>Cc:</b> <a href="mailto:outages@outages.org">outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] [External] Re: CrownCastle / Lightower issue in NY/NJ area</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal">Downtown Manhattan DIA circuit was down as well. However monitoring just reported it as coming back online.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
On Oct 9, 2018, at 7:32 AM, Neil Hanlon via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I can confirm there's a cut in NJ. Somehow our branch offices in disparate locations is affected...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Had to call our account team and the NOC manager to get a straight answer.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">On Oct 9, 2018, at 07:29, Mike Castoro via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black">Yes, I lost my DIA lines in Midtown NYC and Kearny NJ around the same time.
</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black">Mike Castoro</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black">Director of IT</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black">(646) 723-3885</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black"><a href="mailto:mcastoro@adriannapapell.com">mcastoro@adriannapapell.com</a></span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="4" width="98%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Outages <<a href="mailto:outages-bounces@outages.org">outages-bounces@outages.org</a>>
on behalf of Matthew Huff via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>><br>
<b>Sent:</b> Tuesday, October 9, 2018 7:21:58 AM<br>
<b>To:</b> <a href="mailto:outages@outages.org">outages@outages.org</a><br>
<b>Subject:</b> [outages] CrownCastle / Lightower issue in NY/NJ area</span> <o:p>
</o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">External Email!<br>
<br>
<br>
Around 3:45 AM EST we lost our 10GB WAN circuit from NY to NJ. Around 6:00 AM we started having issue passing traffic via our separate Lightower internet circuit (circuit & BGP peering was up, but packets were getting lost somewhere in their ASN).<br>
<br>
I opened a case with CrownCastle about 10 10GB circuit about 5:30 AM, but when I called in to report the internet issue, their support line I get in the queue, but never get a tech. I imagine they are getting slammed with calls.<br>
<br>
<br>
<br>
----<br>
Matthew Huff | 1 Manhattanville Rd<br>
Director of Operations | Purchase, NY 10577<br>
OTA Management LLC | Phone: 914-460-4039<br>
<br>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a></span><o:p></o:p></p>
</div>
<pre style="text-align:center"><hr size="4" width="100%" align="center"></pre>
<pre><br>Outages mailing list<br><a href="mailto:Outages@outages.org">Outages@outages.org</a><br><a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></pre>
</blockquote>
</div>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<hr size="2" width="100%" align="center">
</span></div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:gray"><br>
This email and its contents are confidential. If you have received this email in error, please notify the sender and delete this message. No confidentiality is waived or lost by any transmission errors. Any unauthorized dissemination, distribution or copying
of this email is strictly forbidden. Email transmission cannot be guaranteed to be secure or error-free, as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Therefore, neither Breckinridge nor the
sender will accept liability for any errors or omissions in the contents of this message which arise as a result of email transmission. Breckinridge reserves the right to archive, monitor and review all emails within its network.<br>
<br>
Nothing contained in this email should be relied upon or construed as financial, legal or tax advice. This email is not a solicitation or offer for the purchase or sale of any interest in any securities or as an official confirmation of any transaction. All
investments involve risks, including the potential loss of principal. Any clients mentioned in this email may or may not maintain similar investment objectives as those expressed by your organization or may or may not be familiar to your organization. It is
not known whether these clients approve or disapprove of Breckinridge or its advisory services. Past performance is no guarantee of future results. This message may contain the views and opinions of Breckinridge; these are subject to change.</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>