[outages] [External] Re: CrownCastle / Lightower issue in NY/NJ area

Mike Castoro mcastoro at AdriannaPapell.com
Wed Oct 10 17:22:09 EDT 2018


Post-mortem on Crown Castle / Lightower issue from 10/9...  I would have sent to "outages-discussion" list but I was never added to the list.

"On October 9, 2018, Crown Castle Fiber experienced two significant network events on our core backbone that impacted IP and Ethernet services.  The root cause of the issue was a hardware issue on a Crown Castle Fiber backbone switch in the New York Metro region. As a result of the hardware issue, Crown Castle Fiber had (3) backbone trunks down.  In an effort to restore full capacity, Crown Castle Fiber Network Engineering worked with the hardware vendor support team and it was recommended to reset the faulty hardware.  Services began to restore; however, the hardware did not fully recover resulting in unidirectional routing issues.  Crown Castle Fiber then disabled the hardware, forcing traffic to alternate paths to restore connectivity.

At approximately 2:30pm eastern time, the faulty hardware was inadvertently re-enabled, causing a repeat of the morning's issue.  The hardware was subsequently disabled and routing was stabilized.   Steps have since been taken to prevent any opportunity for reoccurrence.

Any further action taken on this issue will be done in a controlled environment with no impact to customer service.

A complete trouble resolution report will be sent out once both Crown Castle and our hardware vendor have an opportunity to complete a full investigation into the issue.


Crown Castle is committed to improving network performance and operational processes to reduce downtime.   We apologize for the adverse impact this issue may have caused.  If additional testing and verification is required, please contact our Network Operations Center at (855) 93-FIBER."

Mike Castoro | Director of IT
Adrianna Papell Group | 646-723-3885

From: Mike Castoro
Sent: Tuesday, October 09, 2018 9:22 AM
To: Sulaman, Muhammad <msulaman at institute.org>; Brian Weir <whickey at mac.com>; Neil Hanlon <nhanlon at kayak.com>; Daniel Hoye <dhoye at breckinridge.com>
Cc: outages at outages.org
Subject: Re: [outages] [External] Re: CrownCastle / Lightower issue in NY/NJ area

Still having issues. Going up and down in NYC. Kearny NJ is up.
Mike Castoro
Director of IT
(646) 723-3885
mcastoro at adriannapapell.com<mailto:mcastoro at adriannapapell.com>

________________________________
From: Outages <outages-bounces at outages.org<mailto:outages-bounces at outages.org>> on behalf of Daniel Hoye via Outages <outages at outages.org<mailto:outages at outages.org>>
Sent: Tuesday, October 9, 2018 7:44:48 AM
To: Sulaman, Muhammad; Brian Weir; Neil Hanlon
Cc: outages at outages.org<mailto:outages at outages.org>
Subject: Re: [outages] [External] Re: CrownCastle / Lightower issue in NY/NJ area

External Email!
Can confirm issues at PHL colo for Lightower/Crown Castle.  Back up now though.


From: Outages <outages-bounces at outages.org<mailto:outages-bounces at outages.org>> On Behalf Of Sulaman, Muhammad via Outages
Sent: Tuesday, October 9, 2018 7:41 AM
To: Brian Weir <whickey at mac.com<mailto:whickey at mac.com>>; Neil Hanlon <nhanlon at kayak.com<mailto:nhanlon at kayak.com>>
Cc: outages at outages.org<mailto:outages at outages.org>
Subject: Re: [outages] [External] Re: CrownCastle / Lightower issue in NY/NJ area


[Image removed by sender.]This Message was Encrypted.

We had issues this morning connecting to our VPN concentrator located in upstate NY. After reading this last message I tried again and was able to connect.

Cheers,
M

From: Brian Weir via Outages <outages at outages.org<mailto:outages at outages.org>>
Sent: Tuesday, October 9, 2018 7:37 AM
To: Neil Hanlon <nhanlon at kayak.com<mailto:nhanlon at kayak.com>>
Cc: outages at outages.org<mailto:outages at outages.org>
Subject: Re: [outages] [External] Re: CrownCastle / Lightower issue in NY/NJ area

Downtown Manhattan DIA circuit was down as well.  However monitoring just reported it as coming back online.

On Oct 9, 2018, at 7:32 AM, Neil Hanlon via Outages <outages at outages.org<mailto:outages at outages.org>> wrote:
I can confirm there's a cut in NJ. Somehow our branch offices in disparate locations is affected...
Had to call our account team and the NOC manager to get a straight answer.
On Oct 9, 2018, at 07:29, Mike Castoro via Outages <outages at outages.org<mailto:outages at outages.org>> wrote:
Yes, I lost my DIA lines in Midtown NYC and Kearny NJ around the same time.
Mike Castoro
Director of IT
(646) 723-3885
mcastoro at adriannapapell.com<mailto:mcastoro at adriannapapell.com>

________________________________
From: Outages <outages-bounces at outages.org<mailto:outages-bounces at outages.org>> on behalf of Matthew Huff via Outages <outages at outages.org<mailto:outages at outages.org>>
Sent: Tuesday, October 9, 2018 7:21:58 AM
To: outages at outages.org<mailto:outages at outages.org>
Subject: [outages] CrownCastle / Lightower issue in NY/NJ area

External Email!


Around 3:45 AM EST we lost our 10GB WAN circuit from NY to NJ. Around 6:00 AM we started having issue passing traffic via our separate Lightower internet circuit (circuit & BGP peering was up, but packets were getting lost somewhere in their ASN).

I opened a case with CrownCastle about 10 10GB circuit about 5:30 AM, but when I called in to report the internet issue, their support line I get in the queue, but never get a tech. I imagine they are getting slammed with calls.



----
Matthew Huff             | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039


_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages

________________________________

Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages

________________________________

This email and its contents are confidential. If you have received this email in error, please notify the sender and delete this message. No confidentiality is waived or lost by any transmission errors. Any unauthorized dissemination, distribution or copying of this email is strictly forbidden. Email transmission cannot be guaranteed to be secure or error-free, as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Therefore, neither Breckinridge nor the sender will accept liability for any errors or omissions in the contents of this message which arise as a result of email transmission. Breckinridge reserves the right to archive, monitor and review all emails within its network.

Nothing contained in this email should be relied upon or construed as financial, legal or tax advice. This email is not a solicitation or offer for the purchase or sale of any interest in any securities or as an official confirmation of any transaction. All investments involve risks, including the potential loss of principal. Any clients mentioned in this email may or may not maintain similar investment objectives as those expressed by your organization or may or may not be familiar to your organization. It is not known whether these clients approve or disapprove of Breckinridge or its advisory services. Past performance is no guarantee of future results. This message may contain the views and opinions of Breckinridge; these are subject to change.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20181010/fd527e49/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 335 bytes
Desc: image001.jpg
URL: <https://puck.nether.net/pipermail/outages/attachments/20181010/fd527e49/attachment.jpg>


More information about the Outages mailing list