[outages] Level 3 issues....again

Thu Feb 13 12:27:12 EST 2014

I don't think this was TODAYS outage. It was from over a day ago.

On Thu, Feb 13, 2014 at 12:21 PM, Luke Rockwell <
luke.rockwell at alumni.berkeley.edu> wrote:

> Offical L3 log
>
> Network Event Detail
>  Network Event Summary: A suspected Denial of Service (DoS) attack on
> several links was impacting Voice and IP services in multiple markets.  Event Ticket ID:
> 7531588Market Area Affected: Multiple Markets  Ticket Create Date: 2/12/14
> 1:26:49 AM GMT  Impacted For: *2 hours 32 minutes*   Event Status:Restored
>  Resolve Date: 2/13/14 5:02:13 PM GMT  Time Since Last Update: *1 day 12
> hours 39 minutes*   *There are currently no reported impacts to your
> services caused by this network event. If you are experiencing a service
> disruption, please report the issue to Level 3 by submitting a ticket via
> the MyLevel3 Portal or calling 877-4LEVEL3.*
>  Recent Notes
>  DateNotes 2/12/14 4:41:37 AM GMTThe IP NOC and Operations Engineering
> have confirmed that the issue has resolved. It has been advised that
> monitoring will continue to ensure the links remain stable. Should further
> issues be experienced, please contact the Technical Service Center at
> 877-4LEVEL3. 2/12/14 4:01:32 AM GMT The IP NOC has advised that the issue
> is believed to have been mitigated. Operations Engineers are currently
> working to confirm that no further impact is being experienced at this time. 2/12/14
> 2:57:57 AM GMTOperations Engineers continue investigating the issue with
> the IP NOC and Tier III Technical Support. 2/12/14 2:25:58 AM GMTOperations Engineering has advised that Tier III Technical Support is being
> engaged at this time to assist with investigations. 2/12/14 2:03:18 AM GMTThe
> IP NOC continues working diligently with Operations Engineering to identify
> the source of the attack. 2/12/14 1:41:56 AM GMTIt has been confirmed
> that IP services are also being impacted. The IP NOC has advised that that
> the issue is suspected to be the result of a Denial of Service (DoS) attack
> on several links in the United States. Operations Engineering are being
> engaged at this time to assist with investigations. 2/12/14 1:30:01 AM GMTThe
> Voice NOC has reported that the Border Gateway Protocol bounced on an IP
> link, impacting voice services in multiple markets. Investigations are
> currently ongoing to determine the full impact of the issue.
>
> Here is another event I dont know if it is related
> Network Event Detail
>  Network Event Summary: Packet loss across multiple markets in North
> America was impacting IP services.  Event Ticket ID: 7524423Market Area
> Affected: Multiple Markets  Ticket Create Date: 2/10/14 5:06:48 PM GMT  Impacted
> For: *17 hours 8 minutes*  Event Status: Restored  Resolve Date: 2/13/14
> 5:05:39 PM GMT  Time Since Last Update: *2 days 13 hours 45 minutes*   *There
> are currently no reported impacts to your services caused by this network
> event. If you are experiencing a service disruption, please report the
> issue to Level 3 by submitting a ticket via the MyLevel3 Portal or calling
> 877-4LEVEL3.*
>  Recent Notes
>  DateNotes 2/11/14 3:34:25 AM GMTThe Operations Engineering team reported
> that the script that was developed was successful in resolving the packet
> loss issue. It has been informed that the issue has resolved and services
> have restored. If additional issues are experienced, please contact the
> Level 3 Technical Service Center. 2/11/14 2:12:21 AM GMT The Operations
> Engineering team advised that the scripts are currently running at this
> time in an effort to restore services. An estimated restoral time remains
> unknown and the next update will be sent as new information is received. 2/10/14
> 11:30:05 PM GMTAt this time, the Technical Service Center is working with
> the IP NOC to develop a script that can push the filter to a larger group
> of customers to expedite restoration. After the script is run, teams will
> validate deployment success to ensure stability. The Operations Engineering
> team reported that approximately fifty percent of the impact has been
> mitigated and efforts continue to increase that percentage as quickly as
> possible. The next update will be provided at approximately 01:00 GMT or as
> new information is made available. 2/10/14 11:10:30 PM GMT The Operation
> Engineering team is diligently working to restore impacted services. An
> estimated restoration time is unavailable, but updates will be provided as
> soon as more information is made available. 2/10/14 9:00:58 PM GMTThe
> Operation Engineering team has advised that they are investigating a mass
> deployment to expedite service restoral. Unfortunately, an estimated
> restoral time is not available as it may vary for impacted customers. 2/10/14
> 7:51:53 PM GMT The Operations Engineering team continues to work to
> isolate a solution for the packet loss. The investigating group is unable
> to provide an ETTR for this case at this time. 2/10/14 7:10:14 PM GMTThe
> Operations Engineering team continues to troubleshoot the issue in an
> effort to restore services. 2/10/14 5:28:31 PM GMT The IP Technical
> Service Center reported multiple customers that are experiencing an outage
> to IP services due to heavy packet loss across multiple markets in North
> America. Numerous managed devices are unresponsive at this time, and the
> Network Operations Engineering teams are currently working to mitigate the
> packet loss and restore stability to individual customers. An estimated
> restoral time is not available as it may vary for impacted customers.
>
> On Thu, Feb 13, 2014 at 4:59 AM, Chris Stone <cstone at axint.net> wrote:
>
>>  Got that from L3 as well. Looks like my sessions have been up and
>> stable since 02:23:32 MST.
>>
>>
>>
>> On 02/13/2014 03:41 AM, Mitch Patterson wrote:
>>
>> Final notes from them. Any one still seeing issues? My direct level 3
>> circuits look good on a spot check. Still having issues with a ISP that has
>> Level 3 as their only major upstream
>>
>>   *** CASCADED EXTERNAL NOTES 13-Feb-2014 10:32:49 GMT From CASE:
>> 7538219 - Event A packet loss issue impacted IP Services across multiple
>> markets in the North America. The Technical Service Center worked with the
>> IP NOC to develop a configuration change to a larger group of customers to
>> expedite restoration. After the configuration change was run, teams
>> validated the deployment success to ensure stability. The Operations
>> Engineering team reported that the configuration change that was developed
>> was successful in resolving the packet loss issue. It was communicated that
>> the issue has resolved and services have restored. If additional issues are
>> experienced, please contact the Level 3 Technical Service Center.
>>
>>
>> On Thu, Feb 13, 2014 at 4:10 AM, Mitch Patterson <mitpatterson at gmail.com>wrote:
>>
>>> Latest update
>>>
>>>  *** CASCADED EXTERNAL NOTES 13-Feb-2014 09:07:11 GMT From CASE:
>>> 7538219 - Event Security Ops attempted to apply filters on the network
>>> routers to reject the IP traffic of the DDoS attacks.  However, the filters
>>> were unsuccessful due to the multiple sources of the attack.  Security is
>>> attempting another plan to limit the effect of the attack on the targeted
>>> destination of the DDoS attack.
>>>
>>>
>>> On Thu, Feb 13, 2014 at 3:28 AM, Mitch Patterson <mitpatterson at gmail.com
>>> > wrote:
>>>
>>>> Same reply they gave me.
>>>>  On Feb 13, 2014 3:27 AM, "Petter Bruland" <
>>>> Petter.Bruland at allegiantair.com> wrote:
>>>>
>>>>>  We saw similar issues with Level3 at Switch SuperNAP in Las Vegas.
>>>>>
>>>>> Last we heard:
>>>>> Level-3 IP NOC Team and Security Ops are applying filters on affected
>>>>> routers to re-establish routing sessions and restore normal operations but
>>>>> they have not provided any ETR as of now.
>>>>>
>>>>> -P
>>>>>
>>>>>
>>>>>   *Petter Bruland *| Network Engineer
>>>>> Allegiant Travel Company
>>>>> 8360 S. Durango Drive, Las Vegas, NV 89113
>>>>> Phone: (702) 874-3332 <%28702%29%20874-3332> | Cell: (702) 286-6549<%28702%29%20286-6549>
>>>>> petter.bruland at allegiantair.com
>>>>> http://www.allegiantair.com
>>>>>     ------------------------------
>>>>> *From:* Outages [outages-bounces at outages.org] on behalf of Mitch
>>>>> Patterson [mitpatterson at gmail.com]
>>>>> *Sent:* Wednesday, February 12, 2014 11:38 PM
>>>>> *To:* outages at outages.org
>>>>> *Subject:* [outages] Level 3 issues....again
>>>>>
>>>>>  Massive Level 3 issues again. Got this from them:
>>>>>
>>>>>
>>>>>
>>>>> The IP NOC has detected Denial of Service (DoS) attacks on various
>>>>> network routers in Houston, TX. The IP NOC is analyzing the routing
>>>>> protocol session failures to determine the next course of action.
>>>>>
>>>>>
>>>>>
>>>>> IP NOC and Security Ops are working on resolving the BGP sessions
>>>>> failures on network routers affected by the DDoS attack.
>>>>>
>>>>
>>>
>>
>>
>> _______________________________________________
>> Outages mailing listOutages at outages.orghttps://puck.nether.net/mailman/listinfo/outages
>>
>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20140213/bb3eb6a6/attachment.htm>