[outages] Level 3 issues....again

Thu Feb 13 12:21:49 EST 2014

Offical L3 log

Network Event Detail
Network Event Summary:A suspected Denial of Service (DoS) attack on several
links was impacting Voice and IP services in multiple markets.  Event
Ticket ID:7531588Market Area Affected:Multiple Markets  Ticket Create
Date:2/12/14
1:26:49 AM GMT  Impacted For:*2 hours 32 minutes*  Event
Status:Restored  Resolve
Date:2/13/14 5:02:13 PM GMT  Time Since Last Update:*1 day 12 hours 39
minutes*  *There are currently no reported impacts to your services caused
by this network event. If you are experiencing a service disruption, please
report the issue to Level 3 by submitting a ticket via the MyLevel3 Portal
or calling 877-4LEVEL3.*
Recent Notes
DateNotes2/12/14 4:41:37 AM GMTThe IP NOC and Operations Engineering have
confirmed that the issue has resolved. It has been advised that monitoring
will continue to ensure the links remain stable. Should further issues be
experienced, please contact the Technical Service Center at 877-4LEVEL3.2/12/14
4:01:32 AM GMTThe IP NOC has advised that the issue is believed to have
been mitigated. Operations Engineers are currently working to confirm that
no further impact is being experienced at this time.2/12/14 2:57:57 AM
GMTOperations
Engineers continue investigating the issue with the IP NOC and Tier III
Technical Support.2/12/14 2:25:58 AM GMTOperations Engineering has advised
that Tier III Technical Support is being engaged at this time to assist
with investigations.2/12/14 2:03:18 AM GMTThe IP NOC continues working
diligently with Operations Engineering to identify the source of the
attack.2/12/14
1:41:56 AM GMTIt has been confirmed that IP services are also being
impacted. The IP NOC has advised that that the issue is suspected to be the
result of a Denial of Service (DoS) attack on several links in the United
States. Operations Engineering are being engaged at this time to assist
with investigations.2/12/14 1:30:01 AM GMTThe Voice NOC has reported that
the Border Gateway Protocol bounced on an IP link, impacting voice services
in multiple markets. Investigations are currently ongoing to determine the
full impact of the issue.

Here is another event I dont know if it is related
Network Event Detail
Network Event Summary:Packet loss across multiple markets in North America
was impacting IP services.  Event Ticket ID:7524423Market Area
Affected:Multiple
Markets  Ticket Create Date:2/10/14 5:06:48 PM GMT  Impacted For:*17 hours
8 minutes*  Event Status:Restored  Resolve Date:2/13/14 5:05:39 PM GMT  Time
Since Last Update:*2 days 13 hours 45 minutes*  *There are currently no
reported impacts to your services caused by this network event. If you are
experiencing a service disruption, please report the issue to Level 3 by
submitting a ticket via the MyLevel3 Portal or calling 877-4LEVEL3.*
Recent Notes
DateNotes2/11/14 3:34:25 AM GMTThe Operations Engineering team reported
that the script that was developed was successful in resolving the packet
loss issue. It has been informed that the issue has resolved and services
have restored. If additional issues are experienced, please contact the
Level 3 Technical Service Center.2/11/14 2:12:21 AM GMTThe Operations
Engineering team advised that the scripts are currently running at this
time in an effort to restore services. An estimated restoral time remains
unknown and the next update will be sent as new information is received.2/10/14
11:30:05 PM GMTAt this time, the Technical Service Center is working with
the IP NOC to develop a script that can push the filter to a larger group
of customers to expedite restoration. After the script is run, teams will
validate deployment success to ensure stability. The Operations Engineering
team reported that approximately fifty percent of the impact has been
mitigated and efforts continue to increase that percentage as quickly as
possible. The next update will be provided at approximately 01:00 GMT or as
new information is made available.2/10/14 11:10:30 PM GMTThe Operation
Engineering team is diligently working to restore impacted services. An
estimated restoration time is unavailable, but updates will be provided as
soon as more information is made available.2/10/14 9:00:58 PM GMTThe
Operation Engineering team has advised that they are investigating a mass
deployment to expedite service restoral. Unfortunately, an estimated
restoral time is not available as it may vary for impacted customers.2/10/14
7:51:53 PM GMTThe Operations Engineering team continues to work to isolate
a solution for the packet loss. The investigating group is unable to
provide an ETTR for this case at this time.2/10/14 7:10:14 PM GMTThe
Operations Engineering team continues to troubleshoot the issue in an
effort to restore services.2/10/14 5:28:31 PM GMTThe IP Technical Service
Center reported multiple customers that are experiencing an outage to IP
services due to heavy packet loss across multiple markets in North America.
Numerous managed devices are unresponsive at this time, and the Network
Operations Engineering teams are currently working to mitigate the packet
loss and restore stability to individual customers. An estimated restoral
time is not available as it may vary for impacted customers.

On Thu, Feb 13, 2014 at 4:59 AM, Chris Stone <cstone at axint.net> wrote:

>  Got that from L3 as well. Looks like my sessions have been up and stable
> since 02:23:32 MST.
>
>
>
> On 02/13/2014 03:41 AM, Mitch Patterson wrote:
>
> Final notes from them. Any one still seeing issues? My direct level 3
> circuits look good on a spot check. Still having issues with a ISP that has
> Level 3 as their only major upstream
>
>   *** CASCADED EXTERNAL NOTES 13-Feb-2014 10:32:49 GMT From CASE: 7538219
> - Event A packet loss issue impacted IP Services across multiple markets in
> the North America. The Technical Service Center worked with the IP NOC to
> develop a configuration change to a larger group of customers to expedite
> restoration. After the configuration change was run, teams validated the
> deployment success to ensure stability. The Operations Engineering team
> reported that the configuration change that was developed was successful in
> resolving the packet loss issue. It was communicated that the issue has
> resolved and services have restored. If additional issues are experienced,
> please contact the Level 3 Technical Service Center.
>
>
> On Thu, Feb 13, 2014 at 4:10 AM, Mitch Patterson <mitpatterson at gmail.com>wrote:
>
>> Latest update
>>
>>  *** CASCADED EXTERNAL NOTES 13-Feb-2014 09:07:11 GMT From CASE: 7538219
>> - Event Security Ops attempted to apply filters on the network routers to
>> reject the IP traffic of the DDoS attacks.  However, the filters were
>> unsuccessful due to the multiple sources of the attack.  Security is
>> attempting another plan to limit the effect of the attack on the targeted
>> destination of the DDoS attack.
>>
>>
>> On Thu, Feb 13, 2014 at 3:28 AM, Mitch Patterson <mitpatterson at gmail.com>wrote:
>>
>>> Same reply they gave me.
>>>  On Feb 13, 2014 3:27 AM, "Petter Bruland" <
>>> Petter.Bruland at allegiantair.com> wrote:
>>>
>>>>  We saw similar issues with Level3 at Switch SuperNAP in Las Vegas.
>>>>
>>>> Last we heard:
>>>> Level-3 IP NOC Team and Security Ops are applying filters on affected
>>>> routers to re-establish routing sessions and restore normal operations but
>>>> they have not provided any ETR as of now.
>>>>
>>>> -P
>>>>
>>>>
>>>>   *Petter Bruland *| Network Engineer
>>>> Allegiant Travel Company
>>>> 8360 S. Durango Drive, Las Vegas, NV 89113
>>>> Phone: (702) 874-3332 <%28702%29%20874-3332> | Cell: (702) 286-6549<%28702%29%20286-6549>
>>>> petter.bruland at allegiantair.com
>>>> http://www.allegiantair.com
>>>>     ------------------------------
>>>> *From:* Outages [outages-bounces at outages.org] on behalf of Mitch
>>>> Patterson [mitpatterson at gmail.com]
>>>> *Sent:* Wednesday, February 12, 2014 11:38 PM
>>>> *To:* outages at outages.org
>>>> *Subject:* [outages] Level 3 issues....again
>>>>
>>>>  Massive Level 3 issues again. Got this from them:
>>>>
>>>>
>>>>
>>>> The IP NOC has detected Denial of Service (DoS) attacks on various
>>>> network routers in Houston, TX. The IP NOC is analyzing the routing
>>>> protocol session failures to determine the next course of action.
>>>>
>>>>
>>>>
>>>> IP NOC and Security Ops are working on resolving the BGP sessions
>>>> failures on network routers affected by the DDoS attack.
>>>>
>>>
>>
>
>
> _______________________________________________
> Outages mailing listOutages at outages.orghttps://puck.nether.net/mailman/listinfo/outages
>
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20140213/c108bb9b/attachment.htm>