[Outages-discussion] IPv6 outages overnight

[Vinny_Abello at Dell.com]

Level 3 had a router go south after some network maintenance in New York around the time you indicated. It might be related:


Network Event Summary:

A possible network router isolation occurred after scheduled maintenance in New York City, NY impacting IP traffic

Event Ticket ID:

5379276

Market Area Affected:

New York City, NY

Ticket Create Date:

2/9/12 6:36:02 AM GMT

Impacted For:

2 hours 0 minutes

Event Status:

Restored

Resolve Date:

2/9/12 10:02:09 AM GMT

Time Since Last Update:

20 hours 47 minutes

There are currently no reported impacts to your services caused by this network event. If you are experiencing a service disruption, please report the issue to Level 3 by submitting a ticket via the MyLevel3 Portal or calling 877-4LEVEL3.

Recent Notes
Date

Notes

2/9/12 7:52:53 AM GMT

The network router has restored to normal operation.

2/9/12 7:46:07 AM GMT

IP NOC advised the router has restored and the line cards are active. IP NOC and Ops Engineering are waiting for route convergence of IP traffic signifying the incorporation of the router back into the network.

2/9/12 7:04:25 AM GMT

IP NOC has restricted all of the uplinks to the network router to stabilize the CPU load. Unfortunately, the manual isolation did not stabilize the CPU load. Consequently, the field technician on site will power-cycle the unit to re-gain access to the router.

2/9/12 6:42:52 AM GMT

IP NOC advised that a network router in New York City experienced a memory failure and heavy CPU Load with effectively isolated the unit from the network after the completion of a scheduled maintenance activity. IP NOC is unable to access the network router until the heavy CPU load has subsided. IP NOC is monitoring the status of IP traffic around the router to minimize customer traffic disruption.






-----Original Message-----
From: outages-discussion-bounces at outages.org [mailto:outages-discussion-bounces at outages.org] On Behalf Of Frank Bulk
Sent: Thursday, February 09, 2012 10:22 PM
To: 'Mark Kamichoff'
Cc: outages-discussion at outages.org
Subject: Re: [Outages-discussion] IPv6 outages overnight



Thanks, I'm glad to hear it just wasn't me.  My access was ipv6.juniper.net

was uninterrupted during that time.



Frank



-----Original Message-----

From: Mark Kamichoff [mailto:prox at prolixium.com]<mailto:[mailto:prox at prolixium.com]>

Sent: Thursday, February 09, 2012 7:32 PM

To: Frank Bulk

Cc: outages-discussion at outages.org<mailto:outages-discussion at outages.org>

Subject: Re: [Outages-discussion] IPv6 outages overnight



Hi Frank -



On Thu, Feb 09, 2012 at 06:32:08PM -0600, Frank Bulk wrote:

> I monitor over 30 IPv6 websites and Wednesday night between 11:52 pm

> and Thursday morning 12:50 am Central four of those sites became

> inaccessible almost all at the same time.

>              ipv6.t-mobile.com

>              www.t-home.hu<http://www.t-home.hu> [just the one AAAA: 2001:4c48:2:46::1b:1]

>              www.twtelecom.com<http://www.twtelecom.com>

>              www.t-online.de<http://www.t-online.de>

> Those sites (currently) go over different transit links (two over

> Sprint, two over Level3).

>

> Here are traceroutes from 2607:fe28:10::/48.  Unfortunately I was not

> awake when this happened, so I couldn't see where it failed.  BGPlay

> on the IPv6 and IPv4 prefixes I tried don't show any routing events at

> our upstream transit during those times.

>

> Anyone else see anything around that time?



A few of my SmokePing (fping6) graphs showed gaps to a few nodes around

that same time (ipv6.juniper.net, ipv6.akamai.com, etc.).  Judging by

traceroutes I believe it was Level3-related.



- Mark



--

Mark Kamichoff

prox at prolixium.com<mailto:prox at prolixium.com>

http://www.prolixium.com/





_______________________________________________

Outages-discussion mailing list

Outages-discussion at outages.org<mailto:Outages-discussion at outages.org>

https://puck.nether.net/mailman/listinfo/outages-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20120210/40fb9746/attachment-0001.html>