[Outages-discussion] IPv6 outages overnight

Fri Feb 10 01:04:49 EST 2012

It seems to be off by an hour - my issue was from 11:52 pm Central (5:52 AM
GMT) to 12:50 am Central (6:50 AM GMT), while Level3 first noted their issue
at 6:36 AM GMT and marked it resolved by 7:52 AM GMT.  Unless the issue
started happening much earlier and it was Level3's dropping of the router's
links resulted in traffic being re-routed and that's when traffic could go
through.

Frank

From: Vinny_Abello at Dell.com [mailto:Vinny_Abello at Dell.com] 
Sent: Thursday, February 09, 2012 10:44 PM
To: frnkblk at iname.com; prox at prolixium.com
Cc: outages-discussion at outages.org
Subject: RE: [Outages-discussion] IPv6 outages overnight

Level 3 had a router go south after some network maintenance in New York
around the time you indicated. It might be related:

Network Event Summary:

A possible network router isolation occurred after scheduled maintenance in
New York City, NY impacting IP traffic 

Event Ticket ID:

5379276 

Market Area Affected:

New York City, NY 

Ticket Create Date:

2/9/12 6:36:02 AM GMT 

Impacted For:

2 hours 0 minutes 

Event Status:

Restored 

Resolve Date:

2/9/12 10:02:09 AM GMT 

Time Since Last Update:

20 hours 47 minutes 

There are currently no reported impacts to your services caused by this
network event. If you are experiencing a service disruption, please report
the issue to Level 3 by submitting a ticket via the MyLevel3 Portal or
calling 877-4LEVEL3.

Recent Notes

Date 

Notes 

2/9/12 7:52:53 AM GMT 

The network router has restored to normal operation. 

2/9/12 7:46:07 AM GMT 

IP NOC advised the router has restored and the line cards are active. IP NOC
and Ops Engineering are waiting for route convergence of IP traffic
signifying the incorporation of the router back into the network. 

2/9/12 7:04:25 AM GMT 

IP NOC has restricted all of the uplinks to the network router to stabilize
the CPU load. Unfortunately, the manual isolation did not stabilize the CPU
load. Consequently, the field technician on site will power-cycle the unit
to re-gain access to the router. 

2/9/12 6:42:52 AM GMT 

IP NOC advised that a network router in New York City experienced a memory
failure and heavy CPU Load with effectively isolated the unit from the
network after the completion of a scheduled maintenance activity. IP NOC is
unable to access the network router until the heavy CPU load has subsided.
IP NOC is monitoring the status of IP traffic around the router to minimize
customer traffic disruption. 

-----Original Message-----
From: outages-discussion-bounces at outages.org
[mailto:outages-discussion-bounces at outages.org] On Behalf Of Frank Bulk
Sent: Thursday, February 09, 2012 10:22 PM
To: 'Mark Kamichoff'
Cc: outages-discussion at outages.org
Subject: Re: [Outages-discussion] IPv6 outages overnight

Thanks, I'm glad to hear it just wasn't me.  My access was ipv6.juniper.net

was uninterrupted during that time.  

Frank

-----Original Message-----

From: Mark Kamichoff  <mailto:[mailto:prox at prolixium.com]>
[mailto:prox at prolixium.com] 

Sent: Thursday, February 09, 2012 7:32 PM

To: Frank Bulk

Cc:  <mailto:outages-discussion at outages.org> outages-discussion at outages.org

Subject: Re: [Outages-discussion] IPv6 outages overnight

Hi Frank - 

On Thu, Feb 09, 2012 at 06:32:08PM -0600, Frank Bulk wrote:

> I monitor over 30 IPv6 websites and Wednesday night between 11:52 pm

> and Thursday morning 12:50 am Central four of those sites became

> inaccessible almost all at the same time.

>              ipv6.t-mobile.com

>               <http://www.t-home.hu> www.t-home.hu [just the one AAAA:
2001:4c48:2:46::1b:1]

>               <http://www.twtelecom.com> www.twtelecom.com

>               <http://www.t-online.de> www.t-online.de

> Those sites (currently) go over different transit links (two over

> Sprint, two over Level3).

> 

> Here are traceroutes from 2607:fe28:10::/48.  Unfortunately I was not

> awake when this happened, so I couldn't see where it failed.  BGPlay

> on the IPv6 and IPv4 prefixes I tried don't show any routing events at

> our upstream transit during those times.

> 

> Anyone else see anything around that time?  

A few of my SmokePing (fping6) graphs showed gaps to a few nodes around

that same time (ipv6.juniper.net, ipv6.akamai.com, etc.).  Judging by

traceroutes I believe it was Level3-related.

- Mark

-- 

Mark Kamichoff

 <mailto:prox at prolixium.com> prox at prolixium.com

 <http://www.prolixium.com/> http://www.prolixium.com/

_______________________________________________

Outages-discussion mailing list

 <mailto:Outages-discussion at outages.org> Outages-discussion at outages.org

 <https://puck.nether.net/mailman/listinfo/outages-discussion>
https://puck.nether.net/mailman/listinfo/outages-discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20120210/6659d099/attachment.html>