<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='color:#1F497D'>It seems to be off by an hour – my issue was from 11:52 pm Central (5:52 AM GMT) to 12:50 am Central (6:50 AM GMT), while Level3 first noted their issue at 6:36 AM GMT and marked it resolved by 7:52 AM GMT. Unless the issue started happening much earlier and it was Level3’s dropping of the router’s links resulted in traffic being re-routed and that’s when traffic could go through.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Frank<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Vinny_Abello@Dell.com [mailto:Vinny_Abello@Dell.com] <br><b>Sent:</b> Thursday, February 09, 2012 10:44 PM<br><b>To:</b> frnkblk@iname.com; prox@prolixium.com<br><b>Cc:</b> outages-discussion@outages.org<br><b>Subject:</b> RE: [Outages-discussion] IPv6 outages overnight<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoPlainText>Level 3 had a router go south after some network maintenance in New York around the time you indicated. It might be related:<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><table class=MsoNormalTable border=0 cellspacing=0 cellpadding=0 width="100%" style='width:100.0%;border-collapse:collapse'><tr><td width="30%" valign=top style='width:30.0%;padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>Network Event Summary:<o:p></o:p></span></p></td><td style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>A possible network router isolation occurred after scheduled maintenance in New York City, NY impacting IP traffic <o:p></o:p></span></p></td></tr><tr><td valign=top style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>Event Ticket ID:<o:p></o:p></span></p></td><td style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>5379276 <o:p></o:p></span></p></td></tr><tr><td valign=top style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>Market Area Affected:<o:p></o:p></span></p></td><td style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>New York City, NY <o:p></o:p></span></p></td></tr><tr><td valign=top style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>Ticket Create Date:<o:p></o:p></span></p></td><td style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>2/9/12 6:36:02 AM GMT <o:p></o:p></span></p></td></tr><tr><td valign=top style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>Impacted For:<o:p></o:p></span></p></td><td style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><b><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>2 hours 0 minutes</span></b><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'> <o:p></o:p></span></p></td></tr><tr><td valign=top style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>Event Status:<o:p></o:p></span></p></td><td style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>Restored <o:p></o:p></span></p></td></tr><tr><td valign=top style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>Resolve Date:<o:p></o:p></span></p></td><td style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>2/9/12 10:02:09 AM GMT <o:p></o:p></span></p></td></tr><tr><td valign=top style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>Time Since Last Update:<o:p></o:p></span></p></td><td style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><b><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>20 hours 47 minutes</span></b><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'> <o:p></o:p></span></p></td></tr><tr><td colspan=2 valign=top style='padding:2.25pt 3.75pt 1.5pt 3.75pt'><p class=MsoNormal style='line-height:18.0pt'><b><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>There are currently no reported impacts to your services caused by this network event. If you are experiencing a service disruption, please report the issue to Level 3 by submitting a ticket via the MyLevel3 Portal or calling 877-4LEVEL3.</span></b><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'><o:p></o:p></span></p></td></tr></table><p class=MsoNormal style='line-height:18.0pt'><span lang=EN style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>Recent Notes<o:p></o:p></span></p><table class=MsoNormalTable border=0 cellspacing=0 cellpadding=0 width="99%" style='width:99.0%;border-collapse:collapse'><tr><td width="27%" style='width:27.0%;border-top:solid black 1.0pt;border-left:solid black 1.0pt;border-bottom:none;border-right:none;background:#3C3C3C;padding:1.5pt 1.5pt 1.5pt 1.5pt'><p class=MsoNormal style='line-height:18.0pt'><b><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:white'>Date <o:p></o:p></span></b></p></td><td width="73%" style='width:73.0%;border-top:solid black 1.0pt;border-left:none;border-bottom:none;border-right:solid black 1.0pt;background:#3C3C3C;padding:1.5pt 1.5pt 1.5pt 1.5pt'><p class=MsoNormal style='line-height:18.0pt'><b><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:white'>Notes <o:p></o:p></span></b></p></td></tr><tr><td valign=top style='border:none;border-left:solid black 1.0pt;background:white;padding:1.5pt 1.5pt 1.5pt 1.5pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>2/9/12 7:52:53 AM GMT <o:p></o:p></span></p></td><td valign=top style='border:none;border-right:solid black 1.0pt;background:white;padding:1.5pt 1.5pt 1.5pt 1.5pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>The network router has restored to normal operation. <o:p></o:p></span></p></td></tr><tr><td valign=top style='border:none;border-left:solid black 1.0pt;background:#EEEEEE;padding:1.5pt 1.5pt 1.5pt 1.5pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>2/9/12 7:46:07 AM GMT <o:p></o:p></span></p></td><td valign=top style='border:none;border-right:solid black 1.0pt;background:#EEEEEE;padding:1.5pt 1.5pt 1.5pt 1.5pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>IP NOC advised the router has restored and the line cards are active. IP NOC and Ops Engineering are waiting for route convergence of IP traffic signifying the incorporation of the router back into the network. <o:p></o:p></span></p></td></tr><tr><td valign=top style='border:none;border-left:solid black 1.0pt;background:white;padding:1.5pt 1.5pt 1.5pt 1.5pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>2/9/12 7:04:25 AM GMT <o:p></o:p></span></p></td><td valign=top style='border:none;border-right:solid black 1.0pt;background:white;padding:1.5pt 1.5pt 1.5pt 1.5pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>IP NOC has restricted all of the uplinks to the network router to stabilize the CPU load. Unfortunately, the manual isolation did not stabilize the CPU load. Consequently, the field technician on site will power-cycle the unit to re-gain access to the router. <o:p></o:p></span></p></td></tr><tr><td valign=top style='border-top:none;border-left:solid black 1.0pt;border-bottom:solid black 1.0pt;border-right:none;background:#EEEEEE;padding:1.5pt 1.5pt 1.5pt 1.5pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>2/9/12 6:42:52 AM GMT <o:p></o:p></span></p></td><td valign=top style='border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;background:#EEEEEE;padding:1.5pt 1.5pt 1.5pt 1.5pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F'>IP NOC advised that a network router in New York City experienced a memory failure and heavy CPU Load with effectively isolated the unit from the network after the completion of a scheduled maintenance activity. IP NOC is unable to access the network router until the heavy CPU load has subsided. IP NOC is monitoring the status of IP traffic around the router to minimize customer traffic disruption. <o:p></o:p></span></p></td></tr></table><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>-----Original Message-----<br>From: <a href="mailto:outages-discussion-bounces@outages.org">outages-discussion-bounces@outages.org</a> <a href="mailto:[mailto:outages-discussion-bounces@outages.org]">[mailto:outages-discussion-bounces@outages.org]</a> On Behalf Of Frank Bulk<br>Sent: Thursday, February 09, 2012 10:22 PM<br>To: 'Mark Kamichoff'<br>Cc: <a href="mailto:outages-discussion@outages.org">outages-discussion@outages.org</a><br>Subject: Re: [Outages-discussion] IPv6 outages overnight<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Thanks, I'm glad to hear it just wasn't me. My access was ipv6.juniper.net<o:p></o:p></p><p class=MsoPlainText>was uninterrupted during that time. <o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Frank<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>-----Original Message-----<o:p></o:p></p><p class=MsoPlainText>From: Mark Kamichoff <a href="mailto:[mailto:prox@prolixium.com]"><span style='color:windowtext;text-decoration:none'>[mailto:prox@prolixium.com]</span></a> <o:p></o:p></p><p class=MsoPlainText>Sent: Thursday, February 09, 2012 7:32 PM<o:p></o:p></p><p class=MsoPlainText>To: Frank Bulk<o:p></o:p></p><p class=MsoPlainText>Cc: <a href="mailto:outages-discussion@outages.org"><span style='color:windowtext;text-decoration:none'>outages-discussion@outages.org</span></a><o:p></o:p></p><p class=MsoPlainText>Subject: Re: [Outages-discussion] IPv6 outages overnight<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Hi Frank - <o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>On Thu, Feb 09, 2012 at 06:32:08PM -0600, Frank Bulk wrote:<o:p></o:p></p><p class=MsoPlainText>> I monitor over 30 IPv6 websites and Wednesday night between 11:52 pm<o:p></o:p></p><p class=MsoPlainText>> and Thursday morning 12:50 am Central four of those sites became<o:p></o:p></p><p class=MsoPlainText>> inaccessible almost all at the same time.<o:p></o:p></p><p class=MsoPlainText>> ipv6.t-mobile.com<o:p></o:p></p><p class=MsoPlainText>> <a href="http://www.t-home.hu"><span style='color:windowtext;text-decoration:none'>www.t-home.hu</span></a> [just the one AAAA: 2001:4c48:2:46::1b:1]<o:p></o:p></p><p class=MsoPlainText>> <a href="http://www.twtelecom.com"><span style='color:windowtext;text-decoration:none'>www.twtelecom.com</span></a><o:p></o:p></p><p class=MsoPlainText>> <a href="http://www.t-online.de"><span style='color:windowtext;text-decoration:none'>www.t-online.de</span></a><o:p></o:p></p><p class=MsoPlainText>> Those sites (currently) go over different transit links (two over<o:p></o:p></p><p class=MsoPlainText>> Sprint, two over Level3).<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> Here are traceroutes from 2607:fe28:10::/48. Unfortunately I was not<o:p></o:p></p><p class=MsoPlainText>> awake when this happened, so I couldn't see where it failed. BGPlay<o:p></o:p></p><p class=MsoPlainText>> on the IPv6 and IPv4 prefixes I tried don't show any routing events at<o:p></o:p></p><p class=MsoPlainText>> our upstream transit during those times.<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> Anyone else see anything around that time? <o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>A few of my SmokePing (fping6) graphs showed gaps to a few nodes around<o:p></o:p></p><p class=MsoPlainText>that same time (ipv6.juniper.net, ipv6.akamai.com, etc.). Judging by<o:p></o:p></p><p class=MsoPlainText>traceroutes I believe it was Level3-related.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>- Mark<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>-- <o:p></o:p></p><p class=MsoPlainText>Mark Kamichoff<o:p></o:p></p><p class=MsoPlainText><a href="mailto:prox@prolixium.com"><span style='color:windowtext;text-decoration:none'>prox@prolixium.com</span></a><o:p></o:p></p><p class=MsoPlainText><a href="http://www.prolixium.com/"><span style='color:windowtext;text-decoration:none'>http://www.prolixium.com/</span></a><o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>_______________________________________________<o:p></o:p></p><p class=MsoPlainText>Outages-discussion mailing list<o:p></o:p></p><p class=MsoPlainText><a href="mailto:Outages-discussion@outages.org"><span style='color:windowtext;text-decoration:none'>Outages-discussion@outages.org</span></a><o:p></o:p></p><p class=MsoPlainText><a href="https://puck.nether.net/mailman/listinfo/outages-discussion"><span style='color:windowtext;text-decoration:none'>https://puck.nether.net/mailman/listinfo/outages-discussion</span></a><o:p></o:p></p></div></body></html>