<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoPlainText">Level 3 had a router go south after some network maintenance in New York around the time you indicated. It might be related:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-collapse:collapse">
<tbody>
<tr>
<td width="30%" valign="top" style="width:30.0%;padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Network Event Summary:<o:p></o:p></span></p>
</td>
<td style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">A possible network router isolation occurred after scheduled maintenance in New York City, NY impacting IP traffic
<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Event Ticket ID:<o:p></o:p></span></p>
</td>
<td style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">5379276
<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Market Area Affected:<o:p></o:p></span></p>
</td>
<td style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">New York City, NY
<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Ticket Create Date:<o:p></o:p></span></p>
</td>
<td style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">2/9/12 6:36:02 AM GMT
<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Impacted For:<o:p></o:p></span></p>
</td>
<td style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">2 hours 0 minutes</span></b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">
<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Event Status:<o:p></o:p></span></p>
</td>
<td style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Restored
<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Resolve Date:<o:p></o:p></span></p>
</td>
<td style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">2/9/12 10:02:09 AM GMT
<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Time Since Last Update:<o:p></o:p></span></p>
</td>
<td style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">20 hours 47 minutes</span></b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">
<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td colspan="2" valign="top" style="padding:2.25pt 3.75pt 1.5pt 3.75pt">
<p class="MsoNormal" style="line-height:18.0pt"><b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">There are currently no reported impacts to your services caused by this network event. If you are experiencing a service disruption,
please report the issue to Level 3 by submitting a ticket via the MyLevel3 Portal or calling 877-4LEVEL3.</span></b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="line-height:18.0pt"><span lang="EN" style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Recent Notes<o:p></o:p></span></p>
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="99%" style="width:99.0%;border-collapse:collapse;border:none">
<tbody>
<tr>
<td width="27%" style="width:27.0%;border-top:solid black 1.0pt;border-left:solid black 1.0pt;border-bottom:none;border-right:none;background:#3C3C3C;padding:1.5pt 1.5pt 1.5pt 1.5pt">
<p class="MsoNormal" style="line-height:18.0pt"><b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:white">Date
<o:p></o:p></span></b></p>
</td>
<td width="73%" style="width:73.0%;border-top:solid black 1.0pt;border-left:none;border-bottom:none;border-right:solid black 1.0pt;background:#3C3C3C;padding:1.5pt 1.5pt 1.5pt 1.5pt">
<p class="MsoNormal" style="line-height:18.0pt"><b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:white">Notes
<o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td valign="top" style="border:none;border-left:solid black 1.0pt;background:white;padding:1.5pt 1.5pt 1.5pt 1.5pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">2/9/12 7:52:53 AM GMT
<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-right:solid black 1.0pt;background:white;padding:1.5pt 1.5pt 1.5pt 1.5pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">The network router has restored to normal operation.
<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td valign="top" style="border:none;border-left:solid black 1.0pt;background:#EEEEEE;padding:1.5pt 1.5pt 1.5pt 1.5pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">2/9/12 7:46:07 AM GMT
<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-right:solid black 1.0pt;background:#EEEEEE;padding:1.5pt 1.5pt 1.5pt 1.5pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">IP NOC advised the router has restored and the line cards are active. IP NOC and Ops Engineering are waiting for route convergence of
IP traffic signifying the incorporation of the router back into the network. <o:p>
</o:p></span></p>
</td>
</tr>
<tr>
<td valign="top" style="border:none;border-left:solid black 1.0pt;background:white;padding:1.5pt 1.5pt 1.5pt 1.5pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">2/9/12 7:04:25 AM GMT
<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-right:solid black 1.0pt;background:white;padding:1.5pt 1.5pt 1.5pt 1.5pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">IP NOC has restricted all of the uplinks to the network router to stabilize the CPU load. Unfortunately, the manual isolation did not
stabilize the CPU load. Consequently, the field technician on site will power-cycle the unit to re-gain access to the router.
<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td valign="top" style="border-top:none;border-left:solid black 1.0pt;border-bottom:solid black 1.0pt;border-right:none;background:#EEEEEE;padding:1.5pt 1.5pt 1.5pt 1.5pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">2/9/12 6:42:52 AM GMT
<o:p></o:p></span></p>
</td>
<td valign="top" style="border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid black 1.0pt;background:#EEEEEE;padding:1.5pt 1.5pt 1.5pt 1.5pt">
<p class="MsoNormal" style="line-height:18.0pt"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">IP NOC advised that a network router in New York City experienced a memory failure and heavy CPU Load with effectively isolated the
unit from the network after the completion of a scheduled maintenance activity. IP NOC is unable to access the network router until the heavy CPU load has subsided. IP NOC is monitoring the status of IP traffic around the router to minimize customer traffic
disruption. <o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">-----Original Message-----<br>
From: outages-discussion-bounces@outages.org [mailto:outages-discussion-bounces@outages.org] On Behalf Of Frank Bulk<br>
Sent: Thursday, February 09, 2012 10:22 PM<br>
To: 'Mark Kamichoff'<br>
Cc: outages-discussion@outages.org<br>
Subject: Re: [Outages-discussion] IPv6 outages overnight</p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Thanks, I'm glad to hear it just wasn't me. My access was ipv6.juniper.net<o:p></o:p></p>
<p class="MsoPlainText">was uninterrupted during that time. <o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Frank<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">-----Original Message-----<o:p></o:p></p>
<p class="MsoPlainText">From: Mark Kamichoff <a href="mailto:[mailto:prox@prolixium.com]">
<span style="color:windowtext;text-decoration:none">[mailto:prox@prolixium.com]</span></a>
<o:p></o:p></p>
<p class="MsoPlainText">Sent: Thursday, February 09, 2012 7:32 PM<o:p></o:p></p>
<p class="MsoPlainText">To: Frank Bulk<o:p></o:p></p>
<p class="MsoPlainText">Cc: <a href="mailto:outages-discussion@outages.org"><span style="color:windowtext;text-decoration:none">outages-discussion@outages.org</span></a><o:p></o:p></p>
<p class="MsoPlainText">Subject: Re: [Outages-discussion] IPv6 outages overnight<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Hi Frank - <o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">On Thu, Feb 09, 2012 at 06:32:08PM -0600, Frank Bulk wrote:<o:p></o:p></p>
<p class="MsoPlainText">> I monitor over 30 IPv6 websites and Wednesday night between 11:52 pm<o:p></o:p></p>
<p class="MsoPlainText">> and Thursday morning 12:50 am Central four of those sites became<o:p></o:p></p>
<p class="MsoPlainText">> inaccessible almost all at the same time.<o:p></o:p></p>
<p class="MsoPlainText">> ipv6.t-mobile.com<o:p></o:p></p>
<p class="MsoPlainText">> <a href="http://www.t-home.hu"><span style="color:windowtext;text-decoration:none">www.t-home.hu</span></a> [just the one AAAA: 2001:4c48:2:46::1b:1]<o:p></o:p></p>
<p class="MsoPlainText">> <a href="http://www.twtelecom.com"><span style="color:windowtext;text-decoration:none">www.twtelecom.com</span></a><o:p></o:p></p>
<p class="MsoPlainText">> <a href="http://www.t-online.de"><span style="color:windowtext;text-decoration:none">www.t-online.de</span></a><o:p></o:p></p>
<p class="MsoPlainText">> Those sites (currently) go over different transit links (two over<o:p></o:p></p>
<p class="MsoPlainText">> Sprint, two over Level3).<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Here are traceroutes from 2607:fe28:10::/48. Unfortunately I was not<o:p></o:p></p>
<p class="MsoPlainText">> awake when this happened, so I couldn't see where it failed. BGPlay<o:p></o:p></p>
<p class="MsoPlainText">> on the IPv6 and IPv4 prefixes I tried don't show any routing events at<o:p></o:p></p>
<p class="MsoPlainText">> our upstream transit during those times.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Anyone else see anything around that time? <o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">A few of my SmokePing (fping6) graphs showed gaps to a few nodes around<o:p></o:p></p>
<p class="MsoPlainText">that same time (ipv6.juniper.net, ipv6.akamai.com, etc.). Judging by<o:p></o:p></p>
<p class="MsoPlainText">traceroutes I believe it was Level3-related.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">- Mark<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">-- <o:p></o:p></p>
<p class="MsoPlainText">Mark Kamichoff<o:p></o:p></p>
<p class="MsoPlainText"><a href="mailto:prox@prolixium.com"><span style="color:windowtext;text-decoration:none">prox@prolixium.com</span></a><o:p></o:p></p>
<p class="MsoPlainText"><a href="http://www.prolixium.com/"><span style="color:windowtext;text-decoration:none">http://www.prolixium.com/</span></a><o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">_______________________________________________<o:p></o:p></p>
<p class="MsoPlainText">Outages-discussion mailing list<o:p></o:p></p>
<p class="MsoPlainText"><a href="mailto:Outages-discussion@outages.org"><span style="color:windowtext;text-decoration:none">Outages-discussion@outages.org</span></a><o:p></o:p></p>
<p class="MsoPlainText"><a href="https://puck.nether.net/mailman/listinfo/outages-discussion"><span style="color:windowtext;text-decoration:none">https://puck.nether.net/mailman/listinfo/outages-discussion</span></a><o:p></o:p></p>
</div>
</body>
</html>