<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body bgcolor=white lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Access to ipv6.level3.com came up minutes after I sent the message, at 5:32 pm Central.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>From the complete trace I see that the next hop after Kansas City is Denver.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;color:windowtext'>nagios:/home/fbulk# tcptraceroute6 ipv6.level3.com<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;color:windowtext'>traceroute to ipv6.test.Level3.com (2001:1900:2018:3000::105) from xxxxxxxxx, port 80, from port 39309, 30 hops max, 60 bytes packets<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;color:windowtext'>1 router-core.mtcnet.net (2607:fe28:0:1000::1) 0.253 ms 0.317 ms 0.192 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;color:windowtext'>2 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 0.192 ms 0.139 ms 0.135 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;color:windowtext'>3 v6-premier.sxcy-mlx.fbnt.netins.net (2001:5f8:7f0a:2::1) 5.526 ms 5.508 ms 5.488 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;color:windowtext'>4 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 9.829 ms 9.606 ms 9.635 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;color:windowtext'>5 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 10.934 ms 10.868 ms 10.864 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;color:windowtext'>6 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 17.140 ms 17.085 ms 17.168 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;color:windowtext'>7 vl-4062.car2.Denver1.Level3.net (2001:1900:4:1::382) 30.076 ms 30.091 ms 30.036 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;color:windowtext'>8 vl-51.car1.Denver1.Level3.net (2001:1900:13:1::3) 35.225 ms 39.585 ms 30.033 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;color:windowtext'>9 Level3-MOSS.vl-956.car1.Denver1.Level3.net (2001:1900:4:2::fa) 30.552 ms 30.699 ms 30.779 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;color:windowtext'>10 2001:1900:2018:f000:1:0:1:202 (2001:1900:2018:f000:1:0:1:202) 31.697 ms 31.685 ms 31.453 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;color:windowtext'>11 ipv6.test.Level3.com (2001:1900:2018:3000::105) 32.317 ms [open] 32.400 ms 31.888 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;color:windowtext'>nagios:/home/fbulk#<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Frank<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:windowtext'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:windowtext'> outages-bounces@outages.org [mailto:outages-bounces@outages.org] <b>On Behalf Of </b>Frank Bulk<br><b>Sent:</b> Friday, October 05, 2012 5:30 PM<br><b>To:</b> 'Chris Stone'; outages@outages.org<br><b>Subject:</b> Re: [outages] Level3 IPv4 and IPv6 sessions bouncing in Denver, CO<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I’ve not been able to access ipv6.level3.com since 4:11 pm Central … gets stuck in Kansas<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>nagios:/home/fbulk# tcptraceroute6 ipv6.level3.com<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>traceroute to ipv6.test.Level3.com (2001:1900:2018:3000::105) from xxxxxxx, port 80, from port 43472, 30 hops max, 60 bytes packets<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>1 router-core.mtcnet.net (2607:fe28:0:1000::1) 0.328 ms 0.302 ms 0.254 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>2 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 0.312 ms 0.131 ms 0.105 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>3 v6-premier.sxcy-mlx.fbnt.netins.net (2001:5f8:7f0a:2::1) 5.529 ms 5.545 ms 5.478 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 9.515 ms 10.254 ms 9.524 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>5 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 10.914 ms 11.119 ms 10.884 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>6 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 17.358 ms 17.101 ms 17.128 ms<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>7 * * *<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>8 * * *<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>9 * * *<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Frank<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:windowtext'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:windowtext'> <a href="mailto:outages-bounces@outages.org">outages-bounces@outages.org</a> [<a href="mailto:outages-bounces@outages.org">mailto:outages-bounces@outages.org</a>] <b>On Behalf Of </b>Chris Stone<br><b>Sent:</b> Friday, October 05, 2012 5:08 PM<br><b>To:</b> <a href="mailto:outages@outages.org">outages@outages.org</a><br><b>Subject:</b> [outages] Level3 IPv4 and IPv6 sessions bouncing in Denver, CO<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-family:"Verdana","sans-serif"'>Seeing my Level3 BGP sessions bouncing since 15:13:48 MDT and tracing in from a CenturyLink circuit, dies with lots of packetloss on the 2nd to last hop. Have ticket open with Level3 - waiting to hear something back.<br><br>HOST: orion.axint.net Loss% Snt Last Avg Best Wrst StDev<br> 1.|-- 10.10.100.1 0.0% 50 0.5 0.5 0.4 0.7 0.0<br> 2.|-- 207.225.112.1 0.0% 50 41.0 50.0 39.4 181.7 27.6<br> 3.|-- 71.217.188.1 0.0% 50 39.9 41.5 39.3 87.4 6.8<br> 4.|-- 67.14.24.118 0.0% 50 40.7 44.4 39.7 100.5 12.0<br> 5.|-- 63.146.26.134 0.0% 50 59.6 60.8 58.7 92.8 5.3<br> 6.|-- 4.69.147.67 86.0% 50 84.2 277.0 73.0 1084. 366.3<br> 7.|-- ??? 100.0 50 0.0 0.0 0.0 0.0 0.0<br><br>Anyone else seeing L3 problems in Denver?<br><br><br>Chris</span><o:p></o:p></p></div></body></html>