<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">It seems like the Charter issue this morning was “regional” (perhaps a rather small “region”). It sounds like it primarily affected the Madison, WI area, which we hang off of (we’re in Verona, WI).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">We re-enabled our BGP advertisements via Charter about 15 minutes ago and everything seems to be operating normally. Yes, we like to hold off for a while after the up-stream gives the all-clear, just in case
there are some lingering issues that still need to shake out.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks to those who replied with observations and offers of assistance, both on and off-list.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Jeremy Charles <br>
<b>Sent:</b> Friday, February 24, 2017 10:01 AM<br>
<b>To:</b> 'outages@outages.org' <outages@outages.org><br>
<b>Subject:</b> RE: Charter issues this morning?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Update received from Charter just a moment ago.... I think we’ll give it a bit for things to settle down before we put traffic back on them.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:150%">
<b><span style="font-family:"Arial",sans-serif;color:black">STATUS</span></b><span style="font-family:"Arial",sans-serif;color:black">: The SE NOC has been advised that the bad fiber has been re-spliced at this time. Following the completion of these actions,
both fibers have been verified and the original links have restored. If you have any questions or concerns please call 866-603-3199 and reference your ticket number.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Jeremy Charles <br>
<b>Sent:</b> Friday, February 24, 2017 8:45 AM<br>
<b>To:</b> 'outages@outages.org' <<a href="mailto:outages@outages.org">outages@outages.org</a>><br>
<b>Subject:</b> Charter issues this morning?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Anyone else using Charter Internet have issues with connectivity this morning?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We (AS 10359) have site-to-site VPN tunnels to about 150 customers across the US. We started seeing loss of connectivity to about 20% of them at about 06:25 CST. When we filtered out all advertisements sent/received via Charter, we regained
connectivity to everyone. (We have two ISPs in addition to Charter.)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Additionally, our total aggregate Internet consumption jumped from about 200 Mbps to 500 Mbps as soon as Charter was taken out of the mix. It looks like a lot of traffic was preferring to come to us via Charter (which is normal) but just
wasn’t making it here.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">One of our engineers waited on hold for Charter for about 20 minutes and then gave up to go get coffee. :-)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><img border="0" width="495" height="251" id="Picture_x0020_1" src="cid:image001.png@01D28E96.55CAAC30" alt="https://mrtg1.epic.com/mrtg/internet/inbound-borders.png"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">==<o:p></o:p></p>
<p class="MsoNormal">Jeremy Charles<o:p></o:p></p>
<p class="MsoNormal">Epic’s Computer and Technology Services Division<o:p></o:p></p>
<p class="MsoNormal"><a href="mailto:jcharles@epic.com">jcharles@epic.com</a><o:p></o:p></p>
<p class="MsoNormal">608-271-9000<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>