<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">We have opened a case with CenturyLink/Level3 to see if they can put pressure on Verizon. It’s wrecking havoc with our Voip and VPN/Remote desktops.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>From:</b> Outages-discussion <outages-discussion-bounces@outages.org>
<b>On Behalf Of </b>David Rubin<br>
<b>Sent:</b> Thursday, June 18, 2020 10:24 AM<br>
<b>To:</b> Cary Wiedemann <carywiedemann@gmail.com><br>
<b>Cc:</b> outages-discussion@outages.org<br>
<b>Subject:</b> Re: [Outages-discussion] [outages] Fios in NYC area?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">I'm still having users reporting issues today - do we know if this is ongoing / if Verizon is aware of what's going on? I'm guessing they aren't - because why would they be?<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Jun 17, 2020 at 6:40 PM Cary Wiedemann via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">I've definitely seen loss from FiOS circuits in Manhattan and Long Island to many routes (examples: AT&T and Cox, but not Google's local 8.8.8.8 resolver) starting around 8:45pm EDT last night.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Some graphs:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">24-hour Long Island FiOS to Cox:<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><img border="0" width="561" height="87" style="width:5.8437in;height:.9062in" id="_x0000_i1025" src="cid:image001.png@01D6455E.26F6F710" alt="image.png"><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2-hour Long Island FiOS to Cox:<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><img border="0" width="561" height="87" style="width:5.8437in;height:.9062in" id="_x0000_i1026" src="cid:image002.png@01D6455E.26F6F710" alt="image.png"><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">2-hour Manhattan to FiOS to AT&T:<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><img border="0" width="561" height="85" style="width:5.8437in;height:.8854in" id="_x0000_i1027" src="cid:image003.png@01D6455E.26F6F710" alt="image.png"><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">And of course New York is one of the IPv6 FiOS regions that broke traditional traceroute, so it's tougher than normal to nail down which router is the culprit. I have some UDP MTRs running now which should hopefully answer that question.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I've seen several days of high latency and intermittent loss to many Verizon peers via their old
<a href="http://ALTER.NET" target="_blank">ALTER.NET</a> routers in the DC region too, which may or may not be related.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">No, it's not just you. But it's not the last-mile FiOS either. Looks like Verizon backbone routing woes.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">- Cary<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Jun 17, 2020 at 5:13 PM Izzy Goldstein - TeleGo via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black">getting complaints from customers using FiOS of packet loss to our data center<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black">but getting word there is an issue in NYC area with other data centers too?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p style="margin:0in;margin-bottom:.0001pt"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#434343">Izzy Goldstein </span></b><span style="font-size:9.5pt"><o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#434343">Chief Technology Officer </span></i><span style="font-size:9.5pt"><o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#434343">Main:
</span><u><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1155CC"><a href="tel:(212)%20477-1000" target="_blank"><span style="color:#1155CC">(212) 477-1000 x 2085</span></a></span></u><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1155CC">
</span><span style="font-size:9.5pt"><o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#434343">Direct:
</span><u><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1155CC"><a href="tel:(929)%20477-2085" target="_blank"><span style="color:#1155CC">(929) 477-2085</span></a></span></u><span style="font-size:9.5pt"><o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:white;text-align:start;word-spacing:0px">
<span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#434343">Website:</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1155CC">
</span><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222"><a href="http://www.telego.net/" target="_blank"><span style="font-size:10.0pt;color:#1155CC">www.telego.com</span></a></span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#434343">
</span><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222"><o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.5pt"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.5pt"><a href="http://www.telego.com/" target="_blank"><span style="color:windowtext;text-decoration:none"><o:p></o:p></span></a></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.5pt"><a href="http://www.telego.com/" target="_blank"><span style="color:windowtext;text-decoration:none"><o:p> </o:p></span></a></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><i><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#999999"><a href="http://www.telego.com/" target="_blank"><span style="color:#999999;text-decoration:none">Confidentiality Notice: This e-mail may
contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify us immediately by email reply and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the
material in this e-mail is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of TeleGo (T). Employees of TeleGo are expressly required not to make defamatory statements and
not to infringe or authorize any infringement of copyright or any other legal right by email communications. Any such communication is contrary to TeleGo policy and outside the scope of the employment of the individual concerned. TeleGo will not accept any
liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.</span><span style="font-size:9.5pt;font-family:"Calibri",sans-serif;color:windowtext;font-style:normal;text-decoration:none"><o:p></o:p></span></a></span></i></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.5pt"><a href="http://www.telego.com/" target="_blank"><span style="color:windowtext;text-decoration:none"><br>
<u><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#1155CC">TeleGo Hosted PBX</span></u><o:p></o:p></span></a></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><a href="http://www.telego.com/" target="_blank"><span style="color:windowtext;text-decoration:none">_______________________________________________<br>
Outages mailing list<br>
<u><span style="color:blue">Outages@outages.org</span></u><br>
<u><span style="color:blue">https://puck.nether.net/mailman/listinfo/outages</span></u><o:p></o:p></span></a></p>
</blockquote>
</div>
<p class="MsoNormal"><a href="http://www.telego.com/" target="_blank"><span style="color:windowtext;text-decoration:none">_______________________________________________<br>
Outages mailing list<br>
<u><span style="color:blue">Outages@outages.org</span></u><br>
<u><span style="color:blue">https://puck.nether.net/mailman/listinfo/outages</span></u><o:p></o:p></span></a></p>
</blockquote>
</div>
<p class="MsoNormal"><a href="http://www.telego.com/" target="_blank"><span style="color:windowtext;text-decoration:none"><br clear="all">
<o:p></o:p></span></a></p>
<div>
<p class="MsoNormal"><a href="http://www.telego.com/" target="_blank"><span style="color:windowtext;text-decoration:none"><o:p> </o:p></span></a></p>
</div>
<p class="MsoNormal"><a href="http://www.telego.com/" target="_blank"><span style="color:windowtext;text-decoration:none">--
<o:p></o:p></span></a></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><a href="http://www.telego.com/" target="_blank"><span style="color:windowtext;text-decoration:none">David Rubin <o:p></o:p></span></a></p>
</div>
<p class="MsoNormal"><a href="http://www.telego.com/" target="_blank"><span style="color:windowtext;text-decoration:none">Cubex Group
<br>
223 Wall Street - PMB #267 <br>
Huntington, NY 11743 <br>
646.221.3283<o:p></o:p></span></a></p>
</div>
<div>
<div>
<p class="MsoNormal"><a href="http://www.telego.com/" target="_blank"><span style="color:windowtext;text-decoration:none"><o:p> </o:p></span></a></p>
</div>
<div>
<p class="MsoNormal"><b><span style="color:red"><a href="http://www.telego.com/" target="_blank"><span style="color:red;text-decoration:none">If you have a support-related issue, please email to
<u><span style="color:blue">support@cubexgroup.com</span></u>.</span><span style="color:windowtext;font-weight:normal;text-decoration:none"><o:p></o:p></span></a></span></b></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>