<div dir="ltr">I am seeing loss and high latency to Google in Atlanta and Miami over Level3. When I finally spoke with someone within Level3 they mentioned they are moving peerings and is a part of the TWT/L3 merger. We are working with them at a higher level to determine a resolution.</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 29, 2016 at 4:41 PM, Andrew Duey via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>We have been seeing this off and on to many google sites (probably most if not all) but we have tested to <a href="http://www.google.com" target="_blank">www.google.com</a>, <a href="http://www.youtube.com" target="_blank">www.youtube.com</a>, 8.8.8.8, & 8.8.4.4 and been seeing packet loss 15-40%. It is specific to our Level 3 connection and we do not see it on our HE connection (when using alternate IP space to avoid Level 3).</div><div><br></div><div>We've seen this off and on for over a month, but noticed it pronounced on Friday and pronounced today.</div><div><br></div><div>I opened a ticket with Level 3, they said that it's a Google problem and to contact them. Since I am not a direct google IP services customer I can't do much more than e-mail <a href="mailto:noc@google.com" target="_blank">noc@google.com</a> and see what happens (which I already have). </div><div><br></div><div>Our Level 3 upstream router is in Kansas City and we see the issue after the Dallas Level 3 router:</div><div><br></div><div><div>tracert 8.8.8.8</div><div>Tracing route to <a href="http://google-public-dns-a.google.com/" target="_blank">google-public-dns-a.google.com</a> [8.8.8.8]<br>over a maximum of 30 hops:</div><div> 1 1 ms 2 ms 2 ms <a href="http://customer-ip-8-34-97-1.widerangebroadband.net/" target="_blank">customer-ip-8-34-97-1.widerang<wbr>ebroadband.net</a> [8.34.97.1]<br> 2 4 ms 4 ms 2 ms <a href="http://wrb-router-8-33-2-121.widerangebroadband.net/" target="_blank">wrb-router-8-33-2-121.widerang<wbr>ebroadband.net</a> [8.33.2.121]<br> 3 60 ms 11 ms 142 ms <a href="http://ge-6-15.car1.kansascity1.level3.net/" target="_blank">ge-6-15.car1.kansascity1.level<wbr>3.net</a> [4.53.32.105]<br> 4 16 ms 17 ms 19 ms <a href="http://google-level3-100g.dallas1.level3.net/" target="_blank">google-level3-100g.dallas1.lev<wbr>el3.net</a> [4.68.72.138]<br> 5 16 ms 16 ms * 108.170.240.193<br> 6 19 ms 18 ms 18 ms 64.233.175.9<br> 7 17 ms 17 ms * <a href="http://google-public-dns-a.google.com/" target="_blank">google-public-dns-a.google.com</a> [8.8.8.8]<br> 8 17 ms 17 ms 16 ms <a href="http://google-public-dns-a.google.com/" target="_blank">google-public-dns-a.google.com</a> [8.8.8.8]</div><div>Trace complete.</div><div><br></div><div>Damian: If you can assist me with details contact me off-list and I'd be happy to include any Google information in the current Level 3 ticket and escalate it again. At this point Level 3 has said everything is fine until 4.68.72.138 therefore the issue an internal Google network issue starting at 108-170.240.193.</div><div><br></div><div>--Andrew Duey</div></div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Mon, Aug 29, 2016 at 3:24 PM, Matthew Blackstone via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">Google public dns is anycast so trying to hit from a different location will most likely result in a different destination.</div><div class="gmail_default"><font face="verdana, sans-serif"><a href="https://developers.google.com/speed/public-dns/faq" target="_blank">https://developers.google.com/<wbr>speed/public-dns/faq</a></font><br></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_extra"><br clear="all"><div><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><div dir="ltr">Matt Blackstone<div>Network Engineer</div><div><br></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote"><div><div>On Mon, Aug 29, 2016 at 1:54 PM, Damian Menscher via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div><div><div dir="ltr">This appears to be a problem on the Level3 network in the DFW region, which is why only some of you see it. (Those who don't see it are in other regions, peering with Google directly, or using other transit providers.)<div><br></div><div>For those seeing issues, I suggest escalating to Level3, perhaps after trying a reverse-traceroute from <a href="http://lg.level3.net/" target="_blank">http://lg.level3.net/</a> (I can't do this test since you didn't specify your source IP.)</div><div><br></div><div>Damian</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Mon, Aug 29, 2016 at 11:14 AM, Dan White <span dir="ltr"><<a href="mailto:dwhite@olp.net" target="_blank">dwhite@olp.net</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div><div>We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this?<br>
<br>
MTR output:<br>
<br>
Host Loss% Snt Last Avg Best Wrst StDev<br>
1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7<br>
2. <a href="http://olp-67-217-152-5.olp.net" rel="noreferrer" target="_blank">olp-67-217-152-5.olp.net</a> 0.0% 51 0.4 3.7 0.3 27.8 4.9<br>
3. <a href="http://olp-67-217-158-29.olp.net" rel="noreferrer" target="_blank">olp-67-217-158-29.olp.net</a> 0.0% 50 0.3 4.5 0.3 22.0 5.3<br>
4. <a href="http://olp-67-217-151-125.olp.net" rel="noreferrer" target="_blank">olp-67-217-151-125.olp.net</a> 0.0% 50 0.6 4.9 0.6 33.3 6.8<br>
5. <a href="http://olp-67-217-158-30.olp.net" rel="noreferrer" target="_blank">olp-67-217-158-30.olp.net</a> 0.0% 50 0.6 6.4 0.5 64.0 10.5<br>
6. <a href="http://olp-67-217-158-21.olp.net" rel="noreferrer" target="_blank">olp-67-217-158-21.olp.net</a> 0.0% 50 11.4 4.4 1.1 13.6 4.1<br>
7. <a href="http://xe-8-3-0.bar2.tulsa2.level3.net" rel="noreferrer" target="_blank">xe-8-3-0.bar2.tulsa2.level3.ne<wbr>t</a> 0.0% 50 1.2 2.7 1.2 27.5 5.5<br>
8. <a href="http://ae-0-11.bar1.tulsa2.level3.net" rel="noreferrer" target="_blank">ae-0-11.bar1.tulsa2.level3.net</a><wbr> 0.0% 50 2.0 4.1 1.9 44.5 7.8<br>
9. <a href="http://ae-3-3.ebr4.dallas1.level3.net" rel="noreferrer" target="_blank">ae-3-3.ebr4.dallas1.level3.net</a> 91.8% 50 20.5 20.5 20.5 20.5 0.0<br>
10. <a href="http://ae-8-0.ear3.dallas1.level3.net" rel="noreferrer" target="_blank">ae-8-0.ear3.dallas1.level3.net</a> 93.9% 50 20.6 20.6 20.6 20.7 0.0<br>
11. google-level3-100g.dallas1.lev<wbr>e28.0% 50 20.5 20.7 20.4 29.6 1.5<br>
12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0<br>
<a href="tel:13.%2064.233.175.9" value="+13642331759" target="_blank">13. 64.233.175.9</a> 26.0% 50 20.7 20.7 20.7 20.8 0.0<br>
14. <a href="http://google-public-dns-a.google.com" rel="noreferrer" target="_blank">google-public-dns-a.google.com</a> 24.0% 50 20.5 20.6 20.5 25.9 0.9<br>
<br>
The problematic hop appears to be either 10 or 11 (level3-100g). Compare to:<br>
<br>
Host Loss% Snt Last Avg Best Wrst StDev<br>
1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5<br>
2. <a href="http://olp-67-217-152-5.olp.net" rel="noreferrer" target="_blank">olp-67-217-152-5.olp.net</a> 0.0% 124 3.8 4.3 0.3 29.9 5.6<br>
3. <a href="http://olp-67-217-158-29.olp.net" rel="noreferrer" target="_blank">olp-67-217-158-29.olp.net</a> 0.0% 124 0.3 3.5 0.3 26.1 4.9<br>
4. <a href="http://olp-67-217-151-125.olp.net" rel="noreferrer" target="_blank">olp-67-217-151-125.olp.net</a> 0.0% 124 6.6 3.3 0.6 20.0 3.7<br>
5. <a href="http://olp-67-217-158-30.olp.net" rel="noreferrer" target="_blank">olp-67-217-158-30.olp.net</a> 0.0% 124 0.6 7.1 0.6 44.1 9.8<br>
6. <a href="http://olp-67-217-158-21.olp.net" rel="noreferrer" target="_blank">olp-67-217-158-21.olp.net</a> 0.8% 124 11.9 4.8 1.1 14.9 4.3<br>
7. <a href="http://xe-8-3-0.bar2.tulsa2.level3.net" rel="noreferrer" target="_blank">xe-8-3-0.bar2.tulsa2.level3.ne<wbr>t</a> 0.0% 123 1.3 4.0 1.1 68.3 11.3<br>
8. <a href="http://ae-0-11.bar1.tulsa2.level3.net" rel="noreferrer" target="_blank">ae-0-11.bar1.tulsa2.level3.net</a><wbr> 0.8% 123 2.1 3.9 1.9 62.3 8.1<br>
9. <a href="http://ae-3-3.ebr4.dallas1.level3.net" rel="noreferrer" target="_blank">ae-3-3.ebr4.dallas1.level3.net</a> 94.3% 123 20.5 20.6 20.5 20.6 0.0<br>
10. ae-2-3512.edge2.dallas1.level3<wbr>. 0.0% 123 20.4 22.1 20.3 68.0 7.1<br>
11. <a href="http://b.resolvers.level3.net" rel="noreferrer" target="_blank">b.resolvers.level3.net</a> 0.0% 123 20.2 20.3 20.2 22.8 0.3</div></div><span><font color="#888888"><div><div><br>
<br>
-- <br>
Dan White<br>
BTC Broadband<br>
Network Admin Lead<br>
Ph <a href="tel:918.366.0248" value="+19183660248" target="_blank">918.366.0248</a> (direct) main: <a href="tel:%28918%29366-8000" value="+19183668000" target="_blank">(918)366-8000</a><br>
Fax <a href="tel:918.366.6610" value="+19183666610" target="_blank">918.366.6610</a> email: <a href="mailto:dwhite@olp.net" target="_blank">dwhite@olp.net</a><br>
<a href="http://www.btcbroadband.com" rel="noreferrer" target="_blank">http://www.btcbroadband.com</a><br>
<br></div></div><span><font color="#888888">
-- <br>
You received this message because you are subscribed to the Google Groups "external-netops" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:external-netops%2Bunsubscribe@google.com" target="_blank">external-netops+unsubscribe@go<wbr>ogle.com</a>.<br>
To post to this group, send email to <a href="mailto:external-netops@google.com" target="_blank">external-netops@google.com</a>.<br>
Visit this group at <a href="https://groups.google.com/a/google.com/group/external-netops/" rel="noreferrer" target="_blank">https://groups.google.com/a/go<wbr>ogle.com/group/external-netops<wbr>/</a>.<br>
For more options, visit <a href="https://groups.google.com/a/google.com/d/optout" rel="noreferrer" target="_blank">https://groups.google.com/a/go<wbr>ogle.com/d/optout</a>.<br>
</font></span></font></span></blockquote></div><br></div>
<br></div></div><span>______________________________<wbr>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailma<wbr>n/listinfo/outages</a><br>
<br></span></blockquote></div><br></div></div>
<br>______________________________<wbr>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailma<wbr>n/listinfo/outages</a><br>
<br></blockquote></div><br><br clear="all"><br></div></div><span class="HOEnZb"><font color="#888888">-- <br><div data-smartmail="gmail_signature"><div>Thanks,</div><div>Andrew Duey</div><div>Network Engineer, Widerange Broadband</div><div>Direct: <a href="tel:402-327-1101" value="+14023271101" target="_blank">402-327-1101</a></div><div> </div></div>
</font></span></div>
<br>______________________________<wbr>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/<wbr>mailman/listinfo/outages</a><br>
<br></blockquote></div><br></div>