[outages] Packet loss to 8.8.8.8 and other google IPs (Dallas)
Justin
fourzerofour at gmail.com
Tue Aug 30 12:25:45 EDT 2016
I am seeing loss and high latency to Google in Atlanta and Miami over
Level3. When I finally spoke with someone within Level3 they mentioned they
are moving peerings and is a part of the TWT/L3 merger. We are working with
them at a higher level to determine a resolution.
On Mon, Aug 29, 2016 at 4:41 PM, Andrew Duey via Outages <
outages at outages.org> wrote:
> We have been seeing this off and on to many google sites (probably most if
> not all) but we have tested to www.google.com, www.youtube.com, 8.8.8.8,
> & 8.8.4.4 and been seeing packet loss 15-40%. It is specific to our Level
> 3 connection and we do not see it on our HE connection (when using
> alternate IP space to avoid Level 3).
>
> We've seen this off and on for over a month, but noticed it pronounced on
> Friday and pronounced today.
>
> I opened a ticket with Level 3, they said that it's a Google problem and
> to contact them. Since I am not a direct google IP services customer I
> can't do much more than e-mail noc at google.com and see what happens (which
> I already have).
>
> Our Level 3 upstream router is in Kansas City and we see the issue after
> the Dallas Level 3 router:
>
> tracert 8.8.8.8
> Tracing route to google-public-dns-a.google.com [8.8.8.8]
> over a maximum of 30 hops:
> 1 1 ms 2 ms 2 ms customer-ip-8-34-97-1.widerang
> ebroadband.net [8.34.97.1]
> 2 4 ms 4 ms 2 ms wrb-router-8-33-2-121.widerang
> ebroadband.net [8.33.2.121]
> 3 60 ms 11 ms 142 ms ge-6-15.car1.kansascity1.level3.net
> [4.53.32.105]
> 4 16 ms 17 ms 19 ms google-level3-100g.dallas1.level3.net
> [4.68.72.138]
> 5 16 ms 16 ms * 108.170.240.193
> 6 19 ms 18 ms 18 ms 64.233.175.9
> 7 17 ms 17 ms * google-public-dns-a.google.com [8.8.8.8]
> 8 17 ms 17 ms 16 ms google-public-dns-a.google.com [8.8.8.8]
> Trace complete.
>
> Damian: If you can assist me with details contact me off-list and I'd be
> happy to include any Google information in the current Level 3 ticket and
> escalate it again. At this point Level 3 has said everything is fine until
> 4.68.72.138 therefore the issue an internal Google network issue starting
> at 108-170.240.193.
>
> --Andrew Duey
>
> On Mon, Aug 29, 2016 at 3:24 PM, Matthew Blackstone via Outages <
> outages at outages.org> wrote:
>
>> Google public dns is anycast so trying to hit from a different location
>> will most likely result in a different destination.
>> https://developers.google.com/speed/public-dns/faq
>>
>>
>>
>> Matt Blackstone
>> Network Engineer
>>
>>
>> On Mon, Aug 29, 2016 at 1:54 PM, Damian Menscher via Outages <
>> outages at outages.org> wrote:
>>
>>> This appears to be a problem on the Level3 network in the DFW region,
>>> which is why only some of you see it. (Those who don't see it are in other
>>> regions, peering with Google directly, or using other transit providers.)
>>>
>>> For those seeing issues, I suggest escalating to Level3, perhaps after
>>> trying a reverse-traceroute from http://lg.level3.net/ (I can't do this
>>> test since you didn't specify your source IP.)
>>>
>>> Damian
>>>
>>> On Mon, Aug 29, 2016 at 11:14 AM, Dan White <dwhite at olp.net> wrote:
>>>
>>>> We are seeing packet loss to google hosts, hosted in Dallas. Any anyone
>>>> seeing this?
>>>>
>>>> MTR output:
>>>>
>>>> Host Loss% Snt Last Avg Best Wrst
>>>> StDev
>>>> 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2
>>>> 0.7
>>>> 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3
>>>> 27.8 4.9
>>>> 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3
>>>> 22.0 5.3
>>>> 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6
>>>> 33.3 6.8
>>>> 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5
>>>> 64.0 10.5
>>>> 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1
>>>> 13.6 4.1
>>>> 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2
>>>> 27.5 5.5
>>>> 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9
>>>> 44.5 7.8
>>>> 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5
>>>> 20.5 0.0
>>>> 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6
>>>> 20.7 0.0
>>>> 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4
>>>> 29.6 1.5
>>>> 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7
>>>> 20.9 0.0
>>>> 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7
>>>> 20.8 0.0
>>>> 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5
>>>> 25.9 0.9
>>>>
>>>> The problematic hop appears to be either 10 or 11 (level3-100g).
>>>> Compare to:
>>>>
>>>> Host Loss% Snt Last Avg Best Wrst
>>>> StDev
>>>> 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6
>>>> 0.5
>>>> 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3
>>>> 29.9 5.6
>>>> 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3
>>>> 26.1 4.9
>>>> 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6
>>>> 20.0 3.7
>>>> 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6
>>>> 44.1 9.8
>>>> 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1
>>>> 14.9 4.3
>>>> 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1
>>>> 68.3 11.3
>>>> 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9
>>>> 62.3 8.1
>>>> 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5
>>>> 20.6 0.0
>>>> 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3
>>>> 68.0 7.1
>>>> 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2
>>>> 22.8 0.3
>>>>
>>>>
>>>> --
>>>> Dan White
>>>> BTC Broadband
>>>> Network Admin Lead
>>>> Ph 918.366.0248 (direct) main: (918)366-8000
>>>> Fax 918.366.6610 email: dwhite at olp.net
>>>> http://www.btcbroadband.com
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "external-netops" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to external-netops+unsubscribe at google.com.
>>>> To post to this group, send email to external-netops at google.com.
>>>> Visit this group at https://groups.google.com/a/go
>>>> ogle.com/group/external-netops/.
>>>> For more options, visit https://groups.google.com/a/google.com/d/optout
>>>> .
>>>>
>>>
>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>
>
> --
> Thanks,
> Andrew Duey
> Network Engineer, Widerange Broadband
> Direct: 402-327-1101
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20160830/f9d00aab/attachment.htm>
More information about the Outages
mailing list