[outages] Packet loss to 8.8.8.8 and other google IPs (Dallas)
Andrew Duey
andrew.duey at widerangebroadband.net
Mon Aug 29 16:41:44 EDT 2016
We have been seeing this off and on to many google sites (probably most if
not all) but we have tested to www.google.com, www.youtube.com, 8.8.8.8, &
8.8.4.4 and been seeing packet loss 15-40%. It is specific to our Level 3
connection and we do not see it on our HE connection (when using alternate
IP space to avoid Level 3).
We've seen this off and on for over a month, but noticed it pronounced on
Friday and pronounced today.
I opened a ticket with Level 3, they said that it's a Google problem and to
contact them. Since I am not a direct google IP services customer I can't
do much more than e-mail noc at google.com and see what happens (which I
already have).
Our Level 3 upstream router is in Kansas City and we see the issue after
the Dallas Level 3 router:
tracert 8.8.8.8
Tracing route to google-public-dns-a.google.com [8.8.8.8]
over a maximum of 30 hops:
1 1 ms 2 ms 2 ms customer-ip-8-34-97-1.widerangebroadband.net
[8.34.97.1]
2 4 ms 4 ms 2 ms wrb-router-8-33-2-121.widerangebroadband.net
[8.33.2.121]
3 60 ms 11 ms 142 ms ge-6-15.car1.kansascity1.level3.net
[4.53.32.105]
4 16 ms 17 ms 19 ms google-level3-100g.dallas1.level3.net
[4.68.72.138]
5 16 ms 16 ms * 108.170.240.193
6 19 ms 18 ms 18 ms 64.233.175.9
7 17 ms 17 ms * google-public-dns-a.google.com [8.8.8.8]
8 17 ms 17 ms 16 ms google-public-dns-a.google.com [8.8.8.8]
Trace complete.
Damian: If you can assist me with details contact me off-list and I'd be
happy to include any Google information in the current Level 3 ticket and
escalate it again. At this point Level 3 has said everything is fine until
4.68.72.138 therefore the issue an internal Google network issue starting
at 108-170.240.193.
--Andrew Duey
On Mon, Aug 29, 2016 at 3:24 PM, Matthew Blackstone via Outages <
outages at outages.org> wrote:
> Google public dns is anycast so trying to hit from a different location
> will most likely result in a different destination.
> https://developers.google.com/speed/public-dns/faq
>
>
>
> Matt Blackstone
> Network Engineer
>
>
> On Mon, Aug 29, 2016 at 1:54 PM, Damian Menscher via Outages <
> outages at outages.org> wrote:
>
>> This appears to be a problem on the Level3 network in the DFW region,
>> which is why only some of you see it. (Those who don't see it are in other
>> regions, peering with Google directly, or using other transit providers.)
>>
>> For those seeing issues, I suggest escalating to Level3, perhaps after
>> trying a reverse-traceroute from http://lg.level3.net/ (I can't do this
>> test since you didn't specify your source IP.)
>>
>> Damian
>>
>> On Mon, Aug 29, 2016 at 11:14 AM, Dan White <dwhite at olp.net> wrote:
>>
>>> We are seeing packet loss to google hosts, hosted in Dallas. Any anyone
>>> seeing this?
>>>
>>> MTR output:
>>>
>>> Host Loss% Snt Last Avg Best Wrst
>>> StDev
>>> 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2
>>> 0.7
>>> 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3 27.8
>>> 4.9
>>> 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3 22.0
>>> 5.3
>>> 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6 33.3
>>> 6.8
>>> 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5 64.0
>>> 10.5
>>> 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1 13.6
>>> 4.1
>>> 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2 27.5
>>> 5.5
>>> 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9 44.5
>>> 7.8
>>> 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5 20.5
>>> 0.0
>>> 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6
>>> 20.7 0.0
>>> 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4
>>> 29.6 1.5
>>> 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9
>>> 0.0
>>> 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7
>>> 20.8 0.0
>>> 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5
>>> 25.9 0.9
>>>
>>> The problematic hop appears to be either 10 or 11 (level3-100g). Compare
>>> to:
>>>
>>> Host Loss% Snt Last Avg Best Wrst
>>> StDev
>>> 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6
>>> 0.5
>>> 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3 29.9
>>> 5.6
>>> 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3 26.1
>>> 4.9
>>> 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6 20.0
>>> 3.7
>>> 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6 44.1
>>> 9.8
>>> 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1 14.9
>>> 4.3
>>> 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1 68.3
>>> 11.3
>>> 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9 62.3
>>> 8.1
>>> 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5 20.6
>>> 0.0
>>> 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3
>>> 68.0 7.1
>>> 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2
>>> 22.8 0.3
>>>
>>>
>>> --
>>> Dan White
>>> BTC Broadband
>>> Network Admin Lead
>>> Ph 918.366.0248 (direct) main: (918)366-8000
>>> Fax 918.366.6610 email: dwhite at olp.net
>>> http://www.btcbroadband.com
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "external-netops" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to external-netops+unsubscribe at google.com.
>>> To post to this group, send email to external-netops at google.com.
>>> Visit this group at https://groups.google.com/a/go
>>> ogle.com/group/external-netops/.
>>> For more options, visit https://groups.google.com/a/google.com/d/optout.
>>>
>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
--
Thanks,
Andrew Duey
Network Engineer, Widerange Broadband
Direct: 402-327-1101
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20160829/158eb8c7/attachment.htm>
More information about the Outages
mailing list