<div dir="ltr">Nothing interesting in a traceroute, but here's a ping I ran for 10 or so minutes:<div><br></div><div><div>--- 4.2.2.1 ping statistics ---</div><div>536 packets transmitted, 478 received, 10% packet loss, time 535985ms</div><div>rtt min/avg/max/mdev = 8.328/29.422/900.006/62.915 ms</div></div><div><br></div><div>Traceroute:</div><div><br></div><div><div>traceroute to 4.2.2.1 (4.2.2.1), 30 hops max, 60 byte packets<br></div><div> 1 router.thegrand.systems (192.168.1.1) 0.185 ms 0.118 ms 0.154 ms</div><div> 2 96.120.90.49 (96.120.90.49) 9.340 ms 15.779 ms 16.743 ms</div><div> 3 <a href="http://te-0-7-0-15-sur03.oakland.ca.sfba.comcast.net">te-0-7-0-15-sur03.oakland.ca.sfba.comcast.net</a> (68.85.190.69) 16.887 ms 16.888 ms 16.890 ms</div><div> 4 <a href="http://hu-0-8-0-1-ar01.santaclara.ca.sfba.comcast.net">hu-0-8-0-1-ar01.santaclara.ca.sfba.comcast.net</a> (68.85.57.61) 18.925 ms 19.468 ms <a href="http://hu-0-8-0-0-ar01.santaclara.ca.sfba.comcast.net">hu-0-8-0-0-ar01.santaclara.ca.sfba.comcast.net</a> (68.85.57.57) 19.517 ms</div><div> 5 <a href="http://lag-8.ear1.SanJose1.Level3.net">lag-8.ear1.SanJose1.Level3.net</a> (4.68.127.105) 17.494 ms * *</div><div> 6 <a href="http://ae-2-70.edge2.SanJose1.Level3.net">ae-2-70.edge2.SanJose1.Level3.net</a> (4.69.152.79) 17.216 ms <a href="http://ae-1-60.edge2.SanJose1.Level3.net">ae-1-60.edge2.SanJose1.Level3.net</a> (4.69.152.15) 16.865 ms 16.634 ms</div><div> 7 <a href="http://a.resolvers.level3.net">a.resolvers.level3.net</a> (4.2.2.1) 15.535 ms 13.181 ms 11.661 ms</div></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr">Robbie Trencheny (<a href="http://twitter.com/robbie" target="_blank">@robbie</a>)<div>925-884-3728<br><a href="http://robbie.io" target="_blank">robbie.io</a></div></div></div></div></div></div>
<br><div class="gmail_quote">On Wed, May 11, 2016 at 10:17 AM, Robbie Trencheny <span dir="ltr"><<a href="mailto:me@robbiet.us" target="_blank">me@robbiet.us</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I'm on DOCSIS 3.1 but I would assume Pandora is Metro E. I'll have a traceroute shortly. <div class="HOEnZb"><div class="h5"><span></span><br><br>On Wednesday, May 11, 2016, John Neiberger <<a href="mailto:jneiberger@gmail.com" target="_blank">jneiberger@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">Sounds like it's something local to your neighborhood. Is this Metro E or DOCSIS?</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">John</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 11, 2016 at 10:44 AM, Robbie Trencheny <span dir="ltr"><<a>me@robbiet.us</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This outage is still on going. Comcast is continually pushing back the ETOR. I was also informed that Pandora.com HQ is affected by the same issue (they are located across the street from me). Just seeing random packet loss. Will try to get a traceroute today. <div><div><span></span><br><br>On Monday, May 9, 2016, Michael Sinatra <<a>michael+outages@burnttofu.net</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I am still seeing significant loss between my work (ESnet in Berkeley) and my home (Comcast Business Class in Uptown Oakland):<br>
<br>
[noc5] ~% ping <a href="http://manasquan.burnttofu.net" target="_blank">manasquan.burnttofu.net</a><br>
PING <a href="http://manasquan.burnttofu.net" target="_blank">manasquan.burnttofu.net</a> (98.210.60.123): 56 data bytes<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=0 ttl=247 time=15.060 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=1 ttl=247 time=12.531 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=2 ttl=247 time=13.307 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=3 ttl=247 time=12.680 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=8 ttl=247 time=12.489 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=9 ttl=247 time=12.865 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=10 ttl=247 time=22.611 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=11 ttl=247 time=22.628 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=12 ttl=247 time=14.488 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=13 ttl=247 time=22.506 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=14 ttl=247 time=12.523 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=16 ttl=247 time=22.536 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=17 ttl=247 time=23.900 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=18 ttl=247 time=12.726 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=19 ttl=247 time=13.563 ms<br>
64 bytes from <a href="http://98.210.60.123" target="_blank">98.210.60.123</a>: icmp_seq=20 ttl=247 time=22.434 ms<br>
^C<br>
--- <a href="http://manasquan.burnttofu.net" target="_blank">manasquan.burnttofu.net</a> ping statistics ---<br>
21 packets transmitted, 16 packets received, 23.8% packet loss<br>
round-trip min/avg/max/stddev = 12.489/16.803/23.900/4.681 ms<br>
<br>
ESnet has a number of perfSONAR hosts (e.g. <a href="http://sunn-owamp.es.net" target="_blank">sunn-owamp.es.net</a>) from/to which you can do ping and traceroute if that helps.<br>
<br>
Similarly, I am seeing loss from UC San Francisco to my home network:<br>
<br>
[tulare:~] ms% ping <a href="http://manasquan.burnttofu.net" target="_blank">manasquan.burnttofu.net</a><br>
PING <a href="http://manasquan.burnttofu.net" target="_blank">manasquan.burnttofu.net</a> (98.210.60.123): 56 data bytes<br>
<br>
--- <a href="http://manasquan.burnttofu.net" target="_blank">manasquan.burnttofu.net</a> ping statistics ---<br>
26 packets transmitted, 23 packets received, 11.5% packet loss<br>
round-trip min/avg/max/stddev = 16.218/22.903/50.951/7.144 ms<br>
<br>
I am not seeing any loss between UCSF and a VPS I have in Los Angeles.<br>
<br>
Traceroutes from UCSF to home:<br>
<br>
traceroute to <a href="http://manasquan.burnttofu.net" target="_blank">manasquan.burnttofu.net</a> (98.210.60.123), 64 hops max, 52 byte packets<br>
1 10.131.240.1 (10.131.240.1) 3.862 ms 4.669 ms 6.025 ms<br>
2 10.255.255.0 (10.255.255.0) 3.680 ms 3.583 ms 4.700 ms<br>
3 <a href="http://3002-s251-r01.ucsf.edu" target="_blank">3002-s251-r01.ucsf.edu</a> (169.230.252.9) 3.489 ms 3.166 ms 5.008 ms<br>
4 <a href="http://oak-agg4--ucsf--10ge.cenic.net" target="_blank">oak-agg4--ucsf--10ge.cenic.net</a> (137.164.3.113) 5.633 ms 4.747 ms 9.053 ms<br>
5 <a href="http://dc-svl-agg4--oak-agg4-100ge.cenic.net" target="_blank">dc-svl-agg4--oak-agg4-100ge.cenic.net</a> (137.164.46.145) 5.550 ms 8.902 ms 8.859 ms<br>
6 <a href="http://xe-0-1-0.0.rtr.paix.net.internet2.edu" target="_blank">xe-0-1-0.0.rtr.paix.net.internet2.edu</a> (64.57.21.4) 6.259 ms<br>
<a href="http://xe-1-1-0.0.rtr.paix.net.internet2.edu" target="_blank">xe-1-1-0.0.rtr.paix.net.internet2.edu</a> (64.57.21.6) 5.475 ms<br>
<a href="http://xe-0-1-0.0.rtr.paix.net.internet2.edu" target="_blank">xe-0-1-0.0.rtr.paix.net.internet2.edu</a> (64.57.21.4) 5.857 ms<br>
7 <a href="http://te-0-3-0-0-pe02.529bryant.ca.ibone.comcast.net" target="_blank">te-0-3-0-0-pe02.529bryant.ca.ibone.comcast.net</a> (66.208.233.121) 6.002 ms 6.885 ms 6.002 ms<br>
8 * * *<br>
9 <a href="http://be-7922-ar01.santaclara.ca.sfba.comcast.net" target="_blank">be-7922-ar01.santaclara.ca.sfba.comcast.net</a> (68.86.90.94) 14.099 ms 6.931 ms 9.776 ms<br>
10 <a href="http://hu-0-5-0-1-sur03.oakland.ca.sfba.comcast.net" target="_blank">hu-0-5-0-1-sur03.oakland.ca.sfba.comcast.net</a> (68.85.57.62) 8.405 ms 8.775 ms<br>
<a href="http://hu-0-5-0-0-sur03.oakland.ca.sfba.comcast.net" target="_blank">hu-0-5-0-0-sur03.oakland.ca.sfba.comcast.net</a> (68.85.57.58) 10.548 ms<br>
11 <a href="http://te-6-1-acr05.oakland.ca.sfba.comcast.net" target="_blank">te-6-1-acr05.oakland.ca.sfba.comcast.net</a> (68.85.217.178) 8.970 ms 8.436 ms<br>
<a href="http://te-6-0-acr05.oakland.ca.sfba.comcast.net" target="_blank">te-6-0-acr05.oakland.ca.sfba.comcast.net</a> (68.85.190.70) 8.556 ms<br>
12 <a href="http://c-98-210-60-123.hsd1.ca.comcast.net" target="_blank">c-98-210-60-123.hsd1.ca.comcast.net</a> (98.210.60.123) 26.135 ms 43.082 ms 32.402 ms<br>
<br>
Note, hop 8, when it does respond, looks like this:<br>
<br>
8 68.86.86.29 (68.86.86.29) 7.791 ms<br>
68.86.86.145 (68.86.86.145) 7.724 ms<br>
68.86.86.69 (68.86.86.69) 8.259 ms<br>
<br>
michael<br>
<br>
On 5/9/16 2:02 PM, John Neiberger via Outages wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Gotcha. Well, if it happens again, let me know. I'd be glad to take a<br>
look. I haven't heard of any widespread issues in that area. I was<br>
hoping we could narrow it down a bit.<br>
<br>
Thanks,<br>
John<br>
<br>
On Mon, May 9, 2016 at 2:52 PM, Robbie Trencheny <<a>me@robbiet.us</a><br>
<mailto:<a>me@robbiet.us</a>>> wrote:<br>
<br>
John,<br>
<br>
Just looking at my PFSense reports. No traceroutes at the moment as<br>
the problem seems to have cleared up for now. I was seeing packet<br>
loss as high as 15% about 35 minutes ago according to PFSense.<br>
<br>
<br>
On Monday, May 9, 2016, John Neiberger <<a>jneiberger@gmail.com</a><br>
<mailto:<a>jneiberger@gmail.com</a>>> wrote:<br>
<br>
How are you verifying the packet loss? Do you have traceroutes?<br>
<br>
Thanks,<br>
John<br>
<br>
On Mon, May 9, 2016 at 2:42 PM, Robbie Trencheny via Outages<br>
<<a>outages@outages.org</a>> wrote:<br>
<br>
There has been a Comcast outage/degradation of services in<br>
the Bay Area starting at 1403 PDT 05/08/2016. The current<br>
estimated restoration time is 1707 PDT. I couldn't get any<br>
details from Comcast about the issues. I've been seeing high<br>
packet loss. They did tell me that I will receive a bill credit.<br>
<br>
Robbie Trencheny (@robbie <<a href="http://twitter.com/robbie" target="_blank">http://twitter.com/robbie</a>>)<br>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a>Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
<br>
<br>
<br>
--<br>
Robbie Trencheny (@robbie <<a href="http://twitter.com/robbie" target="_blank">http://twitter.com/robbie</a>>)<br>
<a href="tel:925-884-3728" value="+19258843728" target="_blank">925-884-3728</a> <tel:<a href="tel:925-884-3728" value="+19258843728" target="_blank">925-884-3728</a>><br>
<a href="http://robbie.io" target="_blank">robbie.io</a> <<a href="http://robbie.io" target="_blank">http://robbie.io</a>><br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a>Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
</blockquote>
<br>
</blockquote><br><br></div></div><div><div>-- <br><div dir="ltr"><div><div dir="ltr">Robbie Trencheny (<a href="http://twitter.com/robbie" target="_blank">@robbie</a>)<div><a href="tel:925-884-3728" value="+19258843728" target="_blank">925-884-3728</a><br><a href="http://robbie.io" target="_blank">robbie.io</a></div></div></div></div><br>
</div></div></blockquote></div><br></div>
</blockquote><br><br>-- <br><div dir="ltr"><div><div dir="ltr">Robbie Trencheny (<a href="http://twitter.com/robbie" target="_blank">@robbie</a>)<div><a href="tel:925-884-3728" value="+19258843728" target="_blank">925-884-3728</a><br><a href="http://robbie.io" target="_blank">robbie.io</a></div></div></div></div><br>
</div></div></blockquote></div><br></div>