<div dir="ltr">Confirmed - working fine from the South Bay, to the same hosts that were seeing problems earlier.<div><br></div><div> Scott</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Oct 2, 2013 at 6:38 PM, Jeremy Chadwick <span dir="ltr"><<a href="mailto:jdc@koitsu.org" target="_blank">jdc@koitsu.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Follow-up:<br>
<br>
Since roughly 18:14 PDT things look back to normal. Others who saw<br>
issues may want to re-test and follow up as necessary.<br>
<br>
Thanks everyone!<br>
<div class="im HOEnZb"><br>
--<br>
| Jeremy Chadwick <a href="mailto:jdc@koitsu.org">jdc@koitsu.org</a> |<br>
| UNIX Systems Administrator <a href="http://jdc.koitsu.org/" target="_blank">http://jdc.koitsu.org/</a> |<br>
| Making life hard for others since 1977. PGP 4BD6C0CB |<br>
<br>
</div><div class="HOEnZb"><div class="h5">On Wed, Oct 02, 2013 at 04:54:34PM -0700, Jeremy Chadwick wrote:<br>
> This just happened -- looks like something in Comcast's network within<br>
> Oakland broke, reconverged (quickly and seamless), but issues remain.<br>
><br>
> Ren Provo et al, if you're lurking, might want to make someone aware of<br>
> this (if they aren't already). :-)<br>
><br>
> src: 76.102.14.35<br>
> dst: <a href="tel:208.79.90.130" value="+12087990130">208.79.90.130</a><br>
><br>
> === Wed Oct 2 16:14:00 PDT 2013 (1380755640)<br>
> Start: Wed Oct 2 16:14:00 2013<br>
> HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst<br>
> 1.|-- 192.168.1.1 0.0% 40 40 0.4 0.3 0.2 0.4<br>
> 2.|-- 76.102.12.1 0.0% 40 40 9.6 10.8 8.1 73.5<br>
> 3.|-- 68.86.249.253 0.0% 40 40 8.9 10.2 8.3 62.2<br>
> 4.|-- 69.139.199.98 0.0% 40 40 10.6 13.1 9.8 47.4<br>
> 5.|-- 68.85.155.14 0.0% 40 40 12.0 16.8 11.4 87.7<br>
> 6.|-- 68.86.90.157 0.0% 40 40 14.6 17.8 12.6 87.5<br>
> 7.|-- 68.86.89.210 0.0% 40 40 23.7 24.7 20.6 92.9<br>
> 8.|-- 68.86.87.210 0.0% 40 40 25.9 25.0 20.7 99.0<br>
> 9.|-- 173.167.57.138 0.0% 40 40 22.1 24.7 21.3 89.0<br>
> 10.|-- 69.31.127.141 0.0% 40 40 20.5 22.0 19.8 67.1<br>
> 11.|-- 69.31.124.114 0.0% 40 40 20.6 21.6 20.0 45.9<br>
> 12.|-- 67.199.135.102 0.0% 40 40 39.9 36.5 21.1 231.0<br>
> 13.|-- 208.79.90.130 0.0% 40 40 21.2 20.7 20.2 21.7<br>
> === END<br>
><br>
> === Wed Oct 2 16:15:00 PDT 2013 (1380755700)<br>
> Start: Wed Oct 2 16:15:00 2013<br>
> HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst<br>
> 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4<br>
> 2.|-- 76.102.12.1 0.0% 40 40 8.1 9.4 8.1 13.6<br>
> 3.|-- 68.86.249.253 0.0% 40 40 8.7 9.7 8.4 14.1<br>
> 4.|-- 68.86.143.98 0.0% 40 40 13.1 13.1 10.2 21.1<br>
> 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0<br>
> 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0<br>
> 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0<br>
> 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0<br>
> 9.|-- 173.167.57.138 0.0% 40 40 33.0 33.9 31.6 37.1<br>
> 10.|-- 69.31.127.141 5.0% 40 38 30.2 31.4 29.7 39.3<br>
> 11.|-- 69.31.124.114 7.5% 40 37 31.8 38.5 29.8 104.0<br>
> 12.|-- 67.199.135.102 7.5% 40 37 31.1 36.6 30.6 119.7<br>
> 13.|-- 208.79.90.130 7.5% 40 37 30.9 31.9 30.4 38.9<br>
> === END<br>
><br>
> 68.86.249.253 = <a href="http://te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net" target="_blank">te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net</a><br>
> 69.139.199.98 = <a href="http://te-1-1-0-11-ar01.oakland.ca.sfba.comcast.net" target="_blank">te-1-1-0-11-ar01.oakland.ca.sfba.comcast.net</a><br>
> 68.85.155.14 = <a href="http://be-90-ar01.sfsutro.ca.sfba.comcast.net" target="_blank">be-90-ar01.sfsutro.ca.sfba.comcast.net</a><br>
> 68.86.143.98 = <a href="http://te-1-1-0-5-ar01.oakland.ca.sfba.comcast.net" target="_blank">te-1-1-0-5-ar01.oakland.ca.sfba.comcast.net</a><br>
><br>
> And packet loss has gradually been getting worse (impacting all forms of<br>
> packets, not just TCP).<br>
><br>
> === Wed Oct 2 16:45:00 PDT 2013 (1380757500)<br>
> Start: Wed Oct 2 16:45:00 2013<br>
> HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst<br>
> 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4<br>
> 2.|-- 76.102.12.1 0.0% 40 40 8.6 9.9 8.1 14.3<br>
> 3.|-- 68.86.249.253 0.0% 40 40 9.2 10.2 8.4 16.7<br>
> 4.|-- 68.86.143.98 0.0% 40 40 13.9 13.1 10.6 16.6<br>
> 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0<br>
> 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0<br>
> 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0<br>
> 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0<br>
> 9.|-- 173.167.57.138 15.0% 40 34 34.6 34.8 31.7 41.7<br>
> 10.|-- 69.31.127.141 25.0% 40 30 31.3 32.6 30.0 41.7<br>
> 11.|-- 69.31.124.114 5.0% 40 38 31.8 33.5 30.5 58.6<br>
> 12.|-- 67.199.135.102 12.8% 39 34 30.9 37.1 30.9 77.3<br>
> 13.|-- 208.79.90.130 7.7% 39 36 32.9 32.3 30.8 37.3<br>
> === END<br>
><br>
> And here's what I see on the return path, for a brief run:<br>
><br>
> src: 208.79.90.130<br>
> dst: 76.102.14.35<br>
><br>
> Host Loss% Snt Rcv Last Avg Best Wrst<br>
> 1. 208.79.90.129 0.0% 93 93 1.5 6.1 0.9 163.0<br>
> 2. <a href="http://s7.lax.arpnetworks.com" target="_blank">s7.lax.arpnetworks.com</a> 0.0% 93 93 0.7 11.2 0.5 150.5<br>
> 3. <a href="http://ge0-15.as01.lax07.mzima.net" target="_blank">ge0-15.as01.lax07.mzima.net</a> 0.0% <a href="tel:93%20%20%20%2093%20%20%204.1%20%20%202.4%20%20%200.4" value="+19393412404">93 93 4.1 2.4 0.4</a> 64.6<br>
> 4. <a href="http://ae3-50g.cr1.lax1.us.nlayer.net" target="_blank">ae3-50g.cr1.lax1.us.nlayer.net</a> 0.0% <a href="tel:93%20%20%20%2093%20%20%200.6%20%20%202.2%20%20%200.4" value="+19393062204">93 93 0.6 2.2 0.4</a> 56.0<br>
> 5. <a href="http://ae1-50g.ar1.lax1.us.nlayer.net" target="_blank">ae1-50g.ar1.lax1.us.nlayer.net</a> 1.1% <a href="tel:93%20%20%20%2092%20%20%203.1%20%20%203.3%20%20%201.3" value="+19392313313">93 92 3.1 3.3 1.3</a> 6.4<br>
> 6. <a href="http://be-102-pe01.600wseventh.ca.ibone.comcast.ne" target="_blank">be-102-pe01.600wseventh.ca.ibone.comcast.ne</a> 0.0% <a href="tel:93%20%20%20%2093%20%20%203.8%20%20%203.1%20%20%201.1" value="+19393383111">93 93 3.8 3.1 1.1</a> 5.4<br>
> 7. pos-0-8-0-0-cr01.losangeles.ca.ibone.comcas 0.0% 93 93 3.0 2.9 1.5 6.1<br>
> 8. he-2-15-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 92 92 8.6 10.3 8.1 12.5<br>
> 9. <a href="http://he-0-9-0-0-ar01.sfsutro.ca.sfba.comcast.net" target="_blank">he-0-9-0-0-ar01.sfsutro.ca.sfba.comcast.net</a> 0.0% 92 92 12.5 11.6 9.8 13.6<br>
> 10. <a href="http://be-100-ar01.oakland.ca.sfba.comcast.net" target="_blank">be-100-ar01.oakland.ca.sfba.comcast.net</a> 3.3% 92 88 22.0 25.2 21.9 49.6<br>
> 11. te-0-4-0-2-ur06.santaclara.ca.sfba.comcast. 3.3% 92 89 23.8 23.8 23.0 25.4<br>
> 12. <a href="http://te-6-0-acr03.santaclara.ca.sfba.comcast.net" target="_blank">te-6-0-acr03.santaclara.ca.sfba.comcast.net</a> 9.8% 92 83 23.8 24.4 22.6 46.3<br>
> 13. <a href="http://c-76-102-14-35.hsd1.ca.comcast.net" target="_blank">c-76-102-14-35.hsd1.ca.comcast.net</a> 6.5% 92 86 36.3 31.7 30.5 38.3<br>
><br>
> --<br>
> | Jeremy Chadwick <a href="mailto:jdc@koitsu.org">jdc@koitsu.org</a> |<br>
> | UNIX Systems Administrator <a href="http://jdc.koitsu.org/" target="_blank">http://jdc.koitsu.org/</a> |<br>
> | Making life hard for others since 1977. PGP 4BD6C0CB |<br>
><br>
> _______________________________________________<br>
> Outages mailing list<br>
> <a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
> <a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</div></div></blockquote></div><br></div>