<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">Thanks! Every trace I've seen has indicated a problem in Level 3. That would certainly explain why this is so widespread.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 6, 2017 at 11:52 AM, Dovid Bender <span dir="ltr"><<a href="mailto:dovid@telecurve.com" target="_blank">dovid@telecurve.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">From NTT in NYC:<div><div> 3. 165.254.26.201 0.0% 1777 0.8 0.8 0.7 10.0 0.3</div><div> 4. <a href="http://ae-3.r25.nycmny01.us.bb.gin.ntt.net" target="_blank">ae-3.r25.nycmny01.us.bb.gin.<wbr>ntt.net</a> 0.0% 1777 0.7 0.7 0.6 149.4 3.9</div><div> 5. <a href="http://ae-9.r22.asbnva02.us.bb.gin.ntt.net" target="_blank">ae-9.r22.asbnva02.us.bb.gin.<wbr>ntt.net</a> 18.5% 1777 6.3 6.4 6.1 24.9 1.8</div><div> 6. <a href="http://ae-1.r05.asbnva02.us.bb.gin.ntt.net" target="_blank">ae-1.r05.asbnva02.us.bb.gin.<wbr>ntt.net</a> 0.0% 1777 5.9 5.9 5.8 15.6 0.5</div><div> 7. <a href="http://ae-0.a02.asbnva02.us.bb.gin.ntt.net" target="_blank">ae-0.a02.asbnva02.us.bb.gin.<wbr>ntt.net</a> 19.8% 1777 6.5 7.7 6.0 57.0 5.1</div><div> 8. <a href="http://lag-12.ear2.Washington12.Level3.net" target="_blank">lag-12.ear2.Washington12.<wbr>Level3.net</a> 98.6% 1777 7208. 7377. 6927. 8791. 368.8</div><div> 9. <a href="http://ae-2-3.bear1.Boston1.Level3.net" target="_blank">ae-2-3.bear1.Boston1.Level3.<wbr>net</a> 98.1% 1777 131.9 1047. 120.0 6493. 1842.</div><div>10. 4.68.71.218 45.2% 1777 129.0 133.7 119.6 143.6 4.4</div><div>11. <a href="http://be-2-ar01.chartford.ct.hartford.comcast.net" target="_blank">be-2-ar01.chartford.ct.<wbr>hartford.comcast.net</a> 45.9% 1777 139.2 136.3 122.4 146.4 4.5</div><div>12. <a href="http://be-31-sur01.waterbury.ct.hartford.comcast.net" target="_blank">be-31-sur01.waterbury.ct.<wbr>hartford.comcast.net</a> 43.7% 1777 143.0 137.4 121.7 147.8 4.3</div><div>13. 68.87.158.6 44.7% 1777 139.5 137.1 115.0 148.1 4.5</div><div>14. <a href="http://c-73-218-149-14.hsd1.ma.comcast.net" target="_blank">c-73-218-149-14.hsd1.ma.<wbr>comcast.net</a> 45.3% 1776 139.6 146.0 130.5 177.1 5.8</div><div>15. ???</div><div><br></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 6, 2017 at 1:51 PM, Dovid Bender <span dir="ltr"><<a href="mailto:dovid@telecurve.com" target="_blank">dovid@telecurve.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">From DO (NYC):<div><div>Keys: Help Display mode Restart statistics Order of fields quit</div><div> Packets Pings</div><div> Host Loss% Snt Last Avg Best Wrst StDev</div><div> 1. 104.131.255.253 0.0% 12 1.1 1.3 0.3 4.1 1.5</div><div> 2. 138.197.248.160 0.0% 11 13.0 1.6 0.3 13.0 3.8</div><div> 3. <a href="http://ix-ae-3-0.tcore1.N75-New-York.as6453.net" target="_blank">ix-ae-3-0.tcore1.N75-New-York.<wbr>as6453.net</a> 0.0% 11 0.4 0.4 0.3 0.8 0.1</div><div> 4. <a href="http://if-ae-12-2.tcore2.NTO-New-York.as6453.net" target="_blank">if-ae-12-2.tcore2.NTO-New-York<wbr>.as6453.net</a> 0.0% 11 8.5 9.8 6.2 40.8 10.3</div><div> 5. <a href="http://if-ae-30-2.tcore1.AEQ-Ashburn.as6453.net" target="_blank">if-ae-30-2.tcore1.AEQ-Ashburn.<wbr>as6453.net</a> 0.0% 11 6.2 8.8 5.9 24.4 5.7</div><div> 6. ???</div><div> 7. ???</div><div> 8. 4.68.71.218 63.6% 11 27.7 28.4 27.5 30.7 1.6</div><div> 9. <a href="http://be-2-ar01.chartford.ct.hartford.comcast.net" target="_blank">be-2-ar01.chartford.ct.hartfor<wbr>d.comcast.net</a> 40.0% 11 31.7 32.2 31.7 32.9 0.5</div><div>10. <a href="http://be-31-sur01.waterbury.ct.hartford.comcast.net" target="_blank">be-31-sur01.waterbury.ct.hartf<wbr>ord.comcast.net</a> 40.0% 11 33.6 32.7 32.2 33.6 0.6</div><div>11. 68.87.158.14 50.0% 11 33.5 32.5 32.1 33.5 0.6</div><div>12. <a href="http://c-73-218-149-14.hsd1.ct.comcast.net" target="_blank">c-73-218-149-14.hsd1.ct.comcas<wbr>t.net</a> 70.0% 11 39.1 38.7 38.4 39.1 0.3</div><div>13. ???</div><div><br></div><div>From rackspace (Texas)</div><div><div>dev (0.0.0.0) Mon Nov 6 18:50:13 2017</div><div>Keys: Help Display mode Restart statistics Order of fields quit</div><div> Packets Pings</div><div> Host Loss% Snt Last Avg Best Wrst StDev</div><div> 1. 50.56.237.2 0.0% 11 1.0 1.1 0.9 1.8 0.3</div><div> 2. <a href="http://aggr170a-8-core10.dfw3.rackspace.net" target="_blank">aggr170a-8-core10.dfw3.rackspa<wbr>ce.net</a> 0.0% 11 0.9 1.1 0.9 1.5 0.2</div><div> 3. <a href="http://core10-coreb.coreb.dfw3.rackspace.net" target="_blank">core10-coreb.coreb.dfw3.racksp<wbr>ace.net</a> 0.0% 11 3.4 3.5 3.3 3.8 0.1</div><div> 4. <a href="http://be32.dcpe2.dfw1.rackspace.net" target="_blank">be32.dcpe2.dfw1.rackspace.net</a> 0.0% 11 3.6 3.8 3.4 4.4 0.3</div><div> 5. <a href="http://be2-mspe4.dfw30.rackspace.net" target="_blank">be2-mspe4.dfw30.rackspace.net</a> 0.0% 11 5.5 5.7 5.5 6.6 0.3</div><div> 6. <a href="http://ae-22.a01.dllstx04.us.bb.gin.ntt.net" target="_blank">ae-22.a01.dllstx04.us.bb.gin.n<wbr>tt.net</a> 0.0% 11 8.5 6.3 4.8 9.1 1.7</div><div> 7. <a href="http://ae-9.r11.dllstx09.us.bb.gin.ntt.net" target="_blank">ae-9.r11.dllstx09.us.bb.gin.nt<wbr>t.net</a> 0.0% 11 52.3 43.8 42.8 52.3 2.8</div><div> 8. <a href="http://ae-0.r22.dllstx09.us.bb.gin.ntt.net" target="_blank">ae-0.r22.dllstx09.us.bb.gin.nt<wbr>t.net</a> 0.0% 11 14.0 5.8 4.4 14.0 2.9</div><div> 9. <a href="http://ae-1.r22.asbnva02.us.bb.gin.ntt.net" target="_blank">ae-1.r22.asbnva02.us.bb.gin.nt<wbr>t.net</a> 45.5% 11 43.0 43.8 42.2 49.9 3.0</div><div>10. <a href="http://ae-1.r05.asbnva02.us.bb.gin.ntt.net" target="_blank">ae-1.r05.asbnva02.us.bb.gin.nt<wbr>t.net</a> 0.0% 11 42.9 43.2 42.9 44.9 0.6</div><div>11. <a href="http://ae-0.a02.asbnva02.us.bb.gin.ntt.net" target="_blank">ae-0.a02.asbnva02.us.bb.gin.nt<wbr>t.net</a> 50.0% 11 42.6 42.6 42.3 43.0 0.3</div><div>12. ???</div><div>13. <a href="http://ae-2-3.bear1.Boston1.Level3.net" target="_blank">ae-2-3.bear1.Boston1.Level3.ne<wbr>t</a> 90.0% 11 170.9 170.9 170.9 170.9 0.0</div><div>14. 4.68.71.218 20.0% 11 169.2 175.8 169.2 198.9 9.6</div><div>15. <a href="http://be-2-ar01.chartford.ct.hartford.comcast.net" target="_blank">be-2-ar01.chartford.ct.hartfor<wbr>d.comcast.net</a> 60.0% 11 175.0 179.3 167.8 190.8 10.0</div><div>16. <a href="http://be-31-sur01.waterbury.ct.hartford.comcast.net" target="_blank">be-31-sur01.waterbury.ct.hartf<wbr>ord.comcast.net</a> 60.0% 11 180.1 179.1 175.4 183.0 3.2</div><div>17. 68.87.158.10 44.4% 10 174.8 181.7 174.8 189.7 5.6</div><div>18. <a href="http://c-73-218-149-14.hsd1.ma.comcast.net" target="_blank">c-73-218-149-14.hsd1.ma.comcas<wbr>t.net</a> 60.0% 10 173.0 181.0 173.0 191.2 8.7</div><div>19. ???</div><div><br></div><div><br></div><div>From COX in SoCal</div><div><br></div><div><div> 3. <a href="http://wsip-216-231-24-181.oc.oc.cox.net" target="_blank">wsip-216-231-24-181.oc.oc.cox.<wbr>net</a> 0.0% 8 1.8 1.5 1.4 1.8 0.1</div><div> 4. <a href="http://ip68-4-11-200.oc.oc.cox.net" target="_blank">ip68-4-11-200.oc.oc.cox.net</a> 0.0% 8 1.2 1.2 1.2 1.4 0.1</div><div> 5. 68.1.1.167 0.0% 8 2.7 2.8 2.7 2.9 0.1</div><div> 6. <a href="http://ae-7.a00.lsanca20.us.bb.gin.ntt.net" target="_blank">ae-7.a00.lsanca20.us.bb.gin.nt<wbr>t.net</a> 0.0% 8 3.3 3.3 2.7 4.2 0.6</div><div> 7. <a href="http://ae-3.r01.lsanca20.us.bb.gin.ntt.net" target="_blank">ae-3.r01.lsanca20.us.bb.gin.nt<wbr>t.net</a> 0.0% 8 69.4 69.5 69.4 69.8 0.1</div><div> 8. <a href="http://ae-8.r23.lsanca07.us.bb.gin.ntt.net" target="_blank">ae-8.r23.lsanca07.us.bb.gin.nt<wbr>t.net</a> 0.0% 8 2.9 4.6 2.9 12.0 3.1</div><div> 9. <a href="http://ae-6.r22.asbnva02.us.bb.gin.ntt.net" target="_blank">ae-6.r22.asbnva02.us.bb.gin.nt<wbr>t.net</a> 12.5% 8 66.3 66.5 66.2 67.1 0.3</div><div>10. <a href="http://ae-7.r06.asbnva02.us.bb.gin.ntt.net" target="_blank">ae-7.r06.asbnva02.us.bb.gin.nt<wbr>t.net</a> 0.0% 8 66.4 66.4 66.2 66.6 0.1</div><div>11. <a href="http://ae-1.a02.asbnva02.us.bb.gin.ntt.net" target="_blank">ae-1.a02.asbnva02.us.bb.gin.nt<wbr>t.net</a> 0.0% 8 66.2 67.2 66.1 72.8 2.3</div><div>12. ???</div><div>13. ???</div><div>14. 4.68.71.218 75.0% 8 202.6 204.3 202.6 206.0 2.4</div><div>15. <a href="http://be-2-ar01.chartford.ct.hartford.comcast.net" target="_blank">be-2-ar01.chartford.ct.hartfor<wbr>d.comcast.net</a> 12.5% 8 202.3 203.3 197.6 208.0 4.4</div><div>16. <a href="http://be-31-sur01.waterbury.ct.hartford.comcast.net" target="_blank">be-31-sur01.waterbury.ct.hartf<wbr>ord.comcast.net</a> 37.5% 8 192.5 199.6 192.5 205.0 4.9</div><div>17. 68.87.158.14 42.9% 8 200.4 202.3 197.8 208.6 4.6</div><div>18. <a href="http://c-73-218-149-14.hsd1.ct.comcast.net" target="_blank">c-73-218-149-14.hsd1.ct.comcas<wbr>t.net</a> 42.9% 8 207.9 208.2 203.0 214.8 4.9</div><div>19. ???</div><div><br></div></div><div><br></div><div><br></div></div><div><br></div><div><br></div><div><br></div></div><div><br></div><div><br></div></div><div class="m_3564741768585495496HOEnZb"><div class="m_3564741768585495496h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 6, 2017 at 1:38 PM, John Neiberger via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">Our on-net tools don't indicate a problem within the network, but I've seen a couple of traceroutes that indicate a problem on Level 3, which certainly would affect us. We're trying to nail this down. Any traceroutes you have would be helpful.</div></div><div class="m_3564741768585495496m_-2562743707483664291HOEnZb"><div class="m_3564741768585495496m_-2562743707483664291h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 6, 2017 at 11:35 AM, Brian R. Watters via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:verdana,helvetica,sans-serif;font-size:12pt;color:#000000">Same issue is wide spread in California via both COAX and Fiber connections (Comcast).<br><br><hr id="m_3564741768585495496m_-2562743707483664291m_3673433004924625567m_9007256081046576135zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From: </b>"Gary Leato via Outages" <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>><br><b>To: </b>"Jim Limmer" <<a href="mailto:jlimmer@vna-commh.org" target="_blank">jlimmer@vna-commh.org</a>><br><b>Cc: </b><a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br><b>Sent: </b>Monday, November 6, 2017 10:27:35 AM<br><b>Subject: </b>Re: [outages] cocmast packetloss and high latency<div><div class="m_3564741768585495496m_-2562743707483664291m_3673433004924625567h5"><br><br><div dir="ltr"><span style="font-size:12.8px">we are seeing similiar comcast problems in central illinois rig</span><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 6, 2017 at 12:25 PM, Jim Limmer via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US">
<div class="m_3564741768585495496m_-2562743707483664291m_3673433004924625567m_9007256081046576135m_-4963607378405613808WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Yes. And whats worse is my deli nearby uses Comcast business for their credit card reader.. guess I owe them for lunch tomorrow..
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Gill Sans MT","sans-serif";color:#1f497d">Jim Limmer, CISSP
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Gill Sans MT","sans-serif";color:#1f497d">< Information Security Officer
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Gill Sans MT","sans-serif";color:#1f497d"> Systems and Network Architect ><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Gill Sans MT","sans-serif";color:#1f497d">VNA Community Healthcare<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Candara","sans-serif";color:#1f497d"><a href="http://www.connecticuthomecare.org/" target="_blank"><span style="color:blue">www.connecticuthomecare.org</span></a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Candara","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:navy"><img border="0" width="201" height="126" id="m_3564741768585495496m_-2562743707483664291m_3673433004924625567m_9007256081046576135m_-4963607378405613808Picture_x0020_1" src="cid:image001.jpg@01D35702.B9288D30" alt="cid:image002.jpg@01D332E1.E5C48B90"></span><span style="font-size:10.0pt;font-family:"Candara","sans-serif";color:#1f497d">
<img border="0" width="136" height="40" id="m_3564741768585495496m_-2562743707483664291m_3673433004924625567m_9007256081046576135m_-4963607378405613808Picture_x0020_2" src="cid:image002.png@01D35702.B9288D30" alt="smCISSP-logo-2lines"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Candara","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Candara","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Outages [mailto:<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outage<wbr>s.org</a>]
<b>On Behalf Of </b>Audrius Peciukas via Outages<br>
<b>Sent:</b> Monday, November 06, 2017 1:24 PM<br>
<b>To:</b> <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br>
<b>Subject:</b> [outages] cocmast packetloss and high latency<u></u><u></u></span></p><div><div class="m_3564741768585495496m_-2562743707483664291m_3673433004924625567m_9007256081046576135h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">anyone else is seeing lots of packet loss and 100ms+ latency mostly in east coast on comcast network?<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Tahoma","sans-serif"">Audrius Peciukas</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Tahoma","sans-serif"">1Wire Communications</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="http://www.1wirecom.com/" target="_blank"><span style="font-family:"Tahoma","sans-serif";color:#1155cc">http://www.1wirecom.com</span></a><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
<br>______________________________<wbr>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailma<wbr>n/listinfo/outages</a><br>
<br></blockquote></div><br></div>
<br>______________________________<wbr>_________________<br>Outages mailing list<br><a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailma<wbr>n/listinfo/outages</a><br></div></div></div><br><br><br>-- <br><div><span name="x"></span><p class="MsoNormal"></p><p class="MsoNormal"><b><i>Brian Watters</i></b></p><p class="MsoNormal"><b><i></i></b></p><p class="MsoNormal"><b><i></i></b></p><p class="MsoNormal"><b><i></i></b></p><p class="MsoNormal"><span style="color:rgb(31,73,125)"></span></p><p class="MsoNormal"><img style="border:0px" src="cid:e304fcafd192203d7e75765a1e5ccf679c011788@zimbra"><br></p><p class="MsoNormal"></p><p class="MsoNormal"></p><p class="MsoNormal"></p><p class="MsoNormal"><font class="m_3564741768585495496m_-2562743707483664291m_3673433004924625567m_9007256081046576135Apple-style-span" color="#632423"><a href="tel:(559)%20579-1803" value="+15595791803" target="_blank">559-579-1803</a></font></p><p class="MsoNormal"><font class="m_3564741768585495496m_-2562743707483664291m_3673433004924625567m_9007256081046576135Apple-style-span" color="#632423"><a href="mailto:brianwatters@totlcom.com" target="_blank">brianwatters@totlcom.com</a></font></p><p class="MsoNormal"><font class="m_3564741768585495496m_-2562743707483664291m_3673433004924625567m_9007256081046576135Apple-style-span" color="#632423"><a href="http://www.totlcom.com" target="_blank">http://www.totlcom.com</a></font></p><p class="MsoNormal"><i><span style="color:rgb(99,36,35)">Very satisfied customers are our standard, not our goal.</span></i></p><span name="x"></span><br></div></div></div><br>______________________________<wbr>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailma<wbr>n/listinfo/outages</a><br>
<br></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailma<wbr>n/listinfo/outages</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>