<div dir="ltr"><div><br></div><br><div>Sadly, this is a repeat offender linecard/platform/customer-edge-device/something-or-other. </div><div><br></div><div>This is probably the 3rd time this has taken place for this port for us in the past 14months?</div><div><br></div><div>As a side note - We reported this at 7:45 AM GMT per our ticket timestamp when it originally took place. Not sure why it takes so long for the IP NOC to put it together.</div><div><br></div><div>If there are any Level3 lurkers - may be an indication of other training or tool problems?</div><div><br></div><div>In our ticket notes - they said there were 4 other ports impacted on the linecard with down BGP sessions, so I guess I know who else is on the linecard :) /wave</div><div><br></div><div><br></div><div><br></div><div>--</div><div>jason</div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 16, 2015 at 3:57 PM, Luke Rockwell via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><table cellspacing="1" cellpadding="2" width="99%" style="list-style-type:none;margin:0px;padding:0px;border-collapse:collapse;border-spacing:0px;border:1px solid rgb(0,0,0);color:rgb(51,51,51);font-family:Arial;font-size:12px;background:rgb(243,243,243)"><tbody style="list-style-type:none;margin:0px;padding:0px"><tr style="list-style-type:none;margin:0px;padding:0px"><td valign="top" style="list-style-type:none;padding:2px;background-color:rgb(255,255,255)">7/16/15 8:39:56 PM GMT</td><td valign="top" style="list-style-type:none;padding:2px;background-color:rgb(255,255,255)">The IP NOC is working with the equipment vendor to isolate the root cause of the errors.</td></tr><tr style="list-style-type:none;margin:0px;padding:0px"><td valign="top" style="list-style-type:none;padding:2px;background-color:rgb(238,238,238)">7/16/15 7:54:43 PM GMT</td><td valign="top" style="list-style-type:none;padding:2px;background-color:rgb(238,238,238)">No new information is currently available. Updates will be provided as they are received.</td></tr><tr style="list-style-type:none;margin:0px;padding:0px"><td valign="top" style="list-style-type:none;padding:2px;background-color:rgb(255,255,255)">7/16/15 6:56:49 PM GMT</td><td valign="top" style="list-style-type:none;padding:2px;background-color:rgb(255,255,255)">Investigations are ongoing to isolate the issue.</td></tr><tr style="list-style-type:none;margin:0px;padding:0px"><td valign="top" style="list-style-type:none;padding:2px;background-color:rgb(238,238,238)">7/16/15 6:06:45 PM GMT</td><td valign="top" style="list-style-type:none;padding:2px;background-color:rgb(238,238,238)">The IP NOC reports a card failing in Dallas, TX impacting IP services. The silent card failure was detected when the Technical Service Center began receiving calls from customers. The equipment vendor is being engaged for assistance. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted.</td></tr></tbody></table><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Thu, Jul 16, 2015 at 1:51 PM, Mark Price via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><br><div>FYI, Level3 has acknowledged the issue but no ETR yet.</div><div><br></div><div><br></div><div><span style="font-size:12.8000001907349px">"The event, 9470903, reports a card failing in Dallas, TX impacting IP services. The silent card failure was detected when the Technical Service Center began receiving calls from customers. "</span><br></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px"><br></span></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 16, 2015 at 3:03 AM, Mark Price <span dir="ltr"><<a href="mailto:mprice@tqhosting.com" target="_blank">mprice@tqhosting.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div>Starting at approx 06:00 UTC we had a strange issue with TCP/UDP traffic dying on Level3 in Dallas, although ICMP looked clean. For now we've stopped sending traffic to Level3 in Dallas.</div><div><br></div><div>Anyone else seeing similar issues?</div><span><font color="#888888"><div><br></div><div><br></div><div>-mark</div><div><br></div></font></span></div>
</blockquote></div><br></div>
</div></div><br></div></div><span class="">_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br></span></blockquote></div><br><br clear="all"><div><br></div>
</div></div>
<br>_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br></blockquote></div><br></div>