[outages] Level3 issue in Dallas

[Jason Ackley]

Sadly, this is a repeat offender
linecard/platform/customer-edge-device/something-or-other.

This is probably the 3rd time this has taken place for this port for us in
the past 14months?

As a side note - We reported this at 7:45 AM GMT per our ticket timestamp
when it originally took place. Not sure why it takes so long for the IP NOC
to put it together.

If there are any Level3 lurkers - may be an indication of other training or
tool problems?

In our ticket notes - they said there were 4 other ports impacted on the
linecard with down BGP sessions, so I guess I know who else is on the
linecard :)    /wave



--
jason




On Thu, Jul 16, 2015 at 3:57 PM, Luke Rockwell via Outages <
outages at outages.org> wrote:

> 7/16/15 8:39:56 PM GMTThe IP NOC is working with the equipment vendor to
> isolate the root cause of the errors.7/16/15 7:54:43 PM GMTNo new
> information is currently available. Updates will be provided as they are
> received.7/16/15 6:56:49 PM GMTInvestigations are ongoing to isolate the
> issue.7/16/15 6:06:45 PM GMTThe IP NOC reports a card failing in Dallas,
> TX impacting IP services. The silent card failure was detected when the
> Technical Service Center began receiving calls from customers. The
> equipment vendor is being engaged for assistance. Please be advised that
> updates for this event will be relayed at a minimum of hourly unless
> otherwise noted.
>
> On Thu, Jul 16, 2015 at 1:51 PM, Mark Price via Outages <
> outages at outages.org> wrote:
>
>>
>> FYI, Level3 has acknowledged the issue but no ETR yet.
>>
>>
>> "The event, 9470903, reports a card failing in Dallas, TX impacting IP
>> services. The silent card failure was detected when the Technical Service
>> Center began receiving calls from customers. "
>>
>>
>>
>> On Thu, Jul 16, 2015 at 3:03 AM, Mark Price <mprice at tqhosting.com> wrote:
>>
>>>
>>> Starting at approx 06:00 UTC we had a strange issue with TCP/UDP traffic
>>> dying on Level3 in Dallas, although ICMP looked clean.  For now we've
>>> stopped sending traffic to Level3 in Dallas.
>>>
>>> Anyone else seeing similar issues?
>>>
>>>
>>> -mark
>>>
>>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20150716/99084223/attachment.htm>