[Outages-discussion] [outages] L3/CL route leak for 90min, Mon/6

[Jay Farrell]

Got this RFO second hand. Apparently someone fumbled the config for a
single L3 customer BGP session.

Root Cause: A configuration issue impacted IP services in various markets
across the United States.
Fix Action: The IP NOC reverted a policy change to restore services to a
stable state.
Summary: The IP NOC was informed of a significant client impact which
seemed to originate on the east coast. The IP NOC began investigating, and
soon discovered that the service impact was occurring in various markets
across the United States. The issue was isolated to a policy change that
was implemented to a single router in error while trying to configure an
individual customer BGP. This policy change affected a major public peering
session. The IP NOC reverted the policy change to restore services to a
stable state.
Corrective Actions: An extensive post analysis review will be
conducted to evaluate
preventative measures and corrective actions that can be implemented to
prevent network impact of this magnitude. The individual responsible for
this policy change has been identified.
This service impact has concluded; if additional issues are experienced,
please contact the CenturyLink Technical Service Center. There may be
additional analysis and discovery that occurs as the incident is reviewed
by NOC management. Any available updates will be relayed upon event ticket
closure. At that time, a customer satisfaction survey link may be
available. We strive to provide thorough communications containing the
available information during a service disruption. Please let us know if
the updates you received during this event were satisfactory.

On Mon, Nov 6, 2017 at 6:24 PM, Jay Ashworth via Outages <
outages at outages.org> wrote:

> http://wrd.cm/2Ao2NdV
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20171106/db401ee4/attachment.html>