[outages] 216.115.160.0/20 being blackholed

[Ben Bartsch]

Thanks Team Cymru.  Thanks LONI - ASN 32440 for your work to discover the
problem.

-bb


On Tue, Jul 16, 2013 at 5:16 PM, Don Napier <dnapier at cymru.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 7/16/13 1:29 PM, Don Napier wrote:
> >
> >
> > On 7/16/13 7:29 AM, Ben Bartsch wrote:
> >> FYI - got a call from a customer last night about a Xerox site
> >> unreachable through us.  Found the prefix coming from Cymru.
> >> Instructed our NOC to work with them to resolve.  Cymru first
> >> claimed they were not sending it because it was not a bogon.
> >> Once we proved it was coming from them, they woke somebody up at
> >> our request and found it listed as a 'bad route', whatever that
> >> means. Their workaround was to shut down the peering to us.
> >
> >> Anybody else use Cymru and have this issue?  Our 2 peering
> >> addresses are 38.229.66.20 and 193.231.140.82
> >
> >> Thanks.
> >
> >> -ben
> >
> > Hello Outages list!
> >
> > We have identified a hardware problem associated with the full
> > bogons peering sessions that were taken offline roughly 12 hours
> > ago. The service has been disabled and at this time we do not have
> > an ETA on when service will be restored.
> >
> > We will issue updates at approximately 8am, 12pm, and 5pm EST
> > until this outage is resolved.
> >
> > For the finer details please read below.
> >
> > 8pm EST 7/15/13, Team Cymru was notified of a couple problems with
> > our Bogon service. At this time we did not know the severity and
> > scope of the larger problem.
> >
> > 11pm EST 7/15/13 the issue was escalated from Operations to
> > Engineering.
> >
> > 1am EST 7/16/13, we confirmed the propagation of incorrect
> > prefixes. Workarounds failed to fully mitigate the issue, so all
> > sessions were disabled. This ceased the propagation of incorrect
> > prefixes.
> >
> > ~2am EST 7/16/13, we sent out notices to everyone we have current
> > contact details on regarding the outage.
> >
> > 12:15pm EST 7/16/13, we are working on replacing / reconfiguring
> > hardware to resolve this outage.
> >
> > On behalf of Team Cymru, I would like to apologize for this outage.
> > we are actively working on getting the Bogons back online for
> > distribution.
> >
> > In the interim, we are having listing updates periodically, it is
> > located here:
> >
> > https://www.team-cymru.org/Services/Bogons/
> >
> > For those who are using our service, thank you! Be assured we are
> > working towards resolution.
> >
> > Sincerely,
> >
> > Don Napier Team Cymru Research NFP
>
> Hello Outages,
>
> Service was restored tonight at about EST 530PM 7/16/13.  We replaced
> the faulty hardware (a router in this case) at our data center.  The
> Bogon feeds should be flowing properly now.
>
> In the future, status of the Bogon Reference Project will be listed at:
>
> https://www.team-cymru.org/Services/Bogons/index.html
>
> Thanks you for your patience,
>
> Don Napier
> Team Cymru Research NFP
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
> Comment: GPGTools - http://gpgtools.org
>
> iQEcBAEBAgAGBQJR5cZWAAoJEKgi/9FGk+IZhQ4IAJmdfHYkGyFq/pCLRpd29fFu
> knSeZ13/5TWPMpE1kiXFJv+rtkIFVcI0855nW8XpE6f5c7VvI+6NqMw1HXcY5twD
> 6gnD0rUAwesFTpTMk36iU0+REsKWCsXuItCjMtKr7S3kLNjSQ8ymwRSeh+Fbgqp6
> TYqyhjOwHsuFboII4iy18cg3Y4Gi1hP+sWOhTTSkvEEBxAQTM6xs0ENKXbPAsHkY
> 6tf/R31fREpg0HUcpUewYBv5jYrfUV2/e0Brr9ACkExhb9JxA3vNqBW8y+RXW7yu
> 66Om3LkD/4f3Wud04GohQRvr+vFe8cwK5ePTlSDX/S+kE677ItWaHLhwFyix/QI=
> =7QMB
> -----END PGP SIGNATURE-----
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20130716/f71beda4/attachment.htm>