[outages] 216.115.160.0/20 being blackholed

[Don Napier]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 7/16/13 1:29 PM, Don Napier wrote:
> 
> 
> On 7/16/13 7:29 AM, Ben Bartsch wrote:
>> FYI - got a call from a customer last night about a Xerox site 
>> unreachable through us.  Found the prefix coming from Cymru. 
>> Instructed our NOC to work with them to resolve.  Cymru first 
>> claimed they were not sending it because it was not a bogon.
>> Once we proved it was coming from them, they woke somebody up at
>> our request and found it listed as a 'bad route', whatever that
>> means. Their workaround was to shut down the peering to us.
> 
>> Anybody else use Cymru and have this issue?  Our 2 peering 
>> addresses are 38.229.66.20 and 193.231.140.82
> 
>> Thanks.
> 
>> -ben
> 
> Hello Outages list!
> 
> We have identified a hardware problem associated with the full
> bogons peering sessions that were taken offline roughly 12 hours
> ago. The service has been disabled and at this time we do not have
> an ETA on when service will be restored.
> 
> We will issue updates at approximately 8am, 12pm, and 5pm EST
> until this outage is resolved.
> 
> For the finer details please read below.
> 
> 8pm EST 7/15/13, Team Cymru was notified of a couple problems with
> our Bogon service. At this time we did not know the severity and
> scope of the larger problem.
> 
> 11pm EST 7/15/13 the issue was escalated from Operations to
> Engineering.
> 
> 1am EST 7/16/13, we confirmed the propagation of incorrect
> prefixes. Workarounds failed to fully mitigate the issue, so all
> sessions were disabled. This ceased the propagation of incorrect
> prefixes.
> 
> ~2am EST 7/16/13, we sent out notices to everyone we have current 
> contact details on regarding the outage.
> 
> 12:15pm EST 7/16/13, we are working on replacing / reconfiguring 
> hardware to resolve this outage.
> 
> On behalf of Team Cymru, I would like to apologize for this outage.
> we are actively working on getting the Bogons back online for
> distribution.
> 
> In the interim, we are having listing updates periodically, it is 
> located here:
> 
> https://www.team-cymru.org/Services/Bogons/
> 
> For those who are using our service, thank you! Be assured we are 
> working towards resolution.
> 
> Sincerely,
> 
> Don Napier Team Cymru Research NFP

Hello Outages,

Service was restored tonight at about EST 530PM 7/16/13.  We replaced
the faulty hardware (a router in this case) at our data center.  The
Bogon feeds should be flowing properly now.

In the future, status of the Bogon Reference Project will be listed at:

https://www.team-cymru.org/Services/Bogons/index.html

Thanks you for your patience,

Don Napier
Team Cymru Research NFP
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org

iQEcBAEBAgAGBQJR5cZWAAoJEKgi/9FGk+IZhQ4IAJmdfHYkGyFq/pCLRpd29fFu
knSeZ13/5TWPMpE1kiXFJv+rtkIFVcI0855nW8XpE6f5c7VvI+6NqMw1HXcY5twD
6gnD0rUAwesFTpTMk36iU0+REsKWCsXuItCjMtKr7S3kLNjSQ8ymwRSeh+Fbgqp6
TYqyhjOwHsuFboII4iy18cg3Y4Gi1hP+sWOhTTSkvEEBxAQTM6xs0ENKXbPAsHkY
6tf/R31fREpg0HUcpUewYBv5jYrfUV2/e0Brr9ACkExhb9JxA3vNqBW8y+RXW7yu
66Om3LkD/4f3Wud04GohQRvr+vFe8cwK5ePTlSDX/S+kE677ItWaHLhwFyix/QI=
=7QMB
-----END PGP SIGNATURE-----