[nsp-sec] Daily Reports Delays Continue

[Tim Wilde]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Good afternoon everyone,

I apologize for the lack of updates throughout the day today, but it has
been a busy day, and there hasn't been much to report.

The first phase of the reporting process was again problematic today.
It completed around 4:30pm EST (GMT-0500), though, and the
generation/e-mailing has begun.  Yesterday this took approximately 5
hours to complete.  Today's data set is nearly 25% larger, so I expect
it to take proportionally longer.  I do not expect to be awake to report
the completion of the process, but you can expect it sometime around
midnight EST if that estimate holds true.

Now, for some good news (or at least, different news).  While I had been
avoiding this up to this point, we have made the decision to implement
some sampling of the data sources that have been causing the most
trouble.  What this means is that we will be sending out reports that
include only a portion of the hits for a given IP address each day.  It
is important to note that this change ONLY affects the bots category,
and ONLY affects certain data sources within that category.  We are
making efforts to ensure that a reasonable spread of timestamp
information is provided so that customers behind NATs and proxies can
still be identified.  Additionally, we will be retaining the unsampled
source information (just keeping it outside of our database, and thus
not suitable for reporting on), which we may be able to provide
unsampled timestamps from if needed.  (No firm promises on that, though
- - time pressures may not always allow it, and that's still a ton of data
to search through.)

We understand that sampling will be a less-than-optimal solution for
those with customers behind NATs or proxies; we had to make this
decision to allow us to get the service back to a more normally
functioning state for all of our subscribers.  Again, we have made
efforts to ensure that the data remains as useful as possible for those
in this situation, and welcome feedback as to whether or not
identification becomes more difficult when this change starts appearing
with tomorrow's reports.

Finally, I just want to apologize again for the inconvenience these
issues continue to cause.  Rest assured that we are working diligently
to resolve them in a more complete manner as soon as possible.  We
appreciate your ongoing patience and support.  As always, all
suggestions, comments, questions, or concerns are welcome to
team-cymru at cymru.com.

Regards,
Tim Wilde

- --
Tim Wilde, Senior Software Engineer, Team Cymru, Inc.
twilde at cymru.com | +1-312-924-4033 | http://www.team-cymru.org/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJoznnluRbRini9tgRAtyPAJ9durQrGCv/5IwHmzv7qe2HuRVaCgCeM8Sp
pRDXIYXmqaWPCSJWFUAUvGk=
=sO/F
-----END PGP SIGNATURE-----