Benchmark

[sovaa]

Simple benchmark for original(). More benchmarking could be done if originality.py abstracts away the managers or by implementing in-memory versions of them.

Requires two new dependencies, multiprocessing and randomstate (drop-in replacement for numpy's RandomState that plays well with multiprocessing).

[philipcmonk]

Can you add the dependencies to setup.py and requirements.txt?

[philipcmonk]

It would also be nice to have this for concordance.

[philipcmonk]

Can we benchmark each user against, say, 1000 other users rather than generate new pairs each time? There may be some potential gains in pre-processing the data.

[sovaa]

Can you include e.g. 50 user submissions to the repo? Then the test can add random noise to them to create a few thousands more 'almost unique' during setup. Could be a gziped file to be decompressed during setup as well.

[philipcmonk]

That's a good idea. I'll create an issue for it, but for the time being, generating them in the way you're doing it looks fine (except that we should check each user against 1000 other users).

[philipcmonk]

I realized my description above may not be very clear. I mean that the benchmark should capture this sort of optimization: #12

[sovaa]

• Added benchmark_concordance.py for Create performance benchmark #2
• Added 1500 rows of sample train, predict and results data for tests to sample from for Create example predictions for testing #15

The concordance benchmark is sampling in batched from the example data and adding normally distributed noise to create larger benchmark data.

If the PR is accepted for the bounty my numerai account is sovaa.

[philipcmonk]

Looks good, except the one issue.

[philipcmonk]

As much as I like the new number format, we use Python 3.5 internally, so this should be 100000.

[sovaa]

Ah, runtime.txt said 3.6.1 sot that's why. I'll change the number format.

[philipcmonk]

Thanks! I've sent 30 NMR to sovaa. It should be in your account now.