Get Fashion Mnist 96% recall up to 200 queries/second

[alexklibisz]

I'd like to optimize Elastiknn such that the Fashion Mnist benchmark performance exceeds 200 qps at 96% recall. Currently it's at 180 qps. So this would be about an 11% improvement. There are already several issues under the performance label with ideas towards this goal. I've already merged a few PRs. I'm just opening this issue to formalize the effort and to aggregate PRs that don't otherwise have an issue.

[alexklibisz]

We're now above 200 qps. Last thing left is to just submit a PR to ann-benchmarks w/ the updated versions.

[alexklibisz]

Status update here, after releasing 8.12.2.1.

The non-containerized benchmark is reliably over 200qps @ 96% recall, around 210 qps. Latest update here: ddf637a

The containerized benchmark (running ann-benchmarks and elastiknn in the same container) has improved from ~160qps to ~180qps.

Here are the results using 8.6.2.0:

Model	Parameters	Recall	Queries per Second
eknn-l2lsh	L=100 k=4 w=1024 candidates=500 probes=0	0.378	304.111
eknn-l2lsh	L=100 k=4 w=1024 candidates=1000 probes=0	0.445	246.319
eknn-l2lsh	L=100 k=4 w=1024 candidates=500 probes=3	0.635	245.977
eknn-l2lsh	L=100 k=4 w=1024 candidates=1000 probes=3	0.716	201.608
eknn-l2lsh	L=100 k=4 w=2048 candidates=500 probes=0	0.767	265.545
eknn-l2lsh	L=100 k=4 w=2048 candidates=1000 probes=0	0.846	218.673
eknn-l2lsh	L=100 k=4 w=2048 candidates=500 probes=3	0.921	184.178
eknn-l2lsh	L=100 k=4 w=2048 candidates=1000 probes=3	0.960	160.437

Here are the results using 8.12.2.1:

Model	Parameters	Recall	Queries per Second
eknn-l2lsh	L=100 k=4 w=1024 candidates=500 probes=0	0.378	314.650
eknn-l2lsh	L=100 k=4 w=1024 candidates=1000 probes=0	0.446	247.659
eknn-l2lsh	L=100 k=4 w=1024 candidates=500 probes=3	0.634	258.834
eknn-l2lsh	L=100 k=4 w=1024 candidates=1000 probes=3	0.716	210.380
eknn-l2lsh	L=100 k=4 w=2048 candidates=500 probes=0	0.767	271.442
eknn-l2lsh	L=100 k=4 w=2048 candidates=1000 probes=0	0.846	221.127
eknn-l2lsh	L=100 k=4 w=2048 candidates=500 probes=3	0.921	199.353
eknn-l2lsh	L=100 k=4 w=2048 candidates=1000 probes=3	0.960	171.614