Improve corspearman performance (Matrix & Matrix-Matrix cases)

[PGS62]

if n = size(x,2) then this PR reduces the number of calls made by corspearman(x) to tiedrank from n + n(n-1)/2 to n. Likewise, if m = size(y,2) , the number of calls made by corspearman(x,y) falls from n + n*m to n + m. Hence a big speedup for multi-column input.

In the tests below, the new version of corspearman is in a package KendallTau.

julia> using StatsBase, KendallTau, BenchmarkTools

julia> x = rand(10,10);KendallTau.corspearman(x)==StatsBase.corspearman(x)#compile
true

julia> x = rand(1000,1000);

julia> @btime res_old = StatsBase.corspearman(x);
  18.005 s (2002003 allocations: 7.63 GiB)

julia> @btime res_new = KendallTau.corspearman(x);
  43.331 ms (4014 allocations: 38.50 MiB)

julia> res_new == res_old
true

After the change, the tests in test/rankcorr.jl all pass. The desired handling of NaN values is maintained too.

[PGS62]

Thank you @devmotion for looking at this PR.

My real interest is to improve the performance of corkendall, though whilst working on that I happened to notice the inefficiency in corspearman - hence this PR. If by any chance you have time to look at issue #849 that would be a great help.

[PGS62]

Hello @nalimilan and other maintainers of StatsBase,

Is it possible for this PR to be accepted while there are failures against Julia nightly? I'd like to make a PR for changes to corkendall and it would be simpler to do that if this PR had gone through, since the relevant code is in the same file.

Thanks!

[PGS62]

I plan to submit an alternative PR in which corspearman has better performance than the version in this PR. So I'm closing this PR.