Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separation test precision on OSX #376

Open
bmcfee opened this issue Mar 26, 2024 · 0 comments
Open

Separation test precision on OSX #376

bmcfee opened this issue Mar 26, 2024 · 0 comments
Milestone

Comments

@bmcfee
Copy link
Collaborator

bmcfee commented Mar 26, 2024

Looks like we're hitting numerical precision issues again, sometimes, in the source separation tests on OSX. See https://github.com/craffel/mir_eval/actions/runs/8426164125/job/23106646394?pr=374

The tests passed previously (eg in #370), and since we're now seeing the RNG before every test function execution, the deviation must be coming from the unerlying blas implementation and/or some interaction with the hardware on which it's deploying.

I've been seeing similar weirdness in other packages (pescador, librosa) lately, and there are some strange things happening with openblas and xsimd that expose the non-associativity of floating point arithmetic in cases like this.

I propose to "fixing" this by detecting the execution platform and raising the atol parameter on OSX deployments in test_separation.py. Since the separation metrics are in decibels, I don't think we should be too concerned about raising the tolerance from 0.01dB to 0.05dB, and keeping the stricter tolerance on better-behaved platforms should keep us safe.

Meanwhile, I don't think separation test failures on osx should be a blocker to merging unrelated PRs (eg #374).

@bmcfee bmcfee added this to the 0.8 milestone Mar 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant