New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Flaky unit tests: joblib parallel ValueError #167
Comments
This issue can possible deal with Dict() in 3.5 having insertion order. |
We should write a summary here of what appears to be inconsistent |
I can't seem to find another ticket but this issue might also be related/fix CircleCI inconsistency with parallelization in 3.5. Errors that could pop up can be found here. |
Awesome, thanks. What I see in the logs is that some of our unit tests are failing on python 3.5 but still passing on 3.6 and 3.7.
That would make sense. Next questions/tasks:
|
I started looking into this. Summary: able to repro unreliably/occasionally. Still not sure of the root cause. From sifting through the CircleCI results, it looks like this happens a small percentage of the time. Maybe 10-20%. Note: I filed #311 to track some warning messages I saw in the unit tests. May be related, unsure. Stack trace
Some possibilities
Stuff I tried I tried running some of the unit tests which had failed individually, with no luck. It was only when I ran all of them at once that I was able to repro some failures. But the tests which failed changed a bit each time and seemed unpredictable. Next steps
Not the cause
|
in terms of the necessity to support 3.5... currently 5-10% of our featuretools downloads come from python 3.5. i also looked at a few other ml-related libraries scikit-learn: ~10% so, my thought would be that yes, we should try to support it since there are people using it. if maintaining it is a slowing us down drastically, we could revisit that. checkout whatever package you want here: https://pypistats.org/packages/pandas |
Just saw another instance of this failure on my PR, here. It hasn't magically gone away :) we should dig into this soon |
We're removing support for python 3.5 in #435. But note @angela97lin mentioned she's seen this failure on python 3.6 💩 Updating issue name to correspond. RE comment in #435, I wonder if this issue has something to do with our use of OrderedDict... probably not, just adding to the list of possibilities. @angela97lin do you have any info / links / repro with the 3.6 failure you saw? Was it local or on circleci? |
Sure! I've only run into it via my random_state PR for python 3.6, so I've been trying to debug. Here's that PR: #431 From Slack thread: Here’s a run where I ran into this: https://app.circleci.com/jobs/github/FeatureLabs/evalml/12366 It seems to only happen on CircleCI, so I wonder if that has to do with the issue at all? |
Status We were previously seeing this failure only on python 3.5. Now Angela tweaking the random_state causes this to fail on python 3.6 only. This makes me think there's a race condition which has to do with the ordering of calls to the random number generator. It's quite helpful that it appears to be failing consistently on python 3.6 on Angela's random_state branch. Next steps
|
We should reevaluate if this is still an issue now that @christopherbunn merged #407 |
I haven't seen this issue since #407 was merged. Closing. |
Why are our results for 3.6 and 3.7 consistent but not 3.5?
The text was updated successfully, but these errors were encountered: