New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Simplifying dependencies #323
Conversation
Minimum allowed coverage is Generated by 🐒 cobertura-action against 04e7a02 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See below for more specifics. In general, I'm allowing the latest versions of things to be installed again. Having so many dependencies was causing later versions of pip to run around in circles trying to find compatible combinations of dependencies.
'yamale>=2.0', | ||
'ruamel.yaml>=0.15.0', | ||
'testfixtures', | ||
's3fs[boto3]', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We had to pin s3fs to a really old version so we could have boto3 and s3fs coexist. Turns out we weren't the only ones with this problem so they made a way to resolve the conflict, which we're using now. This allows us to use a newer version of s3fs among other things.
which pip | ||
if [ $DEV -eq 1 ] | ||
then | ||
pip install --ignore-installed -e . | ||
pip install --no-cache-dir --ignore-installed -e . |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Keeps us from filling up a user's home directory with cached pip wheels. IYKYK.
conda create -y --prefix "$MY_CONDA_PREFIX" -c conda-forge "pyarrow>=7.0.0" "python=3.9" "numpy>=1.20.0" "pandas>=1.0.0,!=1.0.4" "dask>=2022.10.0" "distributed>=2021.5" ruby | ||
source deactivate | ||
source activate "$MY_CONDA_PREFIX" | ||
mamba create -y --prefix "$MY_CONDA_PREFIX" -c conda-forge "python=3.10" "pyarrow" "numpy" "pandas" "dask>=2022.10.0" "distributed" "ruby" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Turns out they have mamba now, which makes things a lot faster.
conda deactivate | ||
conda activate "$MY_CONDA_PREFIX" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Swtiching to conda deactivate
which we were getting warnings about.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @nmerket. Looks good to me!
Pull Request Description
Removes most of the dependency version pins so it'll be able to update more easily.
Checklist
Not all may apply
Tests exercising your feature/bug fix (check coverage report on Checks -> BuildStockBatch Tests -> Artifacts)minimum_coverage
in.github/workflows/ci.yml
as necessary.Update validation for project config yaml file changesUpdate existing documentation