Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelization of build_all.py #175

Merged
merged 52 commits into from
May 31, 2024
Merged

Parallelization of build_all.py #175

merged 52 commits into from
May 31, 2024

Conversation

jjacobson95
Copy link
Collaborator

@jjacobson95 jjacobson95 commented May 10, 2024

Everything is working. Ready to Merge.

  • All Docker Images are now built in parallel using docker compose.
  • Samples and Drugs file generation should now run in parallel.
  • All Omics and Experiments file generation should run in parallel once all samples and drugs are completed.
  • CPTAC version updated to the latest ver. The 1.5.8 version was not working for me, so I updated to 1.5.11 and it worked - although sounds like this may not work for you, based on cptac issue #60?
  • Push to Figshare is working. Required arguments: --figshare --version
    Push to PyPI is working. Required arguments: --pypy --version
  • The version argument expects a value such as 0.1.28.
  • high_mem argument for full parallelization. Exclude it for semi-parallelization.
  • Schema checker integrated and each dataset runs in parallel.
  • README.md file updated

In summary, this may be run on a high memory platform, fully validate, upload to figshare and and upload to pypi with the following commands:

export SYNAPSE_AUTH_TOKEN="..."
export PYPI_TOKEN="..."
export FIGSHARE_TOKEN="..."

python build/build_all.py --all --high_mem --pypi --figshare --version 0.1.29

Edit -
Also manually updated all files needed to update github pages which is now live.

…ipt. I think it is the unusual characters in the syn token. This fixes
@jjacobson95
Copy link
Collaborator Author

@sgosline This should be ready to go. Full with with high_mem (and without) was successful. Latest HCMI changes haven't been tested yet, but that shouldn't impact this PR specifically.

This took ~16 hours from start to finish on a c5.9xlarge EC2 instance (36vCPUs, 72Gb Memory).

@jjacobson95 jjacobson95 added the enhancement New feature or request label May 24, 2024
@jjacobson95
Copy link
Collaborator Author

One note, the build/docker/Dockerfile.upload file will have to be updated to remove this line after testing is complete.
RUN git checkout docker-build-multi

@sgosline sgosline merged commit 3427edb into main May 31, 2024
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

None yet

2 participants