Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create_gem rewrite with join #273

Open
JohnHadish opened this issue Jan 18, 2023 · 0 comments
Open

create_gem rewrite with join #273

JohnHadish opened this issue Jan 18, 2023 · 0 comments

Comments

@JohnHadish
Copy link
Collaborator

Description of feature

The create-gem.py script currently uses the pandas merge function, which is able to merge on any column. It turns out that the pandas join function is substantially faster, but is slightly more restrictive in that you can only join on the index.

join is essentially instantaneous, even for large matrices, where as merge becomes slow with large matrices. This is most important when you are dealing with massive matrices (greater than ~30,000 samples) where merge gets rather slow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant