Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reducing qcodes import times #4616

Merged
merged 43 commits into from
Oct 4, 2022
Merged

Conversation

edumur
Copy link
Contributor

@edumur edumur commented Sep 14, 2022

Hi all,

Following this discussion #4565 and this issue #4543 I am proposing this pull request to work on reducing qcodes loading time.

I have started with pandas.
I tried to propagate what @jenshnielsen did to make pandas import lazy-import.
I am not sure I have succeeded since I still see pandas being imported while doing importtime-waterfall.
I would gladly improve the request if someone help me on this.

If this becomes a successful endeavor, I will continue by making other import lazy-import.

@edumur
Copy link
Contributor Author

edumur commented Sep 14, 2022

And also, I do not no how to run qcodes test and check if I didn't break anything ^^'.
Again, if someone may help me.

@jenshnielsen
Copy link
Collaborator

Thanks, I will try to have a look asab and see if I can figure out why/if pandas is still being imported eagerly

edumur and others added 3 commits September 16, 2022 10:35
Since xarray imports pandas, this also make pandas lazy import
@edumur
Copy link
Contributor Author

edumur commented Sep 16, 2022

Found out that pandas was imported by xarray so I passed xarray in lazy import.

On the data_set file, I got some error and had to write the type of some parameters as 'xr.Dataset', see line 693 for example. I do not know why and if this is correct.

@edumur
Copy link
Contributor Author

edumur commented Sep 16, 2022

Yep, forget what I wrote, it is fixed.

@jenshnielsen
Copy link
Collaborator

Looks good. Did you verify that you need to also make changes to the tests. Importing qcodes should not import the tests module. If it does that seems like a bug we should also fix.

@edumur
Copy link
Contributor Author

edumur commented Sep 16, 2022

You right, test file do not matters.
Could you remind me how to run the test locally?

@edumur
Copy link
Contributor Author

edumur commented Sep 16, 2022

Alright there is maybe the import of Monitor to work on but I am waiting opinion(s) from others.
On my pc it looks like it is 20% faster, can anyone look on their machine?

Copy link
Contributor

@eendebakpt eendebakpt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up! Can you fix the last linting errors? I will test import time on my machine
(and the branch needs to be rebased to master)

qcodes/data/data_set.py Outdated Show resolved Hide resolved
qcodes/tests/dataset/helper_functions.py Outdated Show resolved Hide resolved
qcodes/tests/dataset/helper_functions.py Outdated Show resolved Hide resolved
@edumur
Copy link
Contributor Author

edumur commented Sep 29, 2022

Should be good

@edumur
Copy link
Contributor Author

edumur commented Oct 3, 2022

Done.

I did not get what you meant by "documentation about where to import it from"...

@jenshnielsen
Copy link
Collaborator

Looks good. thanks @edumur

@microsoft microsoft deleted a comment from bors bot Oct 4, 2022
@jenshnielsen
Copy link
Collaborator

bors merge

@bors
Copy link
Contributor

bors bot commented Oct 4, 2022

🕐 Waiting for PR status (Github check) to be set, probably by CI. Bors will automatically try to run when all required PR statuses are set.

@jenshnielsen
Copy link
Collaborator

bors merge

@bors bors bot merged commit 77f1609 into microsoft:master Oct 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants