Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dataloader improvements #2806

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open

Dataloader improvements #2806

wants to merge 36 commits into from

Conversation

activesoull
Copy link
Contributor

@activesoull activesoull commented Mar 21, 2024 •
edited

🚀 🚀 Pull Request

Impact

  • Bug fix (non-breaking change which fixes expected existing functionality)
  • Enhancement/New feature (adds functionality without impacting existing logic)
  • Breaking change (fix or feature that would cause existing functionality to change)

Description

  • Improved dataloader multiprocessing
    - reduced IPC overhead
    - removed ShuffleBuffer filling when the shuffle is True but the shuffle buffer logic is still there
  • added nii, nii.gz and dcm support

Things to be aware of

Things to worry about

Additional Context

@codecov Codecov
Copy link

codecov bot commented Mar 21, 2024 •
edited

Codecov Report

Attention: Patch coverage is 85.93750% with 9 lines in your changes are missing coverage. Please review.

Files Patch % Lines
deeplake/enterprise/dataloader.py 73.68% 5 Missing ⚠️
deeplake/integrations/pytorch/common.py 77.77% 2 Missing ⚠️
deeplake/core/dataset/deeplake_query_dataset.py 80.00% 1 Missing ⚠️
deeplake/core/storage/gcs.py 0.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@activesoull activesoull changed the title Torch dl Dataloader improvemements Mar 26, 2024
@activesoull activesoull changed the title Dataloader improvemements Dataloader improvements Mar 26, 2024
@activesoull activesoull added the full-tests Trigger Full versioned OS tests label Mar 26, 2024
@activesoull activesoull added full-tests Trigger Full versioned OS tests and removed full-tests Trigger Full versioned OS tests labels Mar 28, 2024
@activesoull activesoull added full-tests Trigger Full versioned OS tests and removed full-tests Trigger Full versioned OS tests labels Mar 29, 2024
@activesoull activesoull added full-tests Trigger Full versioned OS tests and removed full-tests Trigger Full versioned OS tests labels Apr 1, 2024
@activesoull activesoull closed this Apr 2, 2024
@activesoull activesoull reopened this Apr 2, 2024
@activesoull activesoull added full-tests Trigger Full versioned OS tests and removed full-tests Trigger Full versioned OS tests labels Apr 2, 2024
@sonarcloud SonarCloud
Copy link

sonarcloud bot commented Apr 24, 2024

Quality Gate Passed Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
47.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
full-tests Trigger Full versioned OS tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@activesoull @khustup2