changes for audio/video support in hub.ingest #1643

neel2299 · 2022-05-01T12:58:20Z

🚀 🚀 Pull Request

Checklist:

My code follows the style guidelines of this project and the Contributing document
I have commented my code, particularly in hard-to-understand areas
I have kept the coverage-rate up
I have performed a self-review of my own code and resolved any problems
I have checked to ensure there aren't any other open Pull Requests for the same change
I have described and made corresponding changes to the relevant documentation
New and existing unit tests pass locally with my changes

Changes

PR for issue #1556

neel2299 · 2022-05-02T10:01:26Z

Tagging @tatevikh for visibility.

tatevikh · 2022-05-02T10:23:14Z

Hi @neel2299 thanks for your contribution. We'll review it and get back to you:)

hub/auto/unstructured/image_classification.py

tatevikh · 2022-07-19T12:00:21Z

Hi @neel2299 . How is the progress with this PR? Do you need any help here? Pleas do let us know:)

neel2299 · 2022-07-20T19:29:28Z

Hi @tatevikh, thanks for reminding me. It skipped my mind that I had this PR open, I will get this done as soon as possible.

FayazRahman

Few comments, also fix merge conflicts

FayazRahman · 2022-08-15T08:49:12Z

hub/api/dataset.py

@@ -31,6 +31,29 @@
 from hub.util.cache_chain import generate_chain
 from hub.core.storage.hub_memory_object import HubMemoryObject

+IMAGE_FORMAT_NAME = [


these are already listed in hub/compression.py. Could you use that?

Yep, I have updated it.

Can you reuse that list instead of listing it here again?

Ah, my bad! I had switched to using the list from compression.py but forgot to remove list here.
Removed it now!

hub/auto/unstructured/classification.py

FayazRahman · 2022-09-03T18:42:44Z

hub/api/dataset.py

 from hub.util.storage import get_storage_and_cache_chain, storage_provider_from_path
 from hub.util.compute import get_compute_provider
 from hub.util.remove_cache import get_base_storage
 from hub.util.cache_chain import generate_chain
 from hub.core.storage.hub_memory_object import HubMemoryObject

+IMAGE_COMPRESSIONS = IMAGE_COMPRESSIONS.copy()


maybe you can do something like _image_compressions in hub/htype.py instead of this. https://github.com/activeloopai/Hub/blob/main/hub/htype.py#L208

Ah! That makes more sense. Forgot you use upper case only for constants.

No, I meant you need to include the other compression aliases other than jpg, like in the mentioned file

I removed "dcm" like in htype.py as well. Hope this works.

codecov · 2022-09-16T11:48:01Z

Codecov Report

Base: 92.60% // Head: 92.60% // Decreases project coverage by -0.00% ⚠️

Coverage data is based on head (d26c5d8) compared to base (72bc6b4).
Patch coverage: 82.14% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1643      +/-   ##
==========================================
- Coverage   92.60%   92.60%   -0.01%     
==========================================
  Files         245      245              
  Lines       25984    26023      +39     
==========================================
+ Hits        24063    24098      +35     
- Misses       1921     1925       +4

Flag	Coverage Δ
unittests	`92.60% <82.14%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
hub/auto/tests/test_ingestion.py	`98.71% <ø> (ø)`
hub/integrations/wandb/wandb.py	`48.05% <40.00%> (-0.21%)`	⬇️
hub/api/dataset.py	`89.45% <73.33%> (-0.92%)`	⬇️
hub/auto/unstructured/classification.py	`87.23% <82.35%> (ø)`
hub/__init__.py	`94.73% <100.00%> (ø)`
hub/compression.py	`97.95% <100.00%> (+0.23%)`	⬆️
hub/constants.py	`100.00% <100.00%> (ø)`
hub/core/compression.py	`89.64% <100.00%> (ø)`
hub/experimental/convert_to_hub3.py	`90.24% <100.00%> (+10.75%)`	⬆️
hub/experimental/test_pytorch.py	`99.21% <100.00%> (+0.01%)`	⬆️
... and 3 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

FayazRahman · 2022-09-19T08:35:06Z

@neel2299 Looks good so far, could you also tests for these? See this file for example: https://github.com/activeloopai/Hub/blob/main/hub/auto/tests/test_ingestion.py#L63

adolkhan · 2022-09-20T09:59:00Z

Dear @neel2299, agree with @FayazRahman, could you please add some tests? Lines of code that need tests are highlighted with codecove, you can take a look at it in Files Changed section

neel2299 · 2022-09-21T06:22:49Z

Makes sense @FayazRahman @adolkhan !! I will add them.

tatevikh · 2022-12-16T18:11:05Z

@neel2299 let me know if you'll be working on this otherwise I'll close this PR for now.

tatevikh · 2022-12-30T03:26:52Z

Closing for now.

neel2299 added 4 commits May 1, 2022 18:24

changes for audio/video supprot

29f31ec

black formatting

3a7d9de

modifying docstring

0ed183e

black once again

b055c7c

tatevikh requested a review from FayazRahman May 2, 2022 10:23

FayazRahman suggested changes May 2, 2022

View reviewed changes

hub/auto/unstructured/image_classification.py Outdated Show resolved Hide resolved

seperated into classes

049e217

neel2299 requested a review from FayazRahman August 4, 2022 12:46

FayazRahman reviewed Aug 15, 2022

View reviewed changes

neel2299 added 2 commits August 20, 2022 22:23

black and some changes

05811d3

conflicts

19b79a4

neel2299 requested a review from FayazRahman August 20, 2022 18:08

neel2299 added 3 commits August 21, 2022 00:27

darglint

b22caf4

deleted unused var

0203ad3

mypy fix

973c6f6

FayazRahman added the trigger-test Label trigger to run tests on PRs label Aug 31, 2022

activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Aug 31, 2022

neel2299 added 2 commits September 2, 2022 12:52

tests

1f65bfb

black

98764a3

FayazRahman added the trigger-test Label trigger to run tests on PRs label Sep 2, 2022

activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Sep 2, 2022

compression_lis_copy

1d78612

FayazRahman reviewed Sep 3, 2022

View reviewed changes

lil changes

5f3f803

neel2299 requested review from FayazRahman and removed request for FayazRahman September 5, 2022 06:34

neel2299 added 2 commits September 14, 2022 16:51

var change

48e71aa

black

92a9032

neel2299 force-pushed the master branch from d59e855 to 92a9032 Compare September 14, 2022 12:29

neel2299 and others added 2 commits September 14, 2022 19:46

Merge branch 'main' into master

f3c9e99

kaggle

d26c5d8

FayazRahman added the trigger-test Label trigger to run tests on PRs label Sep 16, 2022

activeloop-bot removed the trigger-test Label trigger to run tests on PRs label Sep 16, 2022

tatevikh requested a review from adolkhan September 19, 2022 08:29

FayazRahman approved these changes Sep 19, 2022

View reviewed changes

FayazRahman self-requested a review September 19, 2022 08:31

tatevikh closed this Dec 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

changes for audio/video support in hub.ingest #1643

changes for audio/video support in hub.ingest #1643

neel2299 commented May 1, 2022 •

edited

neel2299 commented May 2, 2022

tatevikh commented May 2, 2022

tatevikh commented Jul 19, 2022

neel2299 commented Jul 20, 2022

FayazRahman left a comment

FayazRahman Aug 15, 2022

neel2299 Aug 20, 2022 •

edited

FayazRahman Aug 26, 2022

neel2299 Aug 26, 2022

FayazRahman Sep 3, 2022

neel2299 Sep 4, 2022

FayazRahman Sep 6, 2022

neel2299 Sep 14, 2022

codecov bot commented Sep 16, 2022 •

edited

FayazRahman commented Sep 19, 2022

adolkhan commented Sep 20, 2022

neel2299 commented Sep 21, 2022 •

edited

tatevikh commented Dec 16, 2022

tatevikh commented Dec 30, 2022

changes for audio/video support in hub.ingest #1643

changes for audio/video support in hub.ingest #1643

Conversation

neel2299 commented May 1, 2022 • edited

🚀 🚀 Pull Request

Checklist:

Changes

neel2299 commented May 2, 2022

tatevikh commented May 2, 2022

tatevikh commented Jul 19, 2022

neel2299 commented Jul 20, 2022

FayazRahman left a comment

Choose a reason for hiding this comment

FayazRahman Aug 15, 2022

Choose a reason for hiding this comment

neel2299 Aug 20, 2022 • edited

Choose a reason for hiding this comment

FayazRahman Aug 26, 2022

Choose a reason for hiding this comment

neel2299 Aug 26, 2022

Choose a reason for hiding this comment

FayazRahman Sep 3, 2022

Choose a reason for hiding this comment

neel2299 Sep 4, 2022

Choose a reason for hiding this comment

FayazRahman Sep 6, 2022

Choose a reason for hiding this comment

neel2299 Sep 14, 2022

Choose a reason for hiding this comment

codecov bot commented Sep 16, 2022 • edited

Codecov Report

FayazRahman commented Sep 19, 2022

adolkhan commented Sep 20, 2022

neel2299 commented Sep 21, 2022 • edited

tatevikh commented Dec 16, 2022

tatevikh commented Dec 30, 2022

neel2299 commented May 1, 2022 •

edited

neel2299 Aug 20, 2022 •

edited

codecov bot commented Sep 16, 2022 •

edited

neel2299 commented Sep 21, 2022 •

edited