Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable device map #30870

Merged
merged 2 commits into from
May 17, 2024
Merged

Enable device map #30870

merged 2 commits into from
May 17, 2024

Conversation

darshana1406
Copy link
Contributor

What does this PR do?

Fixes #30858

  • Added _no_split_modules = ["VideoLlavaVisionAttention"] to src/transformers/models/video_llava/modeling_video_llava.py
  • Works on a 4 GPU NVIDIA GeForce RTX 2080 Ti setup.

Who can review?

@zucchini-nlp @amyeroberts

@amyeroberts
Copy link
Collaborator

Thanks for adding this feature @darshana1406!

Could you run the following tests in a multi-gpu environment and share the terminal output?

pytest tests/models/video_llava/test_modeling_ video_llava.py -vv -k "offload or parallelism"

@zucchini-nlp
Copy link
Member

Awesome, thanks a lot for adding device map support!

@darshana1406
Copy link
Contributor Author

@amyeroberts Here is the terminal output.

(videocon) darshana.s@gnode084:~/transformers$ python -m pytest tests/models/video_llava/test_modeling_video_llava.py -vv -k "offload or parallelism"
============================================================= test session starts =============================================================
platform linux -- Python 3.10.14, pytest-8.2.0, pluggy-1.5.0 -- /home2/darshana.s/miniconda3/envs/videocon/bin/python
cachedir: .pytest_cache
rootdir: /home2/darshana.s/transformers
configfile: pyproject.toml
plugins: anyio-4.3.0
collected 125 items / 121 deselected / 4 selected                                                                                             

tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_cpu_offload <- tests/test_modeling_common.py PASSED [ 25%]
tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_disk_offload_bin <- tests/test_modeling_common.py PASSED [ 50%]
tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_disk_offload_safetensors <- tests/test_modeling_common.py PASSED [ 75%]
tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_model_parallelism <- tests/test_modeling_common.py PASSED [100%]

============================================================== warnings summary ===============================================================
../miniconda3/envs/videocon/lib/python3.10/site-packages/_pytest/config/__init__.py:1448
  /home2/darshana.s/miniconda3/envs/videocon/lib/python3.10/site-packages/_pytest/config/__init__.py:1448: PytestConfigWarning: Unknown config option: doctest_glob
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_cpu_offload
tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_cpu_offload
tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_disk_offload_bin
tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_disk_offload_bin
tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_disk_offload_safetensors
tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_model_parallelism
tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_model_parallelism
  /home2/darshana.s/miniconda3/envs/videocon/lib/python3.10/site-packages/accelerate/utils/modeling.py:1142: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    logger.warn(

tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_cpu_offload
tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_disk_offload_bin
tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_disk_offload_safetensors
  /home2/darshana.s/miniconda3/envs/videocon/lib/python3.10/site-packages/accelerate/utils/modeling.py:1363: UserWarning: Current model requires 262176 bytes of buffer for offloaded layers, which seems does not fit any GPU's remaining memory. If you are experiencing a OOM later, please consider using offload_buffers=True.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============================================== 4 passed, 121 deselected, 11 warnings in 25.69s ===============================================

amyeroberts
amyeroberts approved these changes May 17, 2024
View reviewed changes
Copy link
Collaborator

@amyeroberts amyeroberts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this really useful feature @darshana1406! ❤️

@amyeroberts amyeroberts merged commit 3802e78 into huggingface:main May 17, 2024
18 checks passed
@darshana1406
Copy link
Contributor Author

Thank you for guiding me! @zucchini-nlp @amyeroberts

itazap pushed a commit that referenced this pull request May 24, 2024
@darshana1406
Enable device map (#30870)
f830beb 
* added_no_split_modules

* added LlavaNextVisionAttention to _no_split_modules
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amyeroberts amyeroberts amyeroberts approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Enabling device_map="auto" for Video-LLaVA
3 participants
@darshana1406 @amyeroberts @zucchini-nlp