add gaudi modeling support in itrex #1438

ClarkChin08 · 2024-03-29T06:11:51Z

Type of Change

gaudi modeling used in itrex for int4 kv-cache support

Signed-off-by: Clark Chin <xi2.chen@intel.com>

github-actions · 2024-03-29T06:12:19Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 Format Scan Tests workflow

Check ID	Status
format-scan (pylint)	success	✅
format-scan (bandit)	success	✅
format-scan (cloc)	success	✅
format-scan (cpplint)	success	✅

These checks are required after the changes to intel_extension_for_transformers/neural_chat/examples/finetuning/multi_modal/eval/mmmu_eval/run_llava.py, intel_extension_for_transformers/neural_chat/models/model_utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/configuration_utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/stopping_criteria.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/modeling_utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/albert/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/albert/modeling_albert.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bart/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bart/modeling_bart.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/modeling_blip.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/modeling_blip_text.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bloom/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bloom/modeling_bloom.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/codegen/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/codegen/modeling_codegen.py and 39 more files...

🟢 Optimize Unit Test workflow

Check ID	Status
optimize-unit-test-baseline	success	✅
optimize-unit-test-PR-test	success	✅
Genreate-OptimizeUT-Report	success	✅

These checks are required after the changes to intel_extension_for_transformers/transformers/modeling/modeling_gaudi/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/configuration_utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/stopping_criteria.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/modeling_utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/albert/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/albert/modeling_albert.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bart/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bart/modeling_bart.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/modeling_blip.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/modeling_blip_text.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bloom/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bloom/modeling_bloom.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/codegen/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/codegen/modeling_codegen.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/esm/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/esm/modeling_esmfold.py and 36 more files...

🟢 NeuralChat Unit Test

Check ID	Status
neuralchat-unit-test-baseline	success	✅
neuralchat-unit-test-PR-test	success	✅
Generate-NeuralChat-Report	success	✅

These checks are required after the changes to intel_extension_for_transformers/neural_chat/models/model_utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/configuration_utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/stopping_criteria.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/modeling_utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/albert/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/albert/modeling_albert.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bart/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bart/modeling_bart.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/modeling_blip.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/modeling_blip_text.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bloom/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bloom/modeling_bloom.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/codegen/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/codegen/modeling_codegen.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/esm/__init__.py and 37 more files...

🟢 Engine Unit Test workflow

Check ID	Status
engine-unit-test-baseline	success	✅
engine-unit-test-PR-test	success	✅
Genreate-Engine-Report	success	✅

These checks are required after the changes to intel_extension_for_transformers/transformers/modeling/modeling_gaudi/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/configuration_utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/stopping_criteria.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/modeling_utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/albert/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/albert/modeling_albert.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bart/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bart/modeling_bart.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/modeling_blip.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/modeling_blip_text.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bloom/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bloom/modeling_bloom.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/codegen/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/codegen/modeling_codegen.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/esm/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/esm/modeling_esmfold.py and 36 more files...

🟢 Chat Bot Test workflow

Check ID	Status	Error details
call-inference-llama-2-7b-chat-hf / inference test	success		✅
call-inference-mpt-7b-chat / inference test	success		✅

These checks are required after the changes to intel_extension_for_transformers/neural_chat/models/model_utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/configuration_utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/stopping_criteria.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/generation/utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/modeling_utils.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/albert/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/albert/modeling_albert.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bart/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bart/modeling_bart.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/modeling_blip.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/blip/modeling_blip_text.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bloom/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/bloom/modeling_bloom.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/codegen/__init__.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/codegen/modeling_codegen.py, intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/esm/__init__.py and 37 more files...

Thank you for your contribution! 💜

Note
This comment is automatically generated and will be updates every 180 seconds within the next 6 hours. If you have any other questions, contact VincyZhang or XuehaoSun for help.

for more information, see https://pre-commit.ci

...ion_for_transformers/neural_chat/examples/finetuning/multi_modal/eval/mmmu_eval/run_llava.py

intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/__init__.py

Signed-off-by: Chen Xi <xi2.chen@intel.com>

for more information, see https://pre-commit.ci

Signed-off-by: Chen Xi <xi2.chen@intel.com>

for more information, see https://pre-commit.ci

airMeng

no requirements updated? At least optimum-habana shall be added

Signed-off-by: Meng, Hengyu <hengyu.meng@intel.com>

Signed-off-by: Chen Xi <xi2.chen@intel.com>

ClarkChin08 · 2024-04-25T05:40:59Z

@lkk12014402 kaokao, please take a look on the new commit on measurement of ppl

Signed-off-by: Chen Xi <xi2.chen@intel.com>

* initial commit Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * mv example Signed-off-by: Yu Zhentao <zhentao.yu@intel.com> * update model dtype Signed-off-by: Yu Zhentao <zhentao.yu@intel.com> * fix multi-round generation without streaming_llm Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * add mem and token num log Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * rebase Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * initial fp8 Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * add ppl eval scripts Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * typo Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * add llama2-13b ppl eval script (align paper) Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * hide kv cache operation inside (v0.1) Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * hide kv cache operation inside (v0.2) Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * hide kv cache operation inside (v0.3) Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * update scripts Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * add README Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * update test scripts Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * remove useless code Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> * update README and rename shell scripts Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> --------- Signed-off-by: Yu, Zhentao <zhentao.yu@intel.com> Signed-off-by: Yu Zhentao <zhentao.yu@intel.com>

for more information, see https://pre-commit.ci

Signed-off-by: Clark Chin <xi2.chen@intel.com>

for more information, see https://pre-commit.ci

Signed-off-by: Chen Xi <xi2.chen@intel.com>

for more information, see https://pre-commit.ci

Signed-off-by: Clark Chin <xi2.chen@intel.com>

Signed-off-by: Chen Xi <xi2.chen@intel.com>

Signed-off-by: Clark Chin <xi2.chen@intel.com>

Signed-off-by: VincyZhang <wenxin.zhang@intel.com>

Signed-off-by: Clark Chin <xi2.chen@intel.com>

for more information, see https://pre-commit.ci

Signed-off-by: Clark Chin <xi2.chen@intel.com>

airMeng

Great start on HPU

Signed-off-by: Clark Chin <xi2.chen@intel.com>

add gaudi modeling support in itrex

56a2893

Signed-off-by: Clark Chin <xi2.chen@intel.com>

ClarkChin08 requested review from PenghuiCheng and lvliang-intel as code owners March 29, 2024 06:11

[pre-commit.ci] auto fixes from pre-commit.com hooks

e0613ad

for more information, see https://pre-commit.ci

ClarkChin08 requested review from airMeng, mengniwang95 and xin3he April 1, 2024 01:17

airMeng reviewed Apr 1, 2024

View reviewed changes

...ion_for_transformers/neural_chat/examples/finetuning/multi_modal/eval/mmmu_eval/run_llava.py Show resolved Hide resolved

intel_extension_for_transformers/transformers/modeling/modeling_gaudi/models/__init__.py Show resolved Hide resolved

ClarkChin08 and others added 4 commits April 9, 2024 11:22

Add test example to itrex

69e81b3

Signed-off-by: Chen Xi <xi2.chen@intel.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

6454315

for more information, see https://pre-commit.ci

add fp8 support and fix bugs

72a4910

Signed-off-by: Chen Xi <xi2.chen@intel.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

4ad3b04

for more information, see https://pre-commit.ci

airMeng reviewed Apr 22, 2024

View reviewed changes

airMeng and others added 2 commits April 24, 2024 16:23

Create requirements.txt

63f0fc0

Signed-off-by: Meng, Hengyu <hengyu.meng@intel.com>

add ppl measurement in gaudi

1da9dfb

Signed-off-by: Chen Xi <xi2.chen@intel.com>

ClarkChin08 requested a review from lkk12014402 April 25, 2024 05:40

fix the ppl acc issue

fb2f7cc

Signed-off-by: Chen Xi <xi2.chen@intel.com>

VincyZhang added the habana label May 13, 2024

zhentaoyu and others added 7 commits May 22, 2024 09:57

[pre-commit.ci] auto fixes from pre-commit.com hooks

3a934c5

for more information, see https://pre-commit.ci

fix the pylint issue

fb2966e

Signed-off-by: Clark Chin <xi2.chen@intel.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

de64700

for more information, see https://pre-commit.ci

fix the pylint issue

455e5c3

Signed-off-by: Chen Xi <xi2.chen@intel.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

83a42b2

for more information, see https://pre-commit.ci

add optimum-habana when pylint

80476f1

Signed-off-by: Clark Chin <xi2.chen@intel.com>

ClarkChin08 requested a review from VincyZhang as a code owner May 22, 2024 07:26

VincyZhang and others added 2 commits May 22, 2024 01:25

Merge branch 'main' into gaudi-support

ad793d4

add pylint comment

5aedccb

Signed-off-by: Chen Xi <xi2.chen@intel.com>

VincyZhang and others added 9 commits May 22, 2024 05:32

Merge branch 'main' into gaudi-support

092ffb6

add comment to avoid pylint check

72f313f

Signed-off-by: Clark Chin <xi2.chen@intel.com>

ignore modeling_gaudi pylint

4642ee0

Signed-off-by: Clark Chin <xi2.chen@intel.com>

manual fix the pylint

0f714b5

Signed-off-by: Clark Chin <xi2.chen@intel.com>

Update pylint.sh

22557de

Signed-off-by: VincyZhang <wenxin.zhang@intel.com>

Merge branch 'main' into gaudi-support

b8b7de5

fix line by line pylint

17614be

Signed-off-by: Clark Chin <xi2.chen@intel.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

e02739f

for more information, see https://pre-commit.ci

disable before the line

2661a1e

Signed-off-by: Clark Chin <xi2.chen@intel.com>

airMeng approved these changes May 23, 2024

View reviewed changes

pylint check

cd82c90

Signed-off-by: Clark Chin <xi2.chen@intel.com>

VincyZhang merged commit 266e055 into main May 24, 2024
22 checks passed

VincyZhang deleted the gaudi-support branch May 24, 2024 02:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add gaudi modeling support in itrex #1438

add gaudi modeling support in itrex #1438

ClarkChin08 commented Mar 29, 2024

github-actions bot commented Mar 29, 2024 •

edited

airMeng left a comment

ClarkChin08 commented Apr 25, 2024

airMeng left a comment

add gaudi modeling support in itrex #1438

add gaudi modeling support in itrex #1438

Conversation

ClarkChin08 commented Mar 29, 2024

Type of Change

github-actions bot commented Mar 29, 2024 • edited

⚡ Required checks status: All passing 🟢

Groups summary

airMeng left a comment

Choose a reason for hiding this comment

ClarkChin08 commented Apr 25, 2024

airMeng left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 29, 2024 •

edited