Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

h2o for kv cache compression #1468

Open
wants to merge 50 commits into
base: main
Choose a base branch
from
Open

h2o for kv cache compression #1468

wants to merge 50 commits into from

Conversation

n1ck-guo
Copy link
Collaborator

@n1ck-guo n1ck-guo commented Apr 10, 2024 •
edited

Type of Change

feature

Description

H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
paper

NTD

  • example
  • refactor code to same style
  • add seq len api
  • support for more models, both sim and real
  • mean accumulate score function

Expected Behavior & Potential Risk

None

How has this PR been tested?

how to reproduce the test (including hardware information)

Dependency Change?

any library dependency introduced or removed

@n1ck-guo
h2o for kv cache compression
41d8647 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 10, 2024 •
edited

⛈️ Required checks status: Has failure 🔴

Warning
If you do not have the access to re-run the CI-Summary bot, please contact VincyZhang for help. If you push a new commit, all of the workflow will be re-triggered.

Groups summary

🔴 Format Scan Tests workflow
Check ID Status Error details
format-scan (pylint) failure download ❌
format-scan (bandit) success ✅
format-scan (cloc) success ✅
format-scan (cpplint) success ✅

These checks are required after the changes to intel_extension_for_transformers/transformers/modeling/kv_cache_compression/__init__.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/h2o.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/__init__.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_bloom.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_gpt_neox.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_llama.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_mistral.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_mixtral.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_opt.py.

🟢 Optimize Unit Test workflow
Check ID Status Error details
optimize-unit-test-baseline success ✅
optimize-unit-test-PR-test success ✅
Genreate-OptimizeUT-Report success ✅

These checks are required after the changes to intel_extension_for_transformers/transformers/modeling/kv_cache_compression/__init__.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/h2o.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/__init__.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_bloom.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_gpt_neox.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_llama.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_mistral.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_mixtral.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_opt.py.

🟢 NeuralChat Unit Test
Check ID Status Error details
neuralchat-unit-test-baseline success ✅
neuralchat-unit-test-PR-test success ✅
Generate-NeuralChat-Report success ✅

These checks are required after the changes to intel_extension_for_transformers/transformers/modeling/kv_cache_compression/__init__.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/h2o.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/__init__.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_bloom.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_gpt_neox.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_llama.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_mistral.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_mixtral.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_opt.py.

🔴 Engine Unit Test workflow
Check ID Status Error details
engine-unit-test-baseline failure download ❌
engine-unit-test-PR-test cancelled 🚫
Genreate-Engine-Report skipped ❓

These checks are required after the changes to intel_extension_for_transformers/transformers/modeling/kv_cache_compression/__init__.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/h2o.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/__init__.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_bloom.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_gpt_neox.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_llama.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_mistral.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_mixtral.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_opt.py.

🟢 Chat Bot Test workflow
Check ID Status Error details
call-inference-llama-2-7b-chat-hf / inference test success ✅
call-inference-mpt-7b-chat / inference test success ✅

These checks are required after the changes to intel_extension_for_transformers/transformers/modeling/kv_cache_compression/__init__.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/h2o.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/__init__.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_bloom.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_gpt_neox.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_llama.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_mistral.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_mixtral.py, intel_extension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_opt.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and will be updates every 180 seconds within the next 6 hours. If you have any other questions, contact VincyZhang or XuehaoSun for help.

BiaoFangAIA and others added 3 commits April 23, 2024 16:36
@BiaoFangAIA
rebuild
c46ea7d 
Signed-off-by: biao.fang <biao.fang@intel.com>
@BiaoFangAIA
merge
95ff9ae 
Signed-off-by: biao.fang <biao.fang@intel.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
9d27733 
for more information, see https://pre-commit.ci
n1ck-guo and others added 4 commits April 25, 2024 03:01
@n1ck-guo
update
444490d 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@n1ck-guo
update
4309089 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@n1ck-guo
merge
8c5272e 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
1b83e52 
for more information, see https://pre-commit.ci
n1ck-guo and others added 3 commits May 7, 2024 04:26
@VincyZhang VincyZhang added the WIP label May 13, 2024
n1ck-guo and others added 7 commits May 14, 2024 01:43
@n1ck-guo
Merge branch 'main' into hengguo/h2o
91d4394
@n1ck-guo
real drop
a83e6d6 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@n1ck-guo
modify real drop code
92c8a62 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
70a1cf3 
for more information, see https://pre-commit.ci
@BiaoFangAIA
fix
bc9eade 
Signed-off-by: biao.fang <biao.fang@intel.com>
@n1ck-guo
update for real drop and sim mode, using the same api
9aa25f6 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@n1ck-guo
Merge branch 'main' into hengguo/h2o
03cdc8d
n1ck-guo and others added 7 commits May 16, 2024 21:14
@n1ck-guo
change to new api
877329d 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@n1ck-guo
Merge branch 'hengguo/h2o' of https://github.com/intel/intel-extensio…
b435e3e 
…n-for-transformers into hengguo/h2o
@n1ck-guo
clean code
5068552 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
5e5f589 
for more information, see https://pre-commit.ci
@n1ck-guo
fix
d0dce7d 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@n1ck-guo
Merge branch 'hengguo/h2o' of https://github.com/intel/intel-extensio…
febb76a 
…n-for-transformers into hengguo/h2o
@n1ck-guo
add example
24c4725 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
n1ck-guo and others added 15 commits May 20, 2024 14:36
@n1ck-guo
Merge branch 'main' into hengguo/h2o
5bd3f16
@n1ck-guo
clean
955e132 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@n1ck-guo
pylint
91efe57 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
4190edb 
for more information, see https://pre-commit.ci
@n1ck-guo
pylint
e71bf92 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@n1ck-guo
pylint
41f016c 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
d49487f 
for more information, see https://pre-commit.ci
@n1ck-guo
fix import error
9ac5eca 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@n1ck-guo
update
09def0b 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@n1ck-guo
Merge branch 'hengguo/h2o' of https://github.com/intel/intel-extensio…
5cae1fd 
…n-for-transformers into hengguo/h2o
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
3042dd4 
for more information, see https://pre-commit.ci
@n1ck-guo
pylint
3a992ab 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@n1ck-guo
pylint
8c89cbc 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@n1ck-guo
merge
072ad76 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
4c26487 
for more information, see https://pre-commit.ci
@n1ck-guo
Copy link
Collaborator Author

pre-commit.ci autofix

pre-commit-ci bot and others added 5 commits May 21, 2024 07:09
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
741c7cd 
for more information, see https://pre-commit.ci
@n1ck-guo
add example readme
0800df2 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@n1ck-guo
update
558dfd9 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@n1ck-guo
merge
d6de2b3 
Signed-off-by: n1ck-guo <heng.guo@intel.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
693983f 
for more information, see https://pre-commit.ci
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PenghuiCheng PenghuiCheng Awaiting requested review from PenghuiCheng PenghuiCheng is a code owner

@VincyZhang VincyZhang Awaiting requested review from VincyZhang

@wenhuach21 wenhuach21 Awaiting requested review from wenhuach21

@yintong-lu yintong-lu Awaiting requested review from yintong-lu

@WeiweiZhang1 WeiweiZhang1 Awaiting requested review from WeiweiZhang1

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
WIP
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@n1ck-guo @wenhuach21 @VincyZhang @BiaoFangAIA