Integrate web UI with chat template #205
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: minmingzhu <minming.zhu@intel.com>
…hu/llm-on-ray into inference_chat_template
Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
2. modify chat template Signed-off-by: minmingzhu <minming.zhu@intel.com>
2. add unit test Signed-off-by: minmingzhu <minming.zhu@intel.com>
* update * fix blocking * update Signed-off-by: Wu, Xiaochang <xiaochang.wu@intel.com> * update Signed-off-by: Wu, Xiaochang <xiaochang.wu@intel.com> * fix setup and getting started Signed-off-by: Wu, Xiaochang <xiaochang.wu@intel.com> * update Signed-off-by: Wu, Xiaochang <xiaochang.wu@intel.com> * update Signed-off-by: Wu, Xiaochang <xiaochang.wu@intel.com> * nit Signed-off-by: Wu, Xiaochang <xiaochang.wu@intel.com> * Add dependencies for tests and update pyproject.toml Signed-off-by: Wu, Xiaochang <xiaochang.wu@intel.com> * Update dependencies and test workflow Signed-off-by: Wu, Xiaochang <xiaochang.wu@intel.com> * Update dependencies and fix torch_dist.py Signed-off-by: Wu, Xiaochang <xiaochang.wu@intel.com> * Update OpenAI SDK installation and start ray cluster Signed-off-by: Wu, Xiaochang <xiaochang.wu@intel.com> --------- Signed-off-by: Wu, Xiaochang <xiaochang.wu@intel.com>
* single test * single test * single test * single test * fix hang error
* use base model mpt-7b instead of mpt-7b-chat Signed-off-by: minmingzhu <minming.zhu@intel.com> * manual setting specify tokenizer Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * update doc/finetune_parameters.md Signed-off-by: minmingzhu <minming.zhu@intel.com> --------- Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
minmingzhu
force-pushed
the
Integrate_web_ui
branch
from
8698a17
to
a8e7b38
Compare
xwu99
reviewed
May 10, 2024
| @@ -6,16 +6,11 @@ cpus_per_worker: 24 | |||
| gpus_per_worker: 0 | |||
| deepspeed: false | |||
| workers_per_group: 2 | |||
| device: cpu | |||
| device: "cpu" | |||
There was a problem hiding this comment.
There is no need to add extra " to yaml. Is it needed to touch this part for your PR?
xwu99
reviewed
May 10, 2024
| @@ -6,17 +6,12 @@ cpus_per_worker: 24 | |||
| gpus_per_worker: 0 | |||
| deepspeed: false | |||
| workers_per_group: 2 | |||
| device: cpu | |||
| device: CPU | |||
There was a problem hiding this comment.
pay attention to use lowercase device for consistency
xwu99
reviewed
May 10, 2024
| @@ -6,16 +6,10 @@ cpus_per_worker: 24 | |||
| gpus_per_worker: 0 | |||
| deepspeed: false | |||
| workers_per_group: 2 | |||
| device: cpu | |||
| device: CPU | |||
There was a problem hiding this comment.
why change the device name to capital case?
xwu99
reviewed
May 10, 2024
| @@ -15,6 +15,7 @@ The following are the parameters supported in the finetuning workflow. | |||
| |lora_config|task_type: CAUSAL_LM<br>r: 8<br>lora_alpha: 32<br>lora_dropout: 0.1|Will be passed to the LoraConfig `__init__()` method, then it'll be used as config to build Peft model object.| | |||
| |deltatuner_config|"algo": "lora"<br>"denas": True<br>"best_model_structure": "/path/to/best_structure_of_deltatuner_model"|Will be passed to the DeltaTunerArguments `__init__()` method, then it'll be used as config to build [Deltatuner model](https://github.com/intel/e2eAIOK/tree/main/e2eAIOK/deltatuner) object.| | |||
| |enable_gradient_checkpointing|False|enable gradient checkpointing to save GPU memory, but will cost more compute runtime| | |||
| |chat_template|None|User-defined chat template.| | |||
There was a problem hiding this comment.
Add description and link to the doc of huggingface otherwise user will not know what it is.
xwu99
reviewed
May 10, 2024
| prompt = "Once upon a time," | ||
| # prompt = "Once upon a time," | ||
| prompt = [ | ||
| {"role": "user", "content": "Which is bigger, the moon or the sun?"}, |
There was a problem hiding this comment.
don't modify this as api_server_simple/query_single.py is for simple protocol. it's not formatted like this. focus on openapi support, don't need to support chat temple for simple protocol if need to change query format.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.