Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

缓存目录使用混乱,MODELSCOPE_CACHE环境变量在不同位置的作用不一致 #845

Closed
wangzhe258369 opened this issue Apr 26, 2024 · 1 comment · Fixed by #861
Assignees

Comments

@wangzhe258369
Copy link

wangzhe258369 commented Apr 26, 2024

Describe the feature

(已经pip install -U modelscope更新到最新版,目前版本号是:modelscope==1.14.0

modelscope项目里目前默认的缓存目录或者说主目录是:~/.cache/modelscope,但是当通过设置环境变量,在这个非默认目录下加载或者下载模型时,就会遇到和默认目录不一致的问题。

比如希望把目标目录改成:/data0/modelscope,希望通过pipeline加载的模型damo/nlp_raner_named-entity-recognition_chinese-base-ecom,以及当前在任何目录下都没有这份模型文件。

在代码中加载模型的语句如下:

import modelscope
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

model_id = "damo/nlp_raner_named-entity-recognition_chinese-base-ecom"
pipe = pipeline(Tasks.named_entity_recognition, model_id)
  • 如果不设置任何变量(默认行为),会创建索引文件~/.cache/modelscope/ast_indexer,下载时的临时目录~/.cache/modelscope/hub/temp,最终模型文件被下载到~/.cache/modelscope/hub/damo/nlp_raner_named-entity-recognition_chinese-base-ecom
  • 设置环境变量MS_CACHE_HOME=/data0/modelscope,这个环境变量完全不起作用,仍然在~/.cache/modelscope目录下载文件
  • 设置环境变量MODELSCOPE_CACHE=/data0/modelscope,则import modelscope时会创建索引文件/data0/modelscope/ast_indexer,同时下载模型文件的临时目录会使用/data0/modelscope/temp,最终模型文件被下载到/data0/modelscope/damo/nlp_raner_named-entity-recognition_chinese-base-ecom。这时/data0/modelscope/hub目录永远不会被使用。
  • 设置环境变量MODELSCOPE_CACHE=~/.cache/modelscope,则import modelscope时会创建索引文件~/.cache/modelscope/ast_indexer,同时下载模型文件的临时目录会使用~/.cache/modelscope/temp,最终模型文件被下载到~/.cache/modelscope/damo/nlp_raner_named-entity-recognition_chinese-base-ecom。这时~/.cache/modelscope/hub目录永远不会被使用。

Motivation
统一环境变量MS_CACHE_HOMEMODELSCOPE_CACHE的作用。当把缓存目录设置到非默认位置时,规范下载行为在hub子目录进行。

@yingdachen
Copy link
Contributor

感谢整理,我们看一下

liuyhwangyh pushed a commit that referenced this issue May 21, 2024
liuyhwangyh added a commit that referenced this issue May 23, 2024
* fix #845

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
liuyhwangyh pushed a commit that referenced this issue May 23, 2024
Supports resumption of downloads from breakpoints, optimized download progress bar, finer display granularity, better experience under low bandwidth, and added function of downloading specified directories.
wenmengzhou pushed a commit that referenced this issue May 24, 2024
* fix #845

Supports resumption of downloads from breakpoints, optimized download progress bar, finer display granularity, better experience under low bandwidth, and added function of downloading specified directories.

* restore push to hub

* fix merge issue

* fix ut issue

---------

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants