Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update readme #3

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
68 changes: 62 additions & 6 deletions README-EN.md
@@ -1,28 +1,84 @@
# hf-mirror-fetch

<div align="center">
<a href="https://pypi.python.org/pypi/hf-mirror-fetch">
<img src="https://img.shields.io/pypi/v/hf-mirror-fetch.svg" alt="PyPI version" />
</a>
<a href="https://github.com/cubenlp/hf-mirror-download/actions/workflows/test.yml">
<img src="https://github.com/cubenlp/hf-mirror-download/actions/workflows/test.yml/badge.svg" alt="Tests" />
</a>
<a href="https://cubenlp.github.io/hf-mirror-download/">
<img src="https://img.shields.io/badge/docs-github_pages-blue.svg" alt="Documentation Status" />
</a>
<a href="https://codecov.io/gh/cubenlp/hf-mirror-download">
<img src="https://codecov.io/gh/cubenlp/hf-mirror-download/branch/main/graph/badge.svg" alt="Coverage" />
</a>
</div>

[English](README-EN.md) | [简体中文](README.md)

`hf-mirror-fetch` 是一个命令行工具,使用镜像站 hf-mirror.com 下载 Hugging Face 模型。

## 安装

通过以下命令安装 `hf-mirror-fetch`:

```bash
pip install hf-mirror-fetch
```

## 使用方法

只需要提供模型页面的 URL 即可,示例:

```bash
hfmf https://huggingface.co/openchat/openchat_3.5/tree/main
hfmf https://huggingface.co/openchat/openchat_3.5
hfmf openchat/openchat_3.5
hfmf openchat/openchat_3.5 --tgt_folder /sshfs/pretrains/
```

参数说明:

- `-f`, `--tgt_folder`: (可选)保存下载文件的目标文件夹。如果未指定,将使用模型名创建一个文件夹。
- `--update`: (可选)更新现有文件,但不包括权重文件。

## 贡献

欢迎任何形式的贡献,无论是功能建议、代码改进还是文档更新。请通过 GitHub 的 Issue 或 Pull Request 提交您的贡献。

## 版权

该项目采用 MIT 许可证。有关更多信息,请查看 LICENSE 文件。


# hf-mirror-fetch

[English](README-EN.md) | [简体中文](README.md)

`hf-mirror-fetch` is a command-line tool designed to streamline the process of downloading machine learning models and related files from the Hugging Face model hub mirror site. This tool is particularly useful in environments where access to the primary Hugging Face repository is restricted or where connectivity to the main site is slow.
`hf-mirror-fetch` is a command-line tool to download Hugging Face models using the mirror site hf-mirror.com.

## Installation

Ensure you have Python 3.6 or higher installed. You can install `hf-mirror-fetch` using the following command:
You can install `hf-mirror-fetch` by:

```bash
pip install hf-mirror-fetch
```

## Usage

Using `hf-mirror-fetch` to download models is straightforward; you only need to provide the URL of the model page. Here is an example of how to use it:
Here are some examples of how to use `hf-mirror-fetch`:

```bash
hfmf -u https://huggingface.co/openchat/openchat_3.5/tree/main
hfmf https://huggingface.co/openchat/openchat_3.5/tree/main
hfmf https://huggingface.co/openchat/openchat_3.5
hfmf openchat/openchat_3.5
hfmf openchat/openchat_3.5 --tgt_folder /sshfs/pretrains/
```

### Parameters
Parameters:

- `-u`, `--url`: The URL of the model's page on hf-mirror.com or huggingface.co.
- `-f`, `--tgt_folder`: (Optional) The target folder to save the downloaded files. If not specified, a folder named after the model will be created.
- `--update`: (Optional) Update existing files except for weights.

Expand Down
31 changes: 23 additions & 8 deletions README.md
@@ -1,30 +1,45 @@
# hf-mirror-fetch

<div align="center">
<a href="https://pypi.python.org/pypi/hf-mirror-fetch">
<img src="https://img.shields.io/pypi/v/hf-mirror-fetch.svg" alt="PyPI version" />
</a>
<a href="https://github.com/cubenlp/hf-mirror-download/actions/workflows/test.yml">
<img src="https://github.com/cubenlp/hf-mirror-download/actions/workflows/test.yml/badge.svg" alt="Tests" />
</a>
<a href="https://cubenlp.github.io/hf-mirror-download/">
<img src="https://img.shields.io/badge/docs-github_pages-blue.svg" alt="Documentation Status" />
</a>
<a href="https://codecov.io/gh/cubenlp/hf-mirror-download">
<img src="https://codecov.io/gh/cubenlp/hf-mirror-download/branch/main/graph/badge.svg" alt="Coverage" />
</a>
</div>

[English](README-EN.md) | [简体中文](README.md)

`hf-mirror-fetch` 是一个命令行工具,旨在简化从 Hugging Face 模型镜像站点下载机器学习模型和相关文件的过程。这个工具特别适用于在访问 Hugging Face 主站受限或连接速度较慢的环境中
`hf-mirror-fetch` 是一个命令行工具,使用镜像站 hf-mirror.com 下载 Hugging Face 模型

## 安装

确保您已安装 Python 3.6 或更高版本。您可以通过以下命令安装 `hf-mirror-fetch`:
通过以下命令安装 `hf-mirror-fetch`:

```bash
pip install hf-mirror-fetch
```

## 使用方法

使用 `hf-mirror-fetch` 下载模型非常简单,只需要提供模型页面的 URL 即可。以下是使用示例
只需要提供模型页面的 URL 即可,示例

```bash
hfmf -u https://huggingface.co/openchat/openchat_3.5/tree/main
hfmf -u https://huggingface.co/openchat/openchat_3.5
hfmf -u openchat/openchat_3.5
hfmf https://huggingface.co/openchat/openchat_3.5/tree/main
hfmf https://huggingface.co/openchat/openchat_3.5
hfmf openchat/openchat_3.5
hfmf openchat/openchat_3.5 --tgt_folder /sshfs/pretrains/
```

### 参数说明
参数说明

- `-u`, `--url`: 模型页面在 hf-mirror.com 或 huggingface.co 上的 URL。
- `-f`, `--tgt_folder`: (可选)保存下载文件的目标文件夹。如果未指定,将使用模型名创建一个文件夹。
- `--update`: (可选)更新现有文件,但不包括权重文件。

Expand Down
8 changes: 5 additions & 3 deletions hf_mirror_fetch/cli.py
Expand Up @@ -8,6 +8,7 @@ def process_url(url):
HF = "https://huggingface.co"
# If URL lacks a known prefix, assume it's intended for Hugging Face
if not url.startswith((ROOT, HF)):
assert not url.startswith("http"), "Invalid URL. Please provide a valid URL from hf-mirror.com or huggingface.co."
url = f"{HF}/{url}" # Prepend with Hugging Face domain
if url.startswith(HF):
url = url.replace(HF, ROOT)
Expand All @@ -19,11 +20,12 @@ def process_url(url):


@click.command()
@click.option('-u', '--url', required=True, help='The URL of the model\'s page on hf-mirror.com or huggingface.co.')
@click.argument('link', help='The URL of the model\'s page on hf-mirror.com or huggingface.co.')
@click.option('-f', '--tgt_folder', default=None, type=str, help='The target folder to save the downloaded files.')
@click.option('--update', is_flag=True, help='Update existing files except for weights.')
def main(url, tgt_folder, update):
def main(link, tgt_folder, update):
"""Console script for hf_mirror_fetch."""
url = process_url(url)
assert link, "Please provide a URL."
url = process_url(link)
download_from_mirror_page(url, tgt_folder, update)
return 0