You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am trying to run training and I get this error when staring up:
HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/glue/paths-info/bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c
[2024-04-18 15:55:06] CRITICAL [olmo.util:158, rank=6] Uncaught HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/glue/paths-info/bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c
I am running on 2 nodes each with 8 GPUs, using the main branch and pytorch 2.2.2+cu121.
This works with just 1 node using 8 GPUs.
The text was updated successfully, but these errors were encountered:
We run into issues like that too. We don't have a robust solution yet, but one trick we do is caching the datasets locally (or once per node or however many file systems you have) as follows and then making HF not call the hub by setting the environment variable HF_DATASETS_OFFLINE=1.
from olmo.eval.downstream import *
tokenizer = Tokenizer.from_file("tokenizers/allenai_gpt-neox-olmo-dolma-v1_5.json")
for x in label_to_task_map.values():
kwargs = {}
if isinstance(x, tuple):
x, kwargs = x
x(tokenizer=tokenizer, **kwargs)
❓ The question
I am trying to run training and I get this error when staring up:
I am running on 2 nodes each with 8 GPUs, using the main branch and pytorch 2.2.2+cu121.
This works with just 1 node using 8 GPUs.
The text was updated successfully, but these errors were encountered: