Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CLI]: wandb.errors.CommError: Run initialization has timed out after 90.0 sec. “Is it wandb'server error?” #7671

Open
Echhoo opened this issue May 20, 2024 · 10 comments

Comments

@Echhoo
Copy link

Echhoo commented May 20, 2024

Describe the bug

wandb: Network error (ReadTimeout), entering retry loop.
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: ERROR Run initialization has timed out after 90.0 sec.

Additional Files

No response

Environment

WandB version:wandb, version 0.16.6

OS:python

Python version:3.8

Versions of relevant libraries:

Additional Context

No response

@umakrishnaswamy
Copy link
Contributor

hey @Echhoo - few questions to help me dig into this:

  • are you running wandb locally or utilizing https://wandb.ai/ ?
  • Could you go ahead and try to refresh your login credentials:
rm ~/.netrc
wandb login --relogin --host=<host_url>
  • do you have any firewalls set up? or are you trying to access wandb through a proxy? would love any details about this as well

@Echhoo
Copy link
Author

Echhoo commented May 21, 2024

hey @Echhoo - few questions to help me dig into this:

  • are you running wandb locally or utilizing https://wandb.ai/ ?
  • Could you go ahead and try to refresh your login credentials:
rm ~/.netrc
wandb login --relogin --host=<host_url>
  • do you have any firewalls set up? or are you trying to access wandb through a proxy? would love any details about this as well

I use wandb utilizing in https://wandb.ai/ and after I try this commend
'''
rm ~/.netrc
wandb login --relogin --host=<host_url>
'''
Wandb still cannot be used and reports the following error message:
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: / Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: ERROR Run initialization has timed out after 90.0 sec.

@umakrishnaswamy
Copy link
Contributor

hey @Echhoo - <host_url> corresponds to "https://api.wandb.ai"

Could you try the above steps with that url replaced? please let me know if you reach the hanging then

@umakrishnaswamy
Copy link
Contributor

Hi @Echhoo, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

Copy link

WandB Internal User commented:
umakrishnaswamy commented:
hey @Echhoo - few questions to help me dig into this:

  • are you running wandb locally or utilizing https://wandb.ai/ ?
  • Could you go ahead and try to refresh your login credentials:
rm ~/.netrc
wandb login --relogin --host=<host_url>
  • do you have any firewalls set up? or are you trying to access wandb through a proxy? would love any details about this as well

@Echhoo
Copy link
Author

Echhoo commented Jun 3, 2024

WandB Internal User commented: umakrishnaswamy commented: hey @Echhoo - few questions to help me dig into this:

  • are you running wandb locally or utilizing https://wandb.ai/ ?
  • Could you go ahead and try to refresh your login credentials:
rm ~/.netrc
wandb login --relogin --host=<host_url>
  • do you have any firewalls set up? or are you trying to access wandb through a proxy? would love any details about this as well

I still failed to connect,but when i use ping:
image
what should i do?

Copy link

Jason Davenport commented:
Hi,

We still have a few questions we need answered to help dig into this:

Are you running wandb locally or utilizing https://wandb.ai/ ?
Do you have any firewalls set up? or are you trying to access wandb through a proxy?

Could you go ahead and try to refresh your login credentials:

rm ~/.netrcwandb login --relogin --host=<host_url>

The answers to these will help us better assist you. Thanks!

Best,
Jason

@shc443
Copy link

shc443 commented Jun 7, 2024

Hello WandB team,

We have seen this issue since last year, but we simply ignored it because it somehow magically worked after few hours.
However, The frequency of this 90 sec error has been increased so so much compared to last year.
especially for today, all our simulation just consistently stopped because of this error.

Is this just WandB's server issue?

Copy link

Jason Davenport commented:
Hi,

We still have a few questions we need answered to help dig into this:

Are you running wandb locally or utilizing https://wandb.ai/ ?
Do you have any firewalls set up? or are you trying to access wandb through a proxy?

Could you go ahead and try to refresh your login credentials:

rm ~/.netrcwandb login --relogin --host=<host_url>

The answers to these will help us better assist you. Thanks!

Best,
Jason

Copy link

Jason Davenport commented:
Hi Internal,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Best,
Weights & Biases

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants