Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors related to uploading artifacts while tracking runs are silent #1522

Open
polyaxon-team opened this issue Dec 12, 2022 · 0 comments
Open

Comments

@polyaxon-team
Copy link
Contributor

Current behavior

From slack:

Another question about logging metrics to a run through a local jupyter notebook. After our conversation on Nov. 30th. ☝️, things were working fine. However recently the dashboard has stopped displaying metrics again, and I'm seeing weird behaviour in polyaxon. I don't know what has changed.
Looking for advice since I'm out of troubleshooting ideas. Details in the thread...
Here's code I have that recreates the problem:

from polyaxon import tracking

tracking.init(
    owner="owner",
    project="project-name",
    name="test_run",
    run_uuid=None,
    is_new=True
)
tracking.set_run_event_logger()

tracking.log_text(name="some_text_metric", text="some text")

for step in range(1, 100):
    tracking.log_metric(name="some_step_metric", value=step/2, step=step)

tracking.log_succeeded()
tracking.end()

After using some debugging using:

from polyaxon.logger import configure_logger

configure_logger(verbose=True)
...

It turns out that :

Thank you for that command there. After looking at the logging from that, I realized that my polyaxon cli host had switched from the url of the gateway deployed on our cluster to https://cloud.polyaxon.com/.
After some tests, it looks like logging metrics through cloud.polyaxon.com causes the issues I was seeing with artifacts. When I switched the polyaxon host to the url of our gateway, then the dashboard started correctly displaying metrics.

Enhancement

As suggested by the user, the upload is happening in a thread, API errors (404/401/403) should show to help the user debug issues:

Any chance that you'd update the code to provide a useful error message when someone tries this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant