Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: make logging more informative during training #310

Merged
merged 2 commits into from Apr 13, 2021
Merged

fix: make logging more informative during training #310

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3a3cd9e
fix: save previous logged time only when logging
sasha-gitg Apr 13, 2021
16692e6
fix: make run completion more informative
sasha-gitg Apr 13, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions google/cloud/aiplatform/training_jobs.py
View file
Open in desktop
Expand Up @@ -646,12 +646,12 @@ def _block_until_complete(self):
)
)
log_wait = min(log_wait * multiplier, max_wait)
previous_time = current_time
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

previous_time = current_time
time.sleep(wait)

self._raise_failure()

_LOGGER.log_action_completed_against_resource("", "run", self)
_LOGGER.log_action_completed_against_resource("run", "completed", self)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the docstring of log_action_completed_against_resource, which is:

        Args:
            noun (str): Noun the action acts on against the resource. Can be empty string.
            action (str): Action started against resource. ie: "Deployed". Can be empty string.
            resource_noun_obj (AiPlatformResourceNoun):
                Resource noun object the action is acting against

I can't tell if these args are correct or not.

What does Noun the action acts on against the resource. mean?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The noun-action combo is for nouns in the following:

  • run completed
  • data imported
  • model deployed

Sometimes these nouns are separate from the resource they act on. Like data/dataset and model/endpoint. Happy to improve the wording in the comment.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay I see!

I was having trouble parsing the sentence. I get it now.

Perhaps could be changed to:
The noun that the action acts on against the resource.,
The action that was started against the resource., etc.

The examples would actually be the most helpful.


if self._gca_resource.model_to_upload and not self.has_failed:
_LOGGER.info(
Expand Down