Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DefaultStrategy only receives successful training replies #1086

Open
srcansiz opened this issue Apr 9, 2024 · 0 comments
Open

DefaultStrategy only receives successful training replies #1086

srcansiz opened this issue Apr 9, 2024 · 0 comments
Labels
bug this issue is about reporting and resolving a suspected bug candidate an individual developer submits a work request to the team (extension proposal, bug, other request)

Comments

@srcansiz
Copy link
Member

srcansiz commented Apr 9, 2024

The default behavior of the default strategy is that if any nodes returns an unsuccessful reply the training round should fail. However, DefaultStrategy.refine function in Experiment.run_once only gets training replies that are success. This is due to Job._get_training_result that does not add success = False replies into training_replies dictionary. This situation results as successful training round even when one or more than one node returns success=False if at least a node return successful training reply.

Simple solution can be to add unsuccessful reply to the training replies and let strategy class do the rest. However, this solution also requires to modify function _extract_received_optimizer_aux_var_from_round to only allow extraction for successful replies.

@srcansiz srcansiz added bug this issue is about reporting and resolving a suspected bug candidate an individual developer submits a work request to the team (extension proposal, bug, other request) labels Apr 9, 2024
@srcansiz srcansiz changed the title DefaultStrategy only receives succesfull training replies DefaultStrategy only receives successful training replies Apr 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug this issue is about reporting and resolving a suspected bug candidate an individual developer submits a work request to the team (extension proposal, bug, other request)
Projects
None yet
Development

No branches or pull requests

1 participant