BUG: HyperDriveStep's allow_reuse param does not work.

[dataders]

I made a reproducible example in this branch of a fork of this repo. This is a big time suck for our team as the majority of our pipelines use HyperDriveStep and we always have downstream steps from HyperDrive.
https://github.com/swanderz/MachineLearningNotebooks/blob/hyperdrive_allow_reuse/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-parameter-tuning-with-hyperdrive.ipynb

[purnesh42H]

@swanderz I see that you are copying training_files to script_folder, at cell
"Copy the training files into the script folder" which would be changing the hash of the source_directory and preventing the reuse. Are you running that everytime?

[dataders]

@purnesh42H thanks for jumping on to help out! The copy script magic was just part of the minimum viable reproducible example I used. Here's the steps to reproduce:

1. Run the whole notebook through

pipeline = Pipeline(workspace=ws, steps=[best_run_step])
pipeline_run = exp.submit(pipeline)
pipeline_run.wait_for_completion()

2. Submit the pipeline again, with the above cells, changing nothing else.
3. HyperDriveStep will run again, ignoring allow_reuse.

I'm interested to see what you get on your side!

[sonnypark]

@swanderz for some of the services such as HD or AutoML, they possibly produce similar output with same config but not exactly same afaik.
I will confirm with HD team and let you know.

[sonnypark]

actually regardless of HD produce same output or not allow_reuse needs to skip HD runs so that user can control it. Will check why it didn't work.

[dataders]

@sonnypark I appreciate you also helping me get answers. Can you please run the reprex and see what you get? When I do it, the stdout says something along the lines of hd_step01 will run and generate new outputs and I wait ~12 minutes for the step to complete, which its a lot longer than I would expect if the step is being reused.

[sonnypark]

Yes "hd_step01 will run and generate new outputs" is what we get when the step is not reused. Will try to repro.

[dataders]

synced with @clmccart 95% chance this is related to #734.

[purnesh42H]

#734 is unrelated. In the case of hyperdrive, there is a change in hyperdrive config setting which is preventing the reuse

[nemasobhani]

Any update on this?

[sonnypark]

Fix has been checked in and will be available on the next release (which will be next week).