[BUG] No such file or directory

[Farhad-Heybati]

Description

tc_bert_azureml.ipynb
Error message:
wc: /mnt/batch/tasks/shared/LS_root/jobs/gaia-ml-wks/azureml/3576038c-f399-4522-8674-383ad7cd316b/mounts/workspaceblobstore/azureml/88f85fcf-eaf4-4d58-bff2-712b35cb50aa/train0,: No such file or directory

How do we replicate the bug?

Just run the notebook tc_bert_azureml

Expected behavior (i.e. solution)

Other Comments

[Farhad-Heybati]

I think the issue is coming from the fact that the output of preprocessing is defined by PipelineData:
output_data = PipelineData(name="train{}".format(str(i)), datastore=ds,
output_path_on_compute='mnli_data/processed_train/batch{}.csv'.format(str(i)))
By default, the path to the data file is based on an environment variable: $AZUREML_DATAREFERENCE
for example:
processed_train_files[0] => $AZUREML_DATAREFERENCE_train0
When I print the value of the file path in preprocessing step and in the training step, these are differents:
preprocess step:
/mnt/batch/tasks/shared/LS_root/jobs/gaia-ml-wks/azureml/273296fe-21e3-44fa-84d1-6a6b33c04031/mounts/workspaceblobstore/azureml/273296fe-21e3-44fa-84d1-6a6b33c04031/train0'
Bert training step:
/mnt/batch/tasks/shared/LS_root/jobs/gaia-ml-wks/azureml/51dd8b67-13a7-40a1-a9ff-eef158a300c5/mounts/workspaceblobstore/azureml/273296fe-21e3-44fa-84d1-6a6b33c04031/train0

The $AZUREML_DATAREFERENCE value change based on the runID