Skip to content
This repository has been archived by the owner on Mar 21, 2024. It is now read-only.

New directory is not created while using flag --no-train #844

Open
1 task done
Adapter668 opened this issue May 22, 2023 · 4 comments
Open
1 task done

New directory is not created while using flag --no-train #844

Adapter668 opened this issue May 22, 2023 · 4 comments
Labels
bug Something isn't working good first issue Good for newcomers

Comments

@Adapter668
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Bug summary

While using flag --no-train there is an error that path does not exist. I tried also using local_weights_path, but still the same error.

Code for reproduction

CUDA_VISIBLE_DEVICES=5 python InnerEye/ML/runner.py --model=DynamicUnetBones --no-train --local_weights_path=outputs/2023-05-19T071816Z_DynamicUnetBones/checkpoints/last.ckpt

Actual outcome

Traceback (most recent call last):
File "InnerEye/ML/runner.py", line 467, in
main()
File "InnerEye/ML/runner.py", line 461, in main
run(project_root=fixed_paths.repository_root_directory(),
File "InnerEye/ML/runner.py", line 457, in run
return runner.run()
File "InnerEye/ML/runner.py", line 220, in run
self.run_in_situ(azure_run_info)
File "InnerEye/ML/runner.py", line 412, in run_in_situ
self.ml_runner.run()
File "/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/InnerEye/ML/run_ml.py", line 378, in run
self.run_inference_for_lightning_models(checkpoint_paths_for_testing)
File "/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/InnerEye/ML/run_ml.py", line 469, in run_inference_for_lightning_models
with change_working_directory(self.container.outputs_folder):
File "/home/.conda/envs/InnerEye3/lib/python3.8/contextlib.py", line 113, in enter
return next(self.gen)
File "/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/InnerEye/Common/common_util.py", line 436, in change_working_directory
os.chdir(new_path)
FileNotFoundError: [Errno 2] No such file or directory: '/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/outputs/2023-05-22T073814Z_DynamicUnetBones'

Error messages

FileNotFoundError: [Errno 2] No such file or directory: '/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/outputs/2023-05-22T073814Z_DynamicUnetBones'

Expected outcome

It should run predictions on test data and save the result either on the local_weights_path directory or the new path (doesn't matter for me)

System info

OS:
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

INNEREYE COMMIT HASH:
commit 2877002 (HEAD -> main, origin/main, origin/HEAD)

CONDA ENV:
name: InnerEye3
channels:

  • pytorch
  • defaults
    dependencies:
  • _libgcc_mutex=0.1=main
  • _openmp_mutex=5.1=1_gnu
  • blas=1.0=mkl
  • blosc=1.21.0=h4ff587b_1
  • bzip2=1.0.8=h7b6447c_0
  • ca-certificates=2022.07.19=h06a4308_0
  • certifi=2022.9.24=py38h06a4308_0
  • cudatoolkit=11.3.1=h2bc3f7f_2
  • ffmpeg=4.2.2=h20bf706_0
  • freetype=2.11.0=h70c0345_0
  • giflib=5.2.1=h7b6447c_0
  • gmp=6.2.1=h295c915_3
  • gnutls=3.6.15=he1e5248_0
  • intel-openmp=2021.4.0=h06a4308_3561
  • jpeg=9e=h7f8727e_0
  • lame=3.100=h7b6447c_0
  • lcms2=2.12=h3be6417_0
  • ld_impl_linux-64=2.38=h1181459_1
  • lerc=3.0=h295c915_0
  • libdeflate=1.8=h7f8727e_5
  • libffi=3.3=he6710b0_2
  • libgcc-ng=11.2.0=h1234567_1
  • libgomp=11.2.0=h1234567_1
  • libidn2=2.3.2=h7f8727e_0
  • libopus=1.3.1=h7b6447c_0
  • libpng=1.6.37=hbc83047_0
  • libstdcxx-ng=11.2.0=h1234567_1
  • libtasn1=4.16.0=h27cfd23_0
  • libtiff=4.4.0=hecacb30_0
  • libunistring=0.9.10=h27cfd23_0
  • libuv=1.40.0=h7b6447c_0
  • libvpx=1.7.0=h439df22_0
  • libwebp=1.2.4=h11a3e52_0
  • libwebp-base=1.2.4=h5eee18b_0
  • lz4-c=1.9.3=h295c915_1
  • mkl=2021.4.0=h06a4308_640
  • mkl-service=2.4.0=py38h7f8727e_0
  • mkl_fft=1.3.1=py38hd3c417c_0
  • mkl_random=1.2.2=py38h51133e4_0
  • ncurses=6.3=h5eee18b_3
  • nettle=3.7.3=hbbd107a_1
  • openh264=2.1.1=h4ff587b_0
  • openssl=1.1.1q=h7f8727e_0
  • pip=20.1.1=py38_1
  • python=3.8.3=hcff3b4d_2
  • python-blosc=1.7.0=py38h7b6447c_0
  • pytorch=1.10.0=py3.8_cuda11.3_cudnn8.2.0_0
  • pytorch-mutex=1.0=cuda
  • readline=8.1.2=h7f8727e_1
  • setuptools=63.4.1=py38h06a4308_0
  • sqlite=3.39.3=h5082296_0
  • tk=8.6.12=h1ccaba5_0
  • torchvision=0.11.1=py38_cu113
  • typing_extensions=4.3.0=py38h06a4308_0
  • wheel=0.37.1=pyhd3eb1b0_0
  • x264=1!157.20191217=h7b6447c_0
  • xz=5.2.6=h5eee18b_0
  • zlib=1.2.12=h5eee18b_3
  • zstd=1.5.2=ha4553b6_0
  • pip:
    • absl-py==1.2.0
    • adal==1.2.7
    • aiofiles==23.1.0
    • aiohttp==3.8.3
    • aiosignal==1.2.0
    • alabaster==0.7.12
    • alembic==1.8.1
    • ansiwrap==0.8.4
    • applicationinsights==0.11.10
    • argon2-cffi==21.3.0
    • argon2-cffi-bindings==21.2.0
    • argparse==1.4.0
    • async-timeout==4.0.2
    • attrs==22.1.0
    • azure-common==1.1.28
    • azure-core==1.26.0
    • azure-graphrbac==0.61.1
    • azure-identity==1.7.0
    • azure-mgmt-authorization==0.61.0
    • azure-mgmt-containerregistry==10.0.0
    • azure-mgmt-core==1.3.2
    • azure-mgmt-datafactory==1.1.0
    • azure-mgmt-keyvault==9.3.0
    • azure-mgmt-resource==12.1.0
    • azure-mgmt-storage==11.2.0
    • azure-storage-blob==12.6.0
    • azureml-automl-core==1.36.1
    • azureml-core==1.36.0.post2
    • azureml-dataprep==2.24.4
    • azureml-dataprep-native==38.0.0
    • azureml-dataprep-rslex==2.0.3
    • azureml-dataset-runtime==1.36.0
    • azureml-mlflow==1.36.0
    • azureml-pipeline==1.36.0
    • azureml-pipeline-core==1.36.0
    • azureml-pipeline-steps==1.36.0
    • azureml-sdk==1.36.0
    • azureml-telemetry==1.36.0
    • azureml-tensorboard==1.36.0
    • azureml-train-automl-client==1.36.0
    • azureml-train-core==1.36.0
    • azureml-train-restclients-hyperdrive==1.36.0
    • babel==2.10.3
    • backcall==0.2.0
    • backports-tempfile==1.0
    • backports-weakref==1.0.post1
    • batchgenerators==0.25
    • beautifulsoup4==4.11.1
    • black==22.10.0
    • bleach==5.0.1
    • cachetools==4.2.4
    • cffi==1.15.1
    • charset-normalizer==2.1.1
    • click==8.1.3
    • cloudpickle==1.6.0
    • colorama==0.4.5
    • commonmark==0.9.1
    • conda-merge==0.1.5
    • contextlib2==21.6.0
    • coverage==6.5.0
    • cryptography==3.3.2
    • cycler==0.11.0
    • databricks-cli==0.17.3
    • dataclasses-json==0.5.2
    • debugpy==1.6.3
    • decorator==5.1.1
    • defusedxml==0.7.1
    • deprecated==1.2.13
    • dicom2nifti==2.4.8
    • dill==0.3.6
    • distro==1.8.0
    • docker==4.3.1
    • docutils==0.18.1
    • dotnetcore2==2.1.23
    • entrypoints==0.4
    • execnet==1.9.0
    • fastjsonschema==2.16.2
    • fastmri==0.2.0
    • flake8==3.8.3
    • flask==2.2.2
    • frozenlist==1.3.1
    • fsspec==2022.8.2
    • furo==2022.6.21
    • fusepy==3.0.1
    • future==0.18.3
    • gitdb==4.0.9
    • gitpython==3.1.7
    • google-auth==1.35.0
    • google-auth-oauthlib==0.4.6
    • gputil==1.4.0
    • greenlet==1.1.3.post0
    • grpcio==1.49.1
    • gunicorn==20.1.0
    • h5py==2.10.0
    • hi-ml==0.2.5
    • hi-ml-azure==0.2.5
    • humanize==4.4.0
    • idna==3.4
    • imageio==2.15.0
    • imagesize==1.4.1
    • importlib-metadata==5.0.0
    • importlib-resources==5.10.0
    • iniconfig==1.1.1
    • innereye-dicom-rt==1.1.0
    • ipykernel==6.16.0
    • ipython==7.31.1
    • ipython-genutils==0.2.0
    • ipywidgets==8.0.2
    • isodate==0.6.1
    • itsdangerous==2.1.2
    • jedi==0.18.2
    • jeepney==0.8.0
    • jinja2==3.0.2
    • jmespath==0.10.0
    • joblib==0.16.0
    • jsonpickle==2.2.0
    • jsonschema==4.16.0
    • jupyter==1.0.0
    • jupyter-client==6.1.5
    • jupyter-console==6.4.4
    • jupyter-core==4.11.1
    • jupyterlab-pygments==0.2.2
    • jupyterlab-widgets==3.0.3
    • kiwisolver==1.4.4
    • lightning-bolts==0.4.0
    • linecache2==1.0.0
    • llvmlite==0.34.0
    • mako==1.2.3
    • markdown==3.4.1
    • markdown-it-py==2.1.0
    • markupsafe==2.1.1
    • marshmallow==3.18.0
    • marshmallow-enum==1.5.1
    • matplotlib==3.3.0
    • matplotlib-inline==0.1.6
    • mccabe==0.6.1
    • mdit-py-plugins==0.3.1
    • mdurl==0.1.2
    • medpy==0.4.0
    • mistune==2.0.4
    • mlflow==1.23.1
    • mlflow-skinny==1.29.0
    • monai==0.6.0
    • more-itertools==8.14.0
    • msal==1.20.0
    • msal-extensions==0.3.1
    • msrest==0.7.1
    • msrestazure==0.6.4
    • multidict==6.0.2
    • mypy==0.910
    • mypy-extensions==0.4.3
    • myst-parser==0.18.0
    • nbclient==0.7.0
    • nbconvert==7.2.1
    • nbformat==5.7.0
    • ndg-httpsclient==0.5.1
    • nest-asyncio==1.5.6
    • networkx==2.8.7
    • nibabel==4.0.2
    • nnunet==1.7.0
    • notebook==6.4.12
    • numba==0.51.2
    • numpy==1.19.1
    • oauthlib==3.2.1
    • opencv-python-headless==4.5.1.48
    • packaging==21.3
    • pandas==1.1.0
    • pandocfilters==1.5.0
    • papermill==2.2.2
    • param==1.9.3
    • parso==0.8.3
    • pathspec==0.10.1
    • pathtools==0.1.2
    • pexpect==4.8.0
    • pickleshare==0.7.5
    • pillow==9.1.1
    • pkgutil-resolve-name==1.3.10
    • platformdirs==2.5.2
    • pluggy==0.13.1
    • portalocker==2.5.1
    • prometheus-client==0.14.1
    • prometheus-flask-exporter==0.20.3
    • prompt-toolkit==3.0.38
    • protobuf==3.20.1
    • psutil==5.7.2
    • ptyprocess==0.7.0
    • py==1.11.0
    • pyarrow==3.0.0
    • pyasn1==0.4.8
    • pyasn1-modules==0.2.8
    • pycodestyle==2.6.0
    • pycparser==2.21
    • pydeprecate==0.3.2
    • pydicom==2.0.0
    • pyflakes==2.2.0
    • pygments==2.15.1
    • pyjwt==2.4.0
    • pynndescent==0.5.7
    • pyopenssl==20.0.1
    • pyparsing==3.0.9
    • pyrsistent==0.18.1
    • pysocks==1.7.1
    • pytest==6.0.1
    • pytest-cov==2.10.1
    • pytest-forked==1.3.0
    • pytest-xdist==1.34.0
    • python-dateutil==2.8.2
    • python-gdcm==3.0.22
    • pytorch-lightning==1.6.5
    • pytz==2022.4
    • pywavelets==1.4.1
    • pyyaml==6.0
    • pyzmq==24.0.1
    • qtconsole==5.3.2
    • qtpy==2.2.1
    • querystring-parser==1.2.4
    • radio==0.1.0
    • requests==2.28.1
    • requests-oauthlib==1.3.1
    • rich==10.13.0
    • rpdb==0.1.6
    • rsa==4.9
    • ruamel-yaml==0.16.12
    • ruamel-yaml-clib==0.2.6
    • runstats==1.8.0
    • scikit-image==0.17.2
    • scikit-learn==0.23.2
    • scipy==1.5.2
    • seaborn==0.10.1
    • secretstorage==3.3.3
    • send2trash==1.8.0
    • simpleitk==1.2.4
    • six==1.15.0
    • sklearn==0.0.post5
    • smmap==5.0.0
    • snowballstemmer==2.2.0
    • soupsieve==2.3.2.post1
    • sphinx==5.0.2
    • sphinx-argparse==0.3.1
    • sphinx-basic-ng==1.0.0b1
    • sphinx-rtd-theme==1.0.0
    • sphinxcontrib-applehelp==1.0.2
    • sphinxcontrib-devhelp==1.0.2
    • sphinxcontrib-htmlhelp==2.0.0
    • sphinxcontrib-jsmath==1.0.1
    • sphinxcontrib-qthelp==1.0.3
    • sphinxcontrib-serializinghtml==1.1.5
    • sqlalchemy==1.4.41
    • sqlparse==0.4.3
    • stopit==1.1.2
    • stringcase==1.2.0
    • tabulate==0.8.7
    • tenacity==8.1.0
    • tensorboard==2.3.0
    • tensorboard-plugin-wit==1.8.1
    • tensorboardx==2.1
    • terminado==0.16.0
    • textwrap3==0.9.2
    • threadpoolctl==3.1.0
    • tifffile==2022.10.10
    • tinycss2==1.1.1
    • toml==0.10.2
    • tomli==2.0.1
    • torchio==0.18.74
    • torchmetrics==0.6.0
    • tornado==6.2
    • tqdm==4.64.1
    • traceback2==1.4.0
    • traitlets==5.9.0
    • typing-inspect==0.8.0
    • umap-learn==0.5.2
    • unittest2==1.1.0
    • urllib3==1.26.7
    • wcwidth==0.2.6
    • webencodings==0.5.1
    • websocket-client==1.4.1
    • werkzeug==2.2.2
    • widgetsnbextension==4.0.3
    • wrapt==1.14.1
    • yacs==0.1.8
    • yarl==1.8.1
    • zipp==3.9.0
      prefix: /home/.conda/envs/InnerEye3
@Adapter668 Adapter668 added the bug Something isn't working label May 22, 2023
@ant0nsc
Copy link
Contributor

ant0nsc commented May 22, 2023

Thanks @Adapter668 - this should be easily fixed, would you like to give it a try?
All you'd need to do is add self.container.outputs_folder.mkdir(parents=True, exist_ok=True) in the run_inference_for_lightning_models, before it calls change_working_directory(self.container.outputs_folder)

@ant0nsc ant0nsc added the good first issue Good for newcomers label May 22, 2023
@Adapter668
Copy link
Author

Thanks, I'll try your solution :) I used workaround and added os.makedirs in common_util.py for now, but I guess it'll be nice in the future if the --no-train flag works without changing code

@ant0nsc
Copy link
Contributor

ant0nsc commented May 30, 2023

Hi @Adapter668 , I was more thinking that you could create a PR with that change to integrate it for later use

@Adapter668
Copy link
Author

Yes, I know, I just need to have some time to test if it works fine and I'll create PR

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Something isn't working good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ant0nsc @Adapter668