Updates aide-svrtk MAP with MONAI-based CNN #13

alenauus · 2023-09-13T10:05:39Z

Resolves #10

Integration of auto-proc-svrtk MONAI CNN 3D brain recon code into MAP
Updated Dockerfile to use latest model weights, and a couple of other changes due to porting code

tomaroberts · 2023-09-13T10:10:58Z

TODO:

improve folder and file structure so that code can be tested locally easily AND works inside the MAP
- currently, the code uses a mix of /home directory and /var/monai directory, which causes some pains when trying to test locally.

alenauus · 2023-09-22T09:54:30Z

the paths to MIRTK were updated to /bin - the docker script is now copied from https://github.com/SVRTK/auto-proc-svrtk

tomaroberts

@alenauus – the MAP is not quite working yet. See line-by-line comments below for some initial suggestions.

To test your changes, I recommend the following process:

Change line 1 in app/Dockerfile to create new version (0.2.1) of the MAP so you don't conflict with the released version (0.2.0), e.g.:

FROM ghcr.io/svrtk/aide-svrtk/map-init:0.2.1 AS build

Make your changes ...
Build the first stage of the MAP (known as "map-init" – this is a workaround because we need to put SVRTK into the MAP, which is third-party non-Python code):

monai-deploy package app -t ghcr.io/svrtk/aide-svrtk/map-init:0.2.1 -r requirements.txt -l DEBUG

Build the final MAP:

docker build -t ghcr.io/svrtk/aide-svrtk/map:0.2.1 app/

Create input and output directories and put some 2D DICOM stacks in the input folder
Run the MAP to test your changes:

monai-deploy run ghcr.io/svrtk/aide-svrtk/map:0.2.1 input/ output/

Everytime you make a change, repeat steps 3–6.

I expect it will error, but hopefully you should see it at least make some progress. Give that a go and then we can look at the next bits to fix.

tomaroberts · 2023-09-28T07:08:40Z

app/Dockerfile


 WORKDIR /home

 # Setup 3D UNet models
-RUN git clone https://github.com/SVRTK/Segmentation_FetalMRI.git --branch svrtk-docker-gpu-0.10 --single-branch /home/Segmentation_FetalMRI
+RUN git clone https://github.com/SVRTK/auto-proc-svrtk.git /home/auto-proc-svrtk


Most of the errors are to do with this line currently. You've changed the location of the recon Shell script, but not updated the rest of the Dockerfile and Python code to reflect the new script location, so... see other comments where I can see you need to make changes.

i fixed paths and was able to run monai-deploy run ghcr.io/svrtk/aide-svrtk/map:0.2.1 input/ output/

tomaroberts · 2023-09-28T07:15:51Z

app/Dockerfile


 # Bugfix: without below, cannot import torch within Python
 # Error: OSError: /opt/hpcx/ompi/lib/libmpi.so.40: undefined symbol: opal_hwloc201_hwloc_get_type_depth
 # Fix: https://forums.developer.nvidia.com/t/issues-building-docker-image-from-ngc-container-nvcr-io-nvidia-pytorch-22-py3/209034/5
 ENV PATH="${PATH}:/opt/hpcx/ompi/bin"
 ENV LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/hpcx/ompi/lib"

-COPY docker-recon-brain-auto.bash /home/scripts/docker-recon-brain-auto.bash
+COPY /home/auto-proc-svrtk/scripts/auto-brain-reconstruction-aide.sh /home/scripts/auto-brain-reconstruction.sh


You can remove this line entirely. The COPY command in Docker copies from your local machine into the corresponding place in the container. It fails here because /home/auto-proc-svrtk/scripts/auto-brain-reconstruction-aide.sh doesn't exist in the aide-svrtk repo.

Instead, you clone the auto-proc-svrtk repo into the container further up this file, which means auto-brain-reconstruction.sh will be downloaded into the container already.

One thing I noticed – you have two scripts in here, one of which is labelled "aide". Make sure you are pointing at the correct script.

updated thank you

tomaroberts · 2023-09-28T07:18:41Z

app/Dockerfile

 RUN mkdir -p /home/recon \
    && mkdir -p /home/output \
    && chmod +x /home/scripts/*


The most important thing here is the chmod on the scripts folder. This enables the Docker container to run the shell scripts in the folder. You need to update this so that it points at the correct folder in the cloned auto-proc-svrtk repo.

I think you can remove the mkdir lines – I don't think they are needed anymore.

updated thank you

tomaroberts · 2023-09-28T07:21:00Z

app/operators/fetal_mri_3d_brain_recon_operator.py

+#            if torch.cuda.is_available():
+#                cnn_mode = "1"
+#                logging.info("SVRTK reconstruction using GPU mode ...")
+#            if not torch.cuda.is_available():
+#                cnn_mode = "-1"
+#                logging.info("SVRTK reconstruction using CPU mode ...")
+#
+#            motion_correction_mode = "-1"  # -1 minor, 1 severe
+#            logging.info("SVRTK reconstruction ...")


I'm not keen on commenting this out. I see you've shifted the cnn_mode/motion_correction_mode variables into the bash script. The problem is that it puts a hard requirement on GPU. In testing, I've found it useful to run in CPU mode.

Leave it for now, but something we can discuss.

the current recon code is fully CPU based (it is fast)

tomaroberts · 2023-09-28T07:22:16Z

app/operators/fetal_mri_3d_brain_recon_operator.py

+#                logging.info("SVRTK reconstruction using CPU mode ...")
+#
+#            motion_correction_mode = "-1"  # -1 minor, 1 severe
+#            logging.info("SVRTK reconstruction ...")

            subprocess.run([
                "/home/scripts/docker-recon-brain-auto.bash",


This line refers to the old recon .bash script. It needs to be updated to reflect the new shell script at the correct location in the container.

updated, thank you

…/aide-svrtk into 10-monai-brain-3d-pipeline

alenauus · 2023-10-01T14:16:18Z

updated paths in the docker file and recon operator & references to auto-svrtk-proc repo

alenauus added 4 commits September 11, 2023 17:37

monai auto-proc-svrtk update

65e02cb

monai auto-proc-svrtk update

4b90fb9

log message update

5572cf4

bash script update

628a93c

tomaroberts changed the title ~~Add MONAI CNN SVRTK~~ Updates aide-svrtk MAP with MONAI-based CNN Sep 13, 2023

updated docker files with software paths

1c8d3f7

alenauus requested a review from tomaroberts September 22, 2023 09:54

tomaroberts requested changes Sep 28, 2023

View reviewed changes

alenauus added 6 commits September 28, 2023 11:29

script path and version updates

691d563

updated file paths

f44dc9d

map version update

451c02e

updated docker file

c76b8e6

Merge branch '10-monai-brain-3d-pipeline' of https://github.com/SVRTK…

7440546

…/aide-svrtk into 10-monai-brain-3d-pipeline

updated description with auto-proc-svrtk ref.

2c6f200

alenauus requested a review from tomaroberts October 1, 2023 14:18

Updates Clinical Workflow file to reflect v0.2.1

cd69bfe

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates aide-svrtk MAP with MONAI-based CNN #13

Updates aide-svrtk MAP with MONAI-based CNN #13

alenauus commented Sep 13, 2023 •

edited by tomaroberts

tomaroberts commented Sep 13, 2023 •

edited

alenauus commented Sep 22, 2023

tomaroberts left a comment •

edited

tomaroberts Sep 28, 2023

alenauus Oct 1, 2023

tomaroberts Sep 28, 2023

alenauus Oct 1, 2023

tomaroberts Sep 28, 2023

alenauus Oct 1, 2023

tomaroberts Sep 28, 2023

alenauus Oct 1, 2023 •

edited

tomaroberts Sep 28, 2023

alenauus Oct 1, 2023

alenauus commented Oct 1, 2023

Updates aide-svrtk MAP with MONAI-based CNN #13

Are you sure you want to change the base?

Updates aide-svrtk MAP with MONAI-based CNN #13

Conversation

alenauus commented Sep 13, 2023 • edited by tomaroberts

tomaroberts commented Sep 13, 2023 • edited

alenauus commented Sep 22, 2023

tomaroberts left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alenauus Oct 1, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alenauus commented Oct 1, 2023

alenauus commented Sep 13, 2023 •

edited by tomaroberts

tomaroberts commented Sep 13, 2023 •

edited

tomaroberts left a comment •

edited

alenauus Oct 1, 2023 •

edited