Deployment of custom Monai Application Package

[ArpanGyawali]

I am trying to deploy a Monai-Application for Skull stripping that I created using Monai label. I have to deploy the application with the trained model for inference purpose.
So, i packaged the application using the monai-deploy. I am using the version 0.5.1. I followed the documentation of monai-deploy-app-sdk for Spleen Segmentation App.
I pushed the package in GitHub package (ghcr.io) using my GitHub token.

I can run the application using "monai-exec" easily but when I try to run the packaged application using this command:
monai-deploy run ghcr.io/my_organization_in_lower_case/my_image_name:latest input_dir output_dir

I need nvidia-docker to be installed in the system. I tried installing it and NVIDIA Container Toolkit based on this documentation by Nvidia, but I again get various errors and now i am stuck in "docker: Error response from daemon: Unknown runtime specified nvidia"

I also looked at this discussion on running a MAP container without using App SDK CLI. But, also here, Nvidia Container Toolkit and/or nvidia-docker 2 needs to be installed for support of GPU.
Can we package the application using any other non-nvidia(nvcr.io) base image so that it can be run in the system without GPU(inference on only CPU), or it doesnt require nvidia-docker to be installed in the system?
Is there is any way to run the application package only on CPU.

I have been stuck in this problem for a long time. Any help would be highly appreciated.

[KnightCoder]

Did you check if docker is still running? Secondly, did you try restarting the docker daemon and then re-packaging and running the MAP? What are the other errors? Please specify the command that you are using to generate the MAP?

[ArpanGyawali]

Thanks for the reply @KnightCoder,
I tried restarting docker, but it didn't work. I tried every solution from here but nothing worked.
I didn't perform repackaging, how can I repackage a pulled image?
Command used for generating the MAP:
monai-deploy package -b nvcr.io/nvidia/pytorch:22.08-py3 my_app --tag ghcr.io/my_organization_in_lower_case/my_image_name:latest -m torchSscript_model_file

And, I also need to run the inference in the system with CPU only. Is there a way we can deploy the model which doesn't require GPU for inference? If yes, then it would be a great help.

[ArpanGyawali]

@vikashg @MMelQin
It would be great if someone could help me out?

[ArpanGyawali]

Thank you @MMelQin
i extracted the metadata header and affine from the nifty files using nibabel and the metadata i accured were

sizeof_hdr: 348, type <class 'numpy.ndarray'>
data_type: b'', type <class 'numpy.ndarray'>
db_name: b'', type <class 'numpy.ndarray'>
extents: 0, type <class 'numpy.ndarray'>
session_error: 0, type <class 'numpy.ndarray'>
regular: b'r', type <class 'numpy.ndarray'>
dim_info: 0, type <class 'numpy.ndarray'>
dim: [  3 512 512 625   1   0   0   0], type <class 'numpy.ndarray'>
intent_p1: 0.0, type <class 'numpy.ndarray'>
intent_p2: 0.0, type <class 'numpy.ndarray'>
intent_p3: 0.0, type <class 'numpy.ndarray'>
intent_code: 0, type <class 'numpy.ndarray'>
datatype: 4, type <class 'numpy.ndarray'>
bitpix: 16, type <class 'numpy.ndarray'>
slice_start: 0, type <class 'numpy.ndarray'>
pixdim: [-1.        0.507812  0.507812  0.625     0.        0.        0.
  0.      ], type <class 'numpy.ndarray'>
vox_offset: 0.0, type <class 'numpy.ndarray'>
scl_slope: nan, type <class 'numpy.ndarray'>
scl_inter: nan, type <class 'numpy.ndarray'>
slice_end: 0, type <class 'numpy.ndarray'>
slice_code: 0, type <class 'numpy.ndarray'>
xyzt_units: 10, type <class 'numpy.ndarray'>
cal_max: 0.0, type <class 'numpy.ndarray'>
cal_min: 0.0, type <class 'numpy.ndarray'>
slice_duration: 0.0, type <class 'numpy.ndarray'>
toffset: 0.0, type <class 'numpy.ndarray'>
glmax: 0, type <class 'numpy.ndarray'>
glmin: 0, type <class 'numpy.ndarray'>
descrip: b'TE=8;Time=64930.224', type <class 'numpy.ndarray'>
aux_file: b'', type <class 'numpy.ndarray'>
qform_code: 1, type <class 'numpy.ndarray'>
sform_code: 1, type <class 'numpy.ndarray'>
quatern_b: 0.0, type <class 'numpy.ndarray'>
quatern_c: 1.0, type <class 'numpy.ndarray'>
quatern_d: 0.0, type <class 'numpy.ndarray'>
qoffset_x: 130.0, type <class 'numpy.ndarray'>
qoffset_y: -157.1919403076172, type <class 'numpy.ndarray'>
qoffset_z: -48.25, type <class 'numpy.ndarray'>
srow_x: [ -0.507812   0.        -0.       130.      ], type <class 'numpy.ndarray'>
srow_y: [  -0.          0.507812   -0.       -157.19194 ], type <class 'numpy.ndarray'>
srow_z: [  0.      0.      0.625 -48.25 ], type <class 'numpy.ndarray'>
intent_name: b'', type <class 'numpy.ndarray'>
magic: b'n+1', type <class 'numpy.ndarray'>
affine: [[  -0.50781202    0.           -0.          130.        ]
 [  -0.            0.50781202   -0.         -157.19194031]
 [   0.            0.            0.625       -48.25      ]
 [   0.            0.            0.            1.        ]], type <class 'numpy.ndarray'>
original_affine: [[  -0.50781202    0.           -0.          130.        ]
 [  -0.            0.50781202   -0.         -157.19194031]
 [   0.            0.            0.625       -48.25      ]
 [   0.            0.            0.            1.        ]], type <class 'numpy.ndarray'>
spatial_shape: [512 512 625], type <class 'numpy.ndarray'>
space: RAS, type <enum 'SpaceKeys'>
original_channel_dim: nan, type <class 'float'>

Even though the below error was thrown

Is it because nibabel doesnt give the metadata named row_pixel_spacing?
How do i get all the metadata that are required?

[vikashg]

Hi @ArpanGyawali what are you using for skull stripping? I tried to use FSL's BET tool through a shutil command and I was getting an segmentation fault error. Are you using a deep learning-based tool or something?

[vikashg]

Also What @MMelQin mentioned about metadata is not the same as the nibabel metadata. if you try LoadImage() from monai transforms. It will give you a dictionary with data and metadata. I think he means that the metadata must be in that format. Can you share a gist of your code ?
Also did you try NiftiDataLoader for your nifti files ?

[vikashg]

Ignore my comment about NiftiDataLoader

[ArpanGyawali]

hi @vikashg @MMelQin yes I used deep learning. I used the segmentation method of monai label that uses UNET architecture for training from which i got the model file.
Then, using monai-deploy-app-sdk i created the skull_strip operator using the same val pre and post transform used during the training process and I created a MAP using the model file obtained after training.

The folder structure I used while using dicom file as input was :
Skull_Strip_App

• app.py
• skull_strip_operator.py
• main.py

skull_strip_operator.py

import logging
from os import path

from numpy import uint8

import monai.deploy.core as md
from monai.deploy.core import ExecutionContext, Image, InputContext, IOType, Operator, OutputContext
from monai.deploy.operators.monai_seg_inference_operator import InMemImageReader, MonaiSegInferenceOperator

from monai.transforms import (
    Activationsd,
    AsDiscreted,
    Compose,
    EnsureChannelFirstd,
    EnsureTyped,
    LoadImaged,
    ScaleIntensityRanged,
    Invertd,
    Spacingd,
    SaveImaged,
    ToTensord,
    KeepLargestConnectedComponentd,
    Flipd,
    Orientationd
)

@md.input("image", Image, IOType.IN_MEMORY)
@md.output("seg_image", Image, IOType.IN_MEMORY)
@md.env(pip_packages=["monai>=0.8.1", "torch>=1.10.2", "numpy>=1.21", "nibabel"])
class SkullStripOperator(Operator):
    """Performs skull striping from a 3d image of Brain CTA"""

    def __init__(self):
        self.logger = logging.getLogger("{}.{}".format(__name__, type(self).__name__))
        super().__init__()
        self._input_dataset_key = "image"
        self._pred_dataset_key = "pred"

    def compute(self, op_input: InputContext, op_output: OutputContext, context: ExecutionContext):

        input_image = op_input.get("image")
        if not input_image:
            raise ValueError("Input image is not found.")

        output_path = context.output.get().path


        # This operator gets an in-memory Image object, so a specialized ImageReader is needed.
        _reader = InMemImageReader(input_image)
        pre_transforms = self.pre_process(_reader, path.join(output_path, "nifty_temp"))
        post_transforms = self.post_process(pre_transforms, path.join(output_path, "prediction_output"))

        # Delegates inference and saving output to the built-in operator.
        infer_operator = MonaiSegInferenceOperator(
            (
                160,
                160,
                160,
            ),
            pre_transforms,
            post_transforms,
        )

        # Setting the keys used in the dictironary based transforms may change.
        infer_operator.input_dataset_key = self._input_dataset_key
        infer_operator.pred_dataset_key = self._pred_dataset_key

        # Now let the built-in operator handles the work with the I/O spec and execution context.
        infer_operator.compute(op_input, op_output, context)

    def pre_process(self, img_reader, out_dir: str = "./nifty_temp") -> Compose:
        """Composes transforms for preprocessing input before predicting on a model."""

        my_key = self._input_dataset_key
        return Compose(
            [
                LoadImaged(keys=my_key, reader=img_reader),
                SaveImaged(
                    keys=my_key,
                    output_dir=out_dir,
                    output_dtype=uint8,
                ),
                EnsureTyped(keys=my_key),
                EnsureChannelFirstd(keys=my_key),
                Spacingd(keys=my_key, pixdim=[1.0, 1.0, 1.0]),
                ScaleIntensityRanged(keys=my_key, a_min=-200, a_max=1000, b_min=0.0, b_max=1.0, clip=True),
            ]
        )

    def post_process(self, pre_transforms: Compose, out_dir: str = "./prediction_output") -> Compose:
        """Composes transforms for postprocessing the prediction results."""

        pred_key = self._pred_dataset_key
        return Compose(
            [
                EnsureTyped(keys=pred_key),
                Orientationd(keys=pred_key, axcodes='RPI', labels=(('R', 'L'), ('P', 'A'), ('I', 'S'))),
                # Orientationd(keys=pred_key, axcodes='RAS'), 
                KeepLargestConnectedComponentd(keys=pred_key),
                Invertd(
                    keys=pred_key,
                    transform=pre_transforms,
                    orig_keys=self._input_dataset_key,
                    nearest_interp=False,
                    to_tensor=True,
                ),
                Activationsd(keys=pred_key, softmax=True),
                AsDiscreted(keys=pred_key, argmax=True),
                
                SaveImaged(
                    keys=pred_key,
                    output_dir=out_dir,
                    output_postfix="seg",
                    output_dtype=uint8,
                ),
            ]
        )

app.py

import logging

from SkullStrip_operator import SkullStripOperator
from pydicom.sr.codedict import codes
from monai.deploy.core.domain import Image
from monai.deploy.core.io_type import IOType

from monai.deploy.core import Application, resource

from monai.deploy.operators.dicom_data_loader_operator import DICOMDataLoaderOperator
from monai.deploy.operators.dicom_seg_writer_operator import DICOMSegmentationWriterOperator, SegmentDescription
from monai.deploy.operators.dicom_series_selector_operator import DICOMSeriesSelectorOperator
from monai.deploy.operators.dicom_series_to_volume_operator import DICOMSeriesToVolumeOperator
from monai.deploy.operators.monai_bundle_inference_operator import (
    BundleConfigNames,
    IOMapping,
    MonaiBundleInferenceOperator,
)

@resource(cpu=1, gpu=1, memory="7Gi")
class AISkullStripApp(Application):
    def __init__(self, *args, **kwargs):
        """Creates an application instance."""
        self._logger = logging.getLogger("{}.{}".format(__name__, type(self).__name__))
        super().__init__(*args, **kwargs)

    def run(self, *args, **kwargs):
        # This method calls the base class to run. Can be omitted if simply calling through.
        self._logger.info(f"Begin {self.run.__name__}")
        super().run(*args, **kwargs)
        self._logger.info(f"End {self.run.__name__}")

    def compose(self):
        """Creates the app specific operators and chain them up in the processing DAG."""

        self._logger.debug(f"Begin {self.compose.__name__}")
        # Creates the custom operator(s) as well as SDK built-in operator(s).
        study_loader_op = DICOMDataLoaderOperator()
        series_selector_op = DICOMSeriesSelectorOperator(rules=Sample_Rules_Text)
        series_to_vol_op = DICOMSeriesToVolumeOperator()
        # Model specific inference operator, supporting MONAI transforms.

        # Creates the model specific segmentation operator
        spleen_seg_op = SkullStripOperator()

        _algorithm_name = "3D segmentation of the Brain for Skull Stripping"
        _algorithm_family = codes.DCM.ArtificialIntelligence
        _algorithm_version = "0.1.0"

        segment_descriptions = [
            SegmentDescription(
                segment_label="brain",
                segmented_property_category=codes.SCT.Organ,
                segmented_property_type=codes.SCT.Brain,
                algorithm_name=_algorithm_name,
                algorithm_family=_algorithm_family,
                algorithm_version=_algorithm_version,
            ),
        ]

        custom_tags = {"SeriesDescription": "Skull Striping using AI"}

        dicom_seg_writer = DICOMSegmentationWriterOperator(
            segment_descriptions=segment_descriptions, custom_tags=custom_tags, omit_empty_frames=False
        )

        self.add_flow(study_loader_op, series_selector_op, {"dicom_study_list": "dicom_study_list"})
        self.add_flow(
            series_selector_op, series_to_vol_op, {"study_selected_series_list": "study_selected_series_list"}
        )
        self.add_flow(series_to_vol_op, spleen_seg_op, {"image": "image"})
        # Note below the dicom_seg_writer requires two inputs, each coming from a source operator.
        self.add_flow(
            series_selector_op, dicom_seg_writer, {"study_selected_series_list": "study_selected_series_list"}
        )
        self.add_flow(spleen_seg_op, dicom_seg_writer, {"seg_image": "seg_image"})
        self._logger.debug(f"End {self.compose.__name__}")

Sample_Rules_Text = """
{
    "selections": [
        {
            "name": "(.*?)",
            "conditions": {
                "StudyDescription": "(.*?)",
                "Modality": "(?i)CT",
                "SeriesDescription": "(.*?)"
            }
        }
    ]
}
"""


if __name__ == "__main__":
    logging.basicConfig(level=logging.DEBUG)
    AISkullStripApp(do_run=True)

Later I needed to use Nifty file as input instead of dicom file so i created a new operator called nifty_loader_operator, that is different from the buildin niftyloader

nifty_loader_operator.py

import numpy as np

import monai.deploy.core as md
from monai.deploy.core import DataPath, ExecutionContext, InputContext, IOType, Operator, OutputContext, Image

from monai.utils import MetaKeys, SpaceKeys

from monai.deploy.utils.importutil import optional_import

nib, _ = optional_import("nibabel")


@md.input("image_path", DataPath, IOType.DISK)
@md.output("image", Image, IOType.IN_MEMORY)
@md.env(pip_packages=["SimpleITK>=2.0.2", "nibabel"])
class NiftiDataLoader(Operator):
    """
    This operator reads a nifti image, extracts the numpy array and forwards it to the next operator
    """


    def compute(self, op_input: InputContext, op_output: OutputContext, context: ExecutionContext):
        input_path = op_input.get().path
        image = self.convert_and_save(input_path)
        op_output.set(image)



    def convert_and_save(self, nii_path):
        img = nib.load(nii_path)
        header = self._get_meta_dict(img)
        header[MetaKeys.AFFINE] = self._get_affine(img)
        data = self._get_array_data(img)
        return Image(data, header)
    
    def _get_meta_dict(self, img) -> dict:
        # swap to little endian as PyTorch doesn't support big endian
        header2 = dict()
        try:
            header1 = img.header.as_byteswapped("<")
        except ValueError:
            header1 = img.header
        header1 = dict(header1)
        header2["row_pixel_spacing"] = header1["pixdim"][1]
        header2["col_pixel_spacing"] = header1["pixdim"][2]
        header2["depth_pixel_spacing"] = header1["pixdim"][3]
        header1.update(header2)
        return header1
    
    def _get_affine(self, img):
        return np.array(img.affine, copy=True)
    
    
    def _get_array_data(self, img):
        data = np.asanyarray(img.dataobj, order="C")
        print(f'**********{data.shape}******')
        return data

app.py

import logging

from skull_strip_operator import SkullStripOperator
from pydicom.sr.codedict import codes
from monai.deploy.core.domain import Image
from monai.deploy.core.io_type import IOType

from monai.deploy.core import Application, resource

from nifty_data_loader_operator2 import NiftiDataLoader
from monai.deploy.operators.monai_bundle_inference_operator import (
    BundleConfigNames,
    IOMapping,
    MonaiBundleInferenceOperator,
)

@resource(cpu=1, gpu=1, memory="7Gi")
class AISkullStripApp(Application):
    def __init__(self, *args, **kwargs):
        """Creates an application instance."""
        self._logger = logging.getLogger(
            "{}.{}".format(__name__, type(self).__name__))
        super().__init__(*args, **kwargs)

    def run(self, *args, **kwargs):
        # This method calls the base class to run. Can be omitted if simply calling through.
        self._logger.info(f"Begin {self.run.__name__}")
        super().run(*args, **kwargs)
        self._logger.info(f"End {self.run.__name__}")

    def compose(self):
        """Creates the app specific operators and chain them up in the processing DAG."""

        self._logger.debug(f"Begin {self.compose.__name__}")
        nifty_loader_op = NiftiDataLoader()
        skull_strip_op = SkullStripOperator()
        self.add_flow(nifty_loader_op, skull_strip_op, {"image": "image"})
        self._logger.debug(f"End {self.compose.__name__}")
 """


if __name__ == "__main__":
    # Creates the app and test it standalone. When running is this mode, please note the following:
    #     -m <model file>, for model file path
    #     -i <DICOM folder>, for input DICOM CT series folder
    #     -o <output folder>, for the output folder, default $PWD/output
    # e.g.
    #     monai-deploy exec app.py -i input -m model/model.ts
    #
    logging.basicConfig(level=logging.DEBUG)
    AISkullStripApp(do_run=True)

The skull-strip operator is same as before

Doing this i havent got the KeyError for row_pixel_spacing and the code runs completely running using
monai-deploy exec SkullStripApp -i input_nifty -o OutputFolder -m model_file

But when i opened the file and the output segmentation i got the dimension mismatch error:
Shape of main image: (512,512,650)
Shape of segmentation: (650,512,512)

Also the other issues I have got are,
The orientation of the dicom file is RAS and nifty file is RPI
When i load dicom file as main image, then the orientation of the predicted segmentation matches with the main image and they align
But, when i load the nifty format of the same image as main image, then the same predicted segmentation loads up upside down and doesnt align with the main image. I tried to change the Orientationd in post transform but the preicted segmentation either align with dicom image or with nifty or with neither. Why is this? Is there a way to make it aligned with both dicom and nifty main file.

When loading dicom file as main file in ITK snap

When loading nifty file as main and same predicted segmentation in ITK snap

Sorry, I know I have asked a lot in this question.
Can you help me with these issues.

[MMelQin]

@ArpanGyawali Thanks for the questions. No worries, you had just gotten a taste of how tricky it is to navigate the different format and representation of image in different format/frameworks.

• DCIOM Pixel Data are stored differently than NIfTI, row first with the latter column first. I had taken care to make sure the DICOM Series to Vol order the dims correctly, same as ITK convention, and in the App SDK Image class the numpy array reflects that too.
• Since MONAI and the models expect certain dim ordering, e.g. CDHW, the numpy in the Image class is assumed having order the dims as such, though for inference of typical radiology images, there is only one channel.
• Orientation is yet another tricky ones, and there have been many discussions and efforts trying to make it consistent in the various format. In Radiology, the anatomical orientation does not really affect the image position or orientation per se, but more related to relating to the patient's body. The Rad convention is LPS with origin at the top left corner, while neurology is different (don't want to dive into this, but you can read all about it). It get worse when NIfTI images are converted to DICOM, and the results may be at the mercy of the converters' interpretation.
• In any case, if your input is NIfTI, and the segmentation is saved using MONAI saveimage function, I would say the seg would be correct.

[ArpanGyawali]

Thank you so much @MMelQin
I got much insights, that helped me solve orientatiion mismatch problem.

[MMelQin]

Much glad to hear that you sorted this out!