| WARNING | nndet.core.boxes.nms:<module>:26 - nnDetection was not build with GPU support!

[Wangcody123]

❓ Question

Validation sanity check: 0%| | 0/10 [00:00<?, ?it/s]using pin_memory on device 0
Traceback (most recent call last):
File "train.py", line 497, in
train()
File "/var/data/wly/nnDetection-main/scripts/nndet/utils/check.py", line 62, in wrapper
return func(*args, **kwargs)
File "train.py", line 70, in train
_train(
File "train.py", line 290, in _train
trainer.fit(module, datamodule=datamodule)
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 553, in fit
self._run(model)
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 918, in _run
self._dispatch()
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 986, in _dispatch
self.accelerator.start_training(self)
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 92, in start_training
self.training_type_plugin.start_training(trainer)
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in start_training
self._results = trainer.run_stage()
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 996, in run_stage
return self._run_train()
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1031, in _run_train
self._run_sanity_check(self.lightning_module)
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1115, in _run_sanity_check
self._evaluation_loop.run()
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
self.advance(*args, **kwargs)
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 110, in advance
dl_outputs = self.epoch_loop.run(
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
self.advance(*args, **kwargs)
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 110, in advance
output = self.evaluation_step(batch, batch_idx, dataloader_idx)
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 154, in evaluation_step
output = self.trainer.accelerator.validation_step(step_kwargs)
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 211, in validation_step
return self.training_type_plugin.validation_step(*step_kwargs.values())
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 178, in validation_step
return self.model.validation_step(*args, **kwargs)
File "/var/data/wly/nnDetection-main/scripts/nndet/ptmodule/retinaunet/base.py", line 172, in validation_step
losses, prediction = self.model.train_step(
File "/var/data/wly/nnDetection-main/scripts/nndet/core/retina.py", line 146, in train_step
prediction = self.postprocess_for_inference(
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/var/data/wly/nnDetection-main/scripts/nndet/core/retina.py", line 187, in postprocess_for_inference
boxes, probs, labels = self.postprocess_detections(
File "/var/data/wly/nnDetection-main/scripts/nndet/core/retina.py", line 326, in postprocess_detections
boxes, probs, labels = self.postprocess_detections_single_image(boxes, probs, image_shape)
File "/var/data/wly/nnDetection-main/scripts/nndet/core/retina.py", line 375, in postprocess_detections_single_image
keep = box_utils.batched_nms(boxes, probs, labels, self.nms_thresh)
File "/var/data/wly/nnDetection-main/scripts/nndet/core/boxes/nms.py", line 106, in batched_nms
return nms(boxes_for_nms, scores, iou_threshold)
File "/home/wly/miniconda3/envs/lung_medicle/lib/python3.8/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "/var/data/wly/nnDetection-main/scripts/nndet/core/boxes/nms.py", line 78, in nms
return nms_fn(boxes.float(), scores.float(), iou_threshold)
TypeError: 'NoneType' object is not callable
The error during training indicates that the GPU is not functioning properly, despite having four GPUs available

[mibaumgartner]

Something went wrong during the installation, please check the FAQ and other Issues for guidance.

[Wangcody123]

Something went wrong during the installation, please check the FAQ and other Issues for guidance.

Thank you, but during the final step of the model inference process, it shows:
INFO Running preprocessing of test cases INFO Found 0 cases for preprocessing in /var/data/wly/nnDetection-main/projects/Task016_Luna/raw_splitted and 0 already processed cases. INFO Running inference INFO Found 2 models to ensemble INFO Architecture overwrites: {} Anchor overwrites: {} INFO Building architecture according to the plan of RetinaUNetV001 INFO Start channels: 32; head channels: 128; fpn channels: 128 INFO Discarding anchor generator kwargs {'stride': 1} INFO Building:: encoder Encoder: {} INFO Building:: decoder UFPNModular: {'min_out_channels': 8, 'upsampling_mode': 'transpose', 'num_lateral': 1, 'norm_lateral': False, 'activation_lateral': False, 'num_out': 1, 'norm_out': False, 'activation_out': False} INFO Running ATSS Matching with num_candidates=4 and center_in_gt False. INFO Building:: classifier BCECLassifier: {'num_convs': 1, 'norm_channels_per_group': 16, 'norm_affine': True, 'reduction': 'mean', 'loss_weight': 1.0, 'prior_prob': 0.01} INFO Init classifier weights: prior prob 0.01 INFO Building:: regressor GIoURegressor: {'num_convs': 1, 'norm_channels_per_group': 16, 'norm_affine': True, 'reduction': 'sum', 'loss_weight': 1.0, 'learn_scale': True} INFO Learning level-specific scalar in regressor INFO Overwriting regressor conv weight init INFO Building:: head DetectionHeadHNMNative: {} sampler HardNegativeSamplerBatched: {'batch_size_per_image': 32, 'positive_fraction': 0.33, 'pool_size': 20, 'min_neg': 1} INFO Sampling hard negatives on a per-batch basis INFO Building:: segmenter DiCESegmenterFgBg {'dice_kwargs': {'batch_dice': True}} INFO Running batch dice True and do bg False in dice loss. INFO Model Inference Summary: detections_per_img: 100 score_thresh: 0 topk_candidates: 10000 remove_small_boxes: 0.01 nms_thresh: 0.6 INFO Loaded /var/data/wly/nnDetection-main/projects/Task016_Luna/models/Task016_Luna/RetinaUNetV001_D3V001_3d/fold0/model_last.ckpt with INFO Architecture overwrites: {} Anchor overwrites: {} INFO Building architecture according to the plan of RetinaUNetV001 INFO Start channels: 32; head channels: 128; fpn channels: 128 INFO Discarding anchor generator kwargs {'stride': 1} INFO Building:: encoder Encoder: {} INFO Building:: decoder UFPNModular: {'min_out_channels': 8, 'upsampling_mode': 'transpose', 'num_lateral': 1, 'norm_lateral': False, 'activation_lateral': False, 'num_out': 1, 'norm_out': False, 'activation_out': False} INFO Running ATSS Matching with num_candidates=4 and center_in_gt False. INFO Building:: classifier BCECLassifier: {'num_convs': 1, 'norm_channels_per_group': 16, 'norm_affine': True, 'reduction': 'mean', 'loss_weight': 1.0, 'prior_prob': 0.01} INFO Init classifier weights: prior prob 0.01 INFO Building:: regressor GIoURegressor: {'num_convs': 1, 'norm_channels_per_group': 16, 'norm_affine': True, 'reduction': 'sum', 'loss_weight': 1.0, 'learn_scale': True} INFO Learning level-specific scalar in regressor INFO Overwriting regressor conv weight init INFO Building:: head DetectionHeadHNMNative: {} sampler HardNegativeSamplerBatched: {'batch_size_per_image': 32, 'positive_fraction': 0.33, 'pool_size': 20, 'min_neg': 1} INFO Sampling hard negatives on a per-batch basis INFO Building:: segmenter DiCESegmenterFgBg {'dice_kwargs': {'batch_dice': True, 'smooth_nom': 1e-05, 'smooth_denom': 1e-05, 'do_bg': False}} INFO Running batch dice True and do bg False in dice loss. INFO Model Inference Summary: detections_per_img: 100 score_thresh: 0 topk_candidates: 10000 remove_small_boxes: 0.01 nms_thresh: 0.6 INFO Loaded /var/data/wly/nnDetection-main/projects/Task016_Luna/models/Task016_Luna/RetinaUNetV001_D3V001_3d/fold0/model_best.ckpt with INFO Found inference plan: {'model_iou': 1e-05, 'model_nms_fn': <function batched_weighted_nms_model at 0x7f795d071700>, 'model_score_thresh': 0.0, 'model_topk': 1000, 'model_detections_per_image': 100, 'ensemble_iou': 0.2, 'ensemble_nms_fn': <function batched_wbc_ensemble at 0x7f795d071940>, 'ensemble_topk': 1000, 'remove_small_boxes': 2.0, 'ensemble_score_thresh': 0.0} for prediction INFO Adding 2D Mirror TTA for prediction. INFO Adding 3D Mirror TTA for prediction. INFO Using 8 TTA transformations for prediction (one dummy trafo). INFO Found 0 files for inference.

I placed my custom test set in 'raw_splitted/imagesTs', but the prediction tells me that 0 data files were found. How should I address this issue?

[mibaumgartner]

The path where nnDetection looks for the data is mentioned in the first line of your posted message, pleas make sure that the imagesTs folder exists there. The files need to follow the same structure as the imagesTr i.e. end with '_0000' and needs to be nifties (ending with '.nii.gz').
Please double check that all paths and names are correct.

[Wangcody123]

The path where nnDetection looks for the data is mentioned in the first line of your posted message, pleas make sure that the imagesTs folder exists there. The files need to follow the same structure as the imagesTr i.e. end with '_0000' and needs to be nifties (ending with '.nii.gz'). Please double check that all paths and names are correct.
thanks so much，but I'm trying to reproduce the results of luna16 using NNdetection, but I found that the predicted results only contain the coordinates of the boxes, without diameters. Also, the official prepare_eval_cpm.py script provided in Task016 for converting results to luna16 format files only includes the coordinates of the centers, without diameters. I want to know how to solve this issue because without diameters, I can't use the evaluation program provided by luna16 for assessment.

[mibaumgartner]

LUNA16 only requires the prediction of a center point of the lesion and not the diameter. They check if the predicted point is inside the radius of the ground truth lesion.

You can see an example results file here where they also only use points: https://luna16.grand-challenge.org/Evaluation/

[mibaumgartner]

Note however: LUNA16 was run as a 10Fold Cross-Validation as was done in DeepLung; so to reproduce the results you don't need imagesTs which was mentioned in your previous message.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.