Problem with number of classes

[lcoandrade]

Hi again,
I'm working on a Kaggle notebook to peform building footprint extraction.

The dataset on has background and buildings, therefore I have this class definition:

names = ['background', 'buildings']

This is my task:

task = SemanticSegmentationTask(
    model = 'deeplabv3+',
    backbone = 'resnet50',
    weights = True, # to use imagenet. Before we should define weights='imagenet'
    in_channels = 3,
    num_classes = len(names),
    loss = 'focal',
    ignore_index = None,
    lr = 1e-5,
    patience = 5, 
)

And this is my training configuration:

logger = CSVLogger(
    save_dir=OUTPUT_DIR, 
)

checkpoint_callback = ModelCheckpoint(
    every_n_epochs=1,
    dirpath=OUTPUT_DIR,
    filename='checkpoint'
)

trainer = pl.Trainer(
        accelerator='cuda',
        devices=1,
        max_epochs=5,
        callbacks=[checkpoint_callback, ],
        logger=logger,
)

When I train:

trainer.fit(
        model=task,
        datamodule = datamodule,
    )

I get:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[19], line 14
     12 else:
     13     print('Starting training from scratch...')
---> 14     trainer.fit(
     15         model=task,
     16         datamodule = datamodule,
     17     )
     19 print("The time taken to train was: ", timeit.default_timer() - start)

File /opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py:544, in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    542 self.state.status = TrainerStatus.RUNNING
    543 self.training = True
--> 544 call._call_and_handle_interrupt(
    545     self, self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
    546 )

File /opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py:44, in _call_and_handle_interrupt(trainer, trainer_fn, *args, **kwargs)
     42     if trainer.strategy.launcher is not None:
     43         return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
---> 44     return trainer_fn(*args, **kwargs)
     46 except _TunerExitException:
     47     _call_teardown_hook(trainer)

File /opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py:580, in Trainer._fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    573 assert self.state.fn is not None
    574 ckpt_path = self._checkpoint_connector._select_ckpt_path(
    575     self.state.fn,
    576     ckpt_path,
    577     model_provided=True,
    578     model_connected=self.lightning_module is not None,
    579 )
--> 580 self._run(model, ckpt_path=ckpt_path)
    582 assert self.state.stopped
    583 self.training = False

File /opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py:989, in Trainer._run(self, model, ckpt_path)
    984 self._signal_connector.register_signal_handlers()
    986 # ----------------------------
    987 # RUN THE TRAINER
    988 # ----------------------------
--> 989 results = self._run_stage()
    991 # ----------------------------
    992 # POST-Training CLEAN UP
    993 # ----------------------------
    994 log.debug(f"{self.__class__.__name__}: trainer tearing down")

File /opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py:1033, in Trainer._run_stage(self)
   1031 if self.training:
   1032     with isolate_rng():
-> 1033         self._run_sanity_check()
   1034     with torch.autograd.set_detect_anomaly(self._detect_anomaly):
   1035         self.fit_loop.run()

File /opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py:1062, in Trainer._run_sanity_check(self)
   1059 call._call_callback_hooks(self, "on_sanity_check_start")
   1061 # run eval step
-> 1062 val_loop.run()
   1064 call._call_callback_hooks(self, "on_sanity_check_end")
   1066 # reset logger connector

File /opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py:182, in _no_grad_context.<locals>._decorator(self, *args, **kwargs)
    180     context_manager = torch.no_grad
    181 with context_manager():
--> 182     return loop_run(self, *args, **kwargs)

File /opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py:134, in _EvaluationLoop.run(self)
    132     self.batch_progress.is_last_batch = data_fetcher.done
    133     # run step hooks
--> 134     self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
    135 except StopIteration:
    136     # this needs to wrap the `*_step` call too (not just `next`) for `dataloader_iter` support
    137     break

File /opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py:391, in _EvaluationLoop._evaluation_step(self, batch, batch_idx, dataloader_idx, dataloader_iter)
    385 hook_name = "test_step" if trainer.testing else "validation_step"
    386 step_args = (
    387     self._build_step_args_from_hook_kwargs(hook_kwargs, hook_name)
    388     if not using_dataloader_iter
    389     else (dataloader_iter,)
    390 )
--> 391 output = call._call_strategy_hook(trainer, hook_name, *step_args)
    393 self.batch_progress.increment_processed()
    395 if using_dataloader_iter:
    396     # update the hook kwargs now that the step method might have consumed the iterator

File /opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py:309, in _call_strategy_hook(trainer, hook_name, *args, **kwargs)
    306     return None
    308 with trainer.profiler.profile(f"[Strategy]{trainer.strategy.__class__.__name__}.{hook_name}"):
--> 309     output = fn(*args, **kwargs)
    311 # restore current_fx when nested context
    312 pl_module._current_fx_name = prev_fx_name

File /opt/conda/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py:403, in Strategy.validation_step(self, *args, **kwargs)
    401 if self.model != self.lightning_module:
    402     return self._forward_redirection(self.model, self.lightning_module, "validation_step", *args, **kwargs)
--> 403 return self.lightning_module.validation_step(*args, **kwargs)

File /opt/conda/lib/python3.10/site-packages/torchgeo/trainers/segmentation.py:243, in SemanticSegmentationTask.validation_step(self, batch, batch_idx, dataloader_idx)
    241 loss = self.criterion(y_hat, y)
    242 self.log("val_loss", loss)
--> 243 self.val_metrics(y_hat_hard, y)
    244 self.log_dict(self.val_metrics)
    246 if (
    247     batch_idx < 10
    248     and hasattr(self.trainer, "datamodule")
   (...)
    252     and hasattr(self.logger.experiment, "add_figure")
    253 ):

File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /opt/conda/lib/python3.10/site-packages/torchmetrics/collections.py:198, in MetricCollection.forward(self, *args, **kwargs)
    190 @torch.jit.unused
    191 def forward(self, *args: Any, **kwargs: Any) -> Dict[str, Any]:
    192     """Call forward for each metric sequentially.
    193 
    194     Positional arguments (args) will be passed to every metric in the collection, while keyword arguments (kwargs)
    195     will be filtered based on the signature of the individual metric.
    196 
    197     """
--> 198     return self._compute_and_reduce("forward", *args, **kwargs)

File /opt/conda/lib/python3.10/site-packages/torchmetrics/collections.py:334, in MetricCollection._compute_and_reduce(self, method_name, *args, **kwargs)
    332     res = m.compute()
    333 elif method_name == "forward":
--> 334     res = m(*args, **m._filter_kwargs(**kwargs))
    335 else:
    336     raise ValueError("method_name should be either 'compute' or 'forward', but got {method_name}")

File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /opt/conda/lib/python3.10/site-packages/torchmetrics/metric.py:298, in Metric.forward(self, *args, **kwargs)
    296     self._forward_cache = self._forward_full_state_update(*args, **kwargs)
    297 else:
--> 298     self._forward_cache = self._forward_reduce_state_update(*args, **kwargs)
    300 return self._forward_cache

File /opt/conda/lib/python3.10/site-packages/torchmetrics/metric.py:367, in Metric._forward_reduce_state_update(self, *args, **kwargs)
    364 self._enable_grad = True  # allow grads for batch computation
    366 # calculate batch state and compute batch value
--> 367 self.update(*args, **kwargs)
    368 batch_val = self.compute()
    370 # reduce batch and global state

File /opt/conda/lib/python3.10/site-packages/torchmetrics/metric.py:467, in Metric._wrap_update.<locals>.wrapped_func(*args, **kwargs)
    459         if "Expected all tensors to be on" in str(err):
    460             raise RuntimeError(
    461                 "Encountered different devices in metric calculation (see stacktrace for details)."
    462                 " This could be due to the metric class not being on the same device as input."
   (...)
    465                 " device corresponds to the device of the input."
    466             ) from err
--> 467         raise err
    469 if self.compute_on_cpu:
    470     self._move_list_states_to_cpu()

File /opt/conda/lib/python3.10/site-packages/torchmetrics/metric.py:457, in Metric._wrap_update.<locals>.wrapped_func(*args, **kwargs)
    455 with torch.set_grad_enabled(self._enable_grad):
    456     try:
--> 457         update(*args, **kwargs)
    458     except RuntimeError as err:
    459         if "Expected all tensors to be on" in str(err):

File /opt/conda/lib/python3.10/site-packages/torchmetrics/classification/confusion_matrix.py:280, in MulticlassConfusionMatrix.update(self, preds, target)
    278     _multiclass_confusion_matrix_tensor_validation(preds, target, self.num_classes, self.ignore_index)
    279 preds, target = _multiclass_confusion_matrix_format(preds, target, self.ignore_index)
--> 280 confmat = _multiclass_confusion_matrix_update(preds, target, self.num_classes)
    281 self.confmat += confmat

File /opt/conda/lib/python3.10/site-packages/torchmetrics/functional/classification/confusion_matrix.py:337, in _multiclass_confusion_matrix_update(preds, target, num_classes)
    335 unique_mapping = target.to(torch.long) * num_classes + preds.to(torch.long)
    336 bins = _bincount(unique_mapping, minlength=num_classes**2)
--> 337 return bins.reshape(num_classes, num_classes)

RuntimeError: shape '[2, 2]' is invalid for input of size 511

How can I solve this?

[adamjstewart]

We already have a builtin dataset and data module for Inria Aerial Image Labeling. Have you tried using that? It's tested in CI so it's guaranteed to work.

The error message implies to me that the file you are reading contains values larger than 1, so even though there are two classes, the classes aren't 0 or 1. Hopefully that helps with debugging.

[lcoandrade]

I'm using it as a custom dataset (found in Kaggle) because I'm making a comparative study between Torchgeo and Rastervision with my students.

I already showed them how to use a custom dataset on RV and now I'm trying to show them how to perform the same work on TG. The idea is to use our on custom dataset later.

How can I reclass my masks using a custom RasterDataset?
The ones I'm using so far are this:

class MyRasterImage(RasterDataset):
    filename_glob = "*.tif"
    is_image = True
    separate_files = False
    
class MyRasterMask(RasterDataset):
    filename_glob = "*.tif"
    is_image = False
    separate_files = False

[adamjstewart]

I'm making a comparative study between Torchgeo and Rastervision with my students.

If you haven't already seen it, you may be interested in #1742.

One other way to frame the comparative study would be that Raster Vision requires you to implement stuff to get a working Inria dataset whereas TorchGeo has it built-in. But yes, it is good to show how to create new datasets.

Those dataset definitions look fine to me. The notebook you linked to earlier doesn't use these though, so I assume you've updated things locally since then.

I downloaded a sample file from the dataset. The pixel values range from 0 to 255, not 0 to 1. They need to be 0 to 1 in order to do ML. To change this, you could either:

1. Override __getitem__ in MyRasterMask to call the super-class implementation and replace 255 with 1
2. Use a dataset transform to do this
3. Use a data module aug to do this
4. Use 256 classes instead of 2 (not recommended)

Either way, at some point, the masks must be binary.

[lcoandrade]

Good to know about #1742 . Thanks for this info.

Regarding the reclassification. I would like to use a transform (considering that was needed in RV as well, so it would be a good parallel). Through a label_raster_source_kw=dict(raster_transformers=[ReclassTransformer({255: 1})]), that is applied only on the labels.

How would be this reclass transform?

[adamjstewart]

If you use a dataset transform, it will take in a dict and return a dict, so you decide which dict keys are affected. Something as simple as:

class ReclassTransformer:
    def __call__(self, sample):
        x = sample["mask"]
        x[x == 255] = 1
        sample["mask"] = x
        return sample

gt_set = MyRasterMask(paths, transforms=ReclassTransformer())

should work, but I haven't tested it.

If you use a data module augmentation, those are designed to use Kornia augmentations, so the syntax will be pretty different.

[lcoandrade]

It works fine! Thank you very much!