About Running Issues on NVIDIA RTX 4090 Graphics Card

[azhe990510]

When using the RTX 4090 graphics card and installing the MinkowskiEngine, we encountered the following error when performing pooling operations such as AvgPooling. However, no such error occurs when using an RTX 3090.

 File "/root/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/anaconda3/lib/python3.9/site-packages/MinkowskiEngine/MinkowskiPooling.py", line 171, in forward
    outfeat = self.pooling.apply(
  File "/root/anaconda3/lib/python3.9/site-packages/MinkowskiEngine/MinkowskiPooling.py", line 70, in forward
    out_feat, num_nonzero = fw_fn(
RuntimeError: <unknown> at /tmp/pip-install-u79gcb3m/minkowskiengine_c9dccb781db24a2d81cf8b123a030ecd/src/gpu.cu:100

Is there any solution available to support the RTX 4090 graphics card?

[XuyangZhang0223]

I also encounter this error when using my RTX 4060 graphics card, the error is the same as yours (MinkowskiEngine/src/gpu.cu:100) when I set the SparseTensorQuantizationMode to UNWEIGHTED_AVERAGE, but there is no error when it is set to RANDOM_SUBSAMPLE.

Have you find the solution? Thx!

[ZiliangMiao]

I also encounter the same error using RTX 4090
RuntimeError: at /tmp/pip-req-build-up7naarj/src/gpu.cu:100

[ZiliangMiao]

@chrischoy Same problem still unsolved, could you please provide any suggestions?
RTX4090, python=3.8, pytorch=1.10.0-cu111, pytorch-lightning=1.9.0, MinkowskiEngine=0.5.4
If quantization_mode=ME.SparseTensorQuantizationMode.UNWEIGHTED_AVERAGE, it will cause RuntimeError:

  File "/home/user/Projects/MosPretrain/scripts/train.py", line 101, in <module>
    main()
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/user/Projects/MosPretrain/scripts/train.py", line 97, in main
    trainer.fit(model, train_dataloader, val_dataloader, ckpt_path=resume)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 88, in launch
    return function(*args, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1103, in _run
    results = self._run_stage()
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1182, in _run_stage
    self._run_train()
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1195, in _run_train
    self._run_sanity_check()
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1267, in _run_sanity_check
    val_loop.run()
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 152, in advance
    dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 137, in advance
    output = self._evaluation_step(**kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 234, in _evaluation_step
    output = self.trainer._call_strategy_hook(hook_name, *kwargs.values())
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1485, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 359, in validation_step
    return self.model(*args, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 886, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 110, in forward
    return self._forward_module.validation_step(*inputs, **kwargs)
  File "/home/user/Projects/MosPretrain/src/mos4d/models/nusc_models.py", line 88, in validation_step
    out = self.forward(point_clouds)
  File "/home/user/Projects/MosPretrain/src/mos4d/models/nusc_models.py", line 61, in forward
    out = self.model(past_point_clouds)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/Projects/MosPretrain/src/mos4d/models/nusc_models.py", line 200, in forward
    sparse_tensor = tensor_field.sparse()
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiTensorField.py", line 354, in sparse
    features = MinkowskiSPMMAverageFunction().apply(
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/sparse_matrix_functions.py", line 183, in forward
    result, COO, vals = spmm_average(
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/sparse_matrix_functions.py", line 93, in spmm_average
    result, COO, vals = MEB.coo_spmm_average_int32(
RuntimeError: <unknown> at /home/user/Installations/MinkowskiEngine/src/gpu.cu:100