RuntimeError: Predictions and targets are expected to have the same shape

[nullx1337]

Hey,
i try to setup the model on an Ubuntu 22.04 with an RTX 4090.

After running

python launch.py --config configs/dreamcraft3d-coarse-nerf.yaml --train system.prompt_processor.prompt="$prompt" data.image_path="$image_path"

with following image:

i get this output resulting with this error : RuntimeError: Predictions and targets are expected to have the same shape, but got torch.Size([16384]) and torch.Size([16384, 3]).

Here is the whole log of the process

python launch.py --config configs/dreamcraft3d-coarse-nerf.yaml --train system.prompt_processor.prompt="$prompt" data.image_path="$image_path"
/root/DreamCraft3D/venv/lib/python3.10/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.13) or chardet (5.2.0)/charset_normalizer (2.0.12) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/mediapipe_face/mediapipe_face_common.py:7: UserWarning: The module 'mediapipe' is not installed. The package will have limited functionality. Please install it using the command: pip install 'mediapipe'
  warnings.warn(
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_5m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_5m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_11m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_11m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_384 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_384. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_512 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_512. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
    PyTorch 2.1.2+cu121 with CUDA 1201 (you have 2.1.2+cu118)
    Python  3.10.13 (you have 3.10.12)
  Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
  Memory-efficient attention, SwiGLU, sparse and more won't be available.
  Set XFORMERS_MORE_DETAILS=1 for more details
Seed set to 0
[INFO] Loading Deep Floyd ...

A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[unet/diffusion_pytorch_model.fp16.safetensors, safety_checker/model.fp16.safetensors, text_encoder/model.fp16-00002-of-00002.safetensors, text_encoder/model.fp16-00001-of-00002.safetensors]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.safetensors
If this behavior is not expected, please check your folder structure.
Loading pipeline components...:  33%|████████████████████████████████████████▎                                                                                | 1/3 [00:00<00:00,  7.68it/s]You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Loading pipeline components...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 10.42it/s]
[INFO] Loaded Deep Floyd!
[INFO] Loading Stable Zero123 ...
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.53 M params.
Keeping EMAs of 688.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
/root/DreamCraft3D/inputimges/falcon.jpg
[INFO] Loaded Stable Zero123!
[INFO] Using prompt [a minimalsitic marbel stone statue of an owl] and negative prompt []
[INFO] Using view-dependent prompts [side]:[a minimalsitic marbel stone statue of an owl, side view] [front]:[a minimalsitic marbel stone statue of an owl, front view] [back]:[a minimalsitic marbel stone statue of an owl, back view] [overhead]:[a minimalsitic marbel stone statue of an owl, overhead view]
/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
loaded pretrained LPIPS loss from threestudio/utils/lpips/vgg.pth
[INFO] Using 16bit Automatic Mixed Precision (AMP)
[INFO] GPU available: True (cuda), used: True
[INFO] TPU available: False, using: 0 TPU cores
[INFO] IPU available: False, using: 0 IPUs
[INFO] HPU available: False, using: 0 HPUs
[INFO] You are using a CUDA device ('NVIDIA GeForce RTX 4090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
[INFO] single image dataset: load image /root/DreamCraft3D/inputimges/falcon.jpg torch.Size([1, 128, 128, 3])
[INFO] single image dataset: load depth /root/DreamCraft3D/inputimges/falcon.jpg torch.Size([1, 128, 128, 3])
[INFO] single image dataset: load image /root/DreamCraft3D/inputimges/falcon.jpg torch.Size([1, 128, 128, 3])
[INFO] single image dataset: load depth /root/DreamCraft3D/inputimges/falcon.jpg torch.Size([1, 128, 128, 3])
[INFO] LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
[INFO]
  | Name       | Type                 | Params
----------------------------------------------------
0 | geometry   | ImplicitVolume       | 12.6 M
1 | material   | NoMaterial           | 0
2 | background | SolidColorBackground | 0
3 | renderer   | NeRFVolumeRenderer   | 0
----------------------------------------------------
12.6 M    Trainable params
0         Non-trainable params
12.6 M    Total params
50.417    Total estimated model params size (MB)
[INFO] Validation results will be saved to outputs/dreamcraft3d-coarse-nerf/a_minimalsitic_marbel_stone_statue_of_an_owl@20231222-190508/save
/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:441: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=31` in the `DataLoader` to improve performance.
/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:441: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=31` in the `DataLoader` to improve performance.
Epoch 0: |                                                                                                                                                            | 0/? [00:00<?, ?it/s]tensor([[0.6902, 0.6902, 0.7137],
        [0.6941, 0.6941, 0.7176],
        [0.6941, 0.6941, 0.7176],
        ...,
        [0.8471, 0.8549, 0.8549],
        [0.8314, 0.8392, 0.8392],
        [0.8275, 0.8353, 0.8353]], device='cuda:0')
tensor([0., 0., 0.,  ..., 0., 0., 0.], device='cuda:0',
       grad_fn=<IndexBackward0>)
Traceback (most recent call last):
  File "/root/DreamCraft3D/launch.py", line 252, in <module>
    main(args, extras)
  File "/root/DreamCraft3D/launch.py", line 195, in main
    trainer.fit(system, datamodule=dm, ckpt_path=cfg.resume)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 544, in fit
    call._call_and_handle_interrupt(
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 580, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 989, in _run
    results = self._run_stage()
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1035, in _run_stage
    self.fit_loop.run()
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 202, in run
    self.advance()
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 359, in advance
    self.epoch_loop.run(self._data_fetcher)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 136, in run
    self.advance(data_fetcher)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 240, in advance
    batch_output = self.automatic_optimization.run(trainer.optimizers[0], batch_idx, kwargs)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 187, in run
    self._optimizer_step(batch_idx, closure)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 265, in _optimizer_step
    call._call_lightning_module_hook(
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 157, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/core/module.py", line 1291, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/core/optimizer.py", line 151, in step
    step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 230, in optimizer_step
    return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/amp.py", line 77, in optimizer_step
    closure_result = closure()
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 140, in __call__
    self._result = self.closure(*args, **kwargs)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 126, in closure
    step_output = self._step_fn()
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 315, in _training_step
    training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 309, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 382, in training_step
    return self.lightning_module.training_step(*args, **kwargs)
  File "/root/DreamCraft3D/threestudio/systems/dreamcraft3d.py", line 375, in training_step
    out = self.training_substep(batch, batch_idx, guidance="ref", render_type=render_type)
  File "/root/DreamCraft3D/threestudio/systems/dreamcraft3d.py", line 176, in training_substep
    "depth_rel", 1 - self.pearson(valid_pred_depth, valid_gt_depth)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchmetrics/metric.py", line 296, in forward
    self._forward_cache = self._forward_full_state_update(*args, **kwargs)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchmetrics/metric.py", line 311, in _forward_full_state_update
    self.update(*args, **kwargs)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchmetrics/metric.py", line 470, in wrapped_func
    raise err
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchmetrics/metric.py", line 460, in wrapped_func
    update(*args, **kwargs)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchmetrics/regression/pearson.py", line 146, in update
    self.mean_x, self.mean_y, self.var_x, self.var_y, self.corr_xy, self.n_total = _pearson_corrcoef_update(
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchmetrics/functional/regression/pearson.py", line 53, in _pearson_corrcoef_update
    _check_same_shape(preds, target)
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchmetrics/utilities/checks.py", line 42, in _check_same_shape
    raise RuntimeError(
RuntimeError: Predictions and targets are expected to have the same shape, but got torch.Size([16384]) and torch.Size([16384, 3]).

[MrTornado24]

Hi, it seems that the image hasn't been preprocessed, and the depth image has been incorrectly loaded from the RGB image path, leading to mismatched dimensions. Please run preprocess_image.py to obtain the depth and normal images.

[Iliceth]

@MrTornado24 I have the same error.

RuntimeError: Predictions and targets are expected to have the same shape, but got torch.Size([3638]) and torch.Size([3638, 4]).

I tried first with the provided mushroom example and all went fine. Now I try with my own image and it doesn't work.

The preprocess_image.py has been run. The first image is my input, the three following images are what the code has created. But when I then proceed, the size error occurs.

[Iliceth]

@nullx1337 Did you find a way around this eventually?

[Iliceth]

Update, I found what was wrong: I used my original image as the image path throughout, but after preprocessing it, the image path should point to one of the three resulting files, the xxxxxx_rgba.png. A small, but costly mistake, did look passed that the whole time, sorry. In case of your issue, it seems yours points to "/root/DreamCraft3D/inputimges/falcon.jpg", while it should be "/root/DreamCraft3D/inputimges/falcon_rgba.jpg", but you might have found out yourself earlier.