KeyError: 'sk_x' when using fox data from instant-ngp

[alexwang2013]

I want check the result by using the fox data from which is here: fox

I executed the following command:

EXPERIMENT=fox
GROUP=fox
NAME=fox
CONFIG=projects/neuralangelo/configs/custom/${EXPERIMENT}.yaml
GPUS=1  # use >1 for multi-GPU training!
torchrun --nproc_per_node=${GPUS} train.py \
    --logdir=logs/${GROUP}/${NAME} \
    --config=${CONFIG} \
    --show_pbar

output is the following

(Setting affinity with NVML failed, skipping...)
[W Utils.hpp:133] Warning: Environment variable NCCL_ASYNC_ERROR_HANDLING is deprecated; use TORCH_NCCL_ASYNC_ERROR_HANDLING instead (function getCvarInt)
Training with 1 GPUs.
Using random seed 0
Make folder logs/fox/fox
* checkpoint:
   * save_epoch: 9999999999
   * save_iter: 20000
   * save_latest_iter: 9999999999
   * save_period: 9999999999
   * strict_resume: True
* cudnn:
   * benchmark: True
   * deterministic: False
* data:
   * name: dummy
   * num_images: None
   * num_workers: 4
   * preload: True
   * readjust:
      * center: [0.0, 0.0, 0.0]
      * scale: 1.0
   * root: /mnt/d/wang_back_data/neuralangelo/fox
   * train:
      * batch_size: 2
      * image_size: [1920, 1080]
      * subset: None
   * type: projects.neuralangelo.data
   * use_multi_epoch_loader: True
   * val:
      * batch_size: 2
      * image_size: [533, 300]
      * max_viz_samples: 16
      * subset: 4
* image_save_iter: 9999999999
* inference_args:
* local_rank: 0
* logdir: logs/fox/fox
* logging_iter: 9999999999999
* max_epoch: 9999999999
* max_iter: 500000
* metrics_epoch: None
* metrics_iter: None
* model:
   * appear_embed:
      * dim: 8
      * enabled: False
   * background:
      * enabled: True
      * encoding:
         * levels: 10
         * type: fourier
      * encoding_view:
         * levels: 3
         * type: spherical
      * mlp:
         * activ: relu
         * activ_density: softplus
         * activ_density_params:
         * activ_params:
         * hidden_dim: 256
         * hidden_dim_rgb: 128
         * num_layers: 8
         * num_layers_rgb: 2
         * skip: [4]
         * skip_rgb: []
      * view_dep: True
      * white: False
   * object:
      * rgb:
         * encoding_view:
            * levels: 3
            * type: spherical
         * mlp:
            * activ: relu_
            * activ_params:
            * hidden_dim: 256
            * num_layers: 4
            * skip: []
            * weight_norm: True
         * mode: idr
      * s_var:
         * anneal_end: 0.1
         * init_val: 3.0
      * sdf:
         * encoding:
            * coarse2fine:
               * enabled: True
               * init_active_level: 4
               * step: 5000
            * hashgrid:
               * dict_size: 22
               * dim: 8
               * max_logres: 11
               * min_logres: 5
               * range: [-2, 2]
            * levels: 16
            * type: hashgrid
         * gradient:
            * mode: numerical
            * taps: 4
         * mlp:
            * activ: softplus
            * activ_params:
               * beta: 100
            * geometric_init: True
            * hidden_dim: 256
            * inside_out: False
            * num_layers: 1
            * out_bias: 0.5
            * skip: []
            * weight_norm: True
   * render:
      * num_sample_hierarchy: 4
      * num_samples:
         * background: 32
         * coarse: 64
         * fine: 16
      * rand_rays: 512
      * stratified: True
   * type: projects.neuralangelo.model
* nvtx_profile: False
* optim:
   * fused_opt: False
   * params:
      * lr: 0.001
      * weight_decay: 0.01
   * sched:
      * gamma: 10.0
      * iteration_mode: True
      * step_size: 9999999999
      * two_steps: [300000, 400000]
      * type: two_steps_with_warmup
      * warm_up_end: 5000
   * type: AdamW
* pretrained_weight: None
* source_filename: projects/neuralangelo/configs/custom/fox.yaml
* speed_benchmark: False
* test_data:
   * name: dummy
   * num_workers: 0
   * test:
      * batch_size: 1
      * is_lmdb: False
      * roots: None
   * type: imaginaire.datasets.images
* timeout_period: 9999999
* trainer:
   * amp_config:
      * backoff_factor: 0.5
      * enabled: False
      * growth_factor: 2.0
      * growth_interval: 2000
      * init_scale: 65536.0
   * ddp_config:
      * find_unused_parameters: False
      * static_graph: True
   * depth_vis_scale: 0.5
   * ema_config:
      * beta: 0.9999
      * enabled: False
      * load_ema_checkpoint: False
      * start_iteration: 0
   * grad_accum_iter: 1
   * image_to_tensorboard: False
   * init:
      * gain: None
      * type: none
   * loss_weight:
      * curvature: 0.0005
      * eikonal: 0.1
      * render: 1.0
   * type: projects.neuralangelo.trainer
* validation_iter: 5000
* wandb_image_iter: 10000
* wandb_scalar_iter: 100
cudnn benchmark: True
cudnn deterministic: False
Setup trainer.
Using random seed 0
/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/site-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
  warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
model parameter count: 366,702,732
Initialize model weights using type: none, gain: None
Using random seed 0
[rank0]:[W Utils.hpp:106] Warning: Environment variable NCCL_ASYNC_ERROR_HANDLING is deprecated; use TORCH_NCCL_ASYNC_ERROR_HANDLING instead (function getCvarString)
Allow TensorFloat32 operations on supported devices
preloading cameras (train):   0%|                                                                                                                                                                                           | 0/50 [00:00<?, ?it/s]Exception in thread Thread-6Exception in thread Thread-7Exception in thread :
Thread-8Exception in thread Traceback (most recent call last):
Thread-9:
Traceback (most recent call last):
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 932, in _bootstrap_inner
:
:
Traceback (most recent call last):
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 932, in _bootstrap_inner
Traceback (most recent call last):
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 932, in _bootstrap_inner
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 870, in run
    self.run()
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 870, in run
    self.run()
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 870, in run
    self.run()
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/nerf/datasets/base.py", line 29, in _preload_worker
    self._target(*self._args, **self._kwargs)
  File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/nerf/datasets/base.py", line 29, in _preload_worker
    self._target(*self._args, **self._kwargs)
  File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/nerf/datasets/base.py", line 29, in _preload_worker
    self._target(*self._args, **self._kwargs)
  File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/nerf/datasets/base.py", line 29, in _preload_worker
    data_list[idx] = load_func(idx)
  File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/neuralangelo/data.py", line 103, in get_camera
    data_list[idx] = load_func(idx)
      File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/neuralangelo/data.py", line 103, in get_camera
data_list[idx] = load_func(idx)
  File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/neuralangelo/data.py", line 103, in get_camera
    data_list[idx] = load_func(idx)
  File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/neuralangelo/data.py", line 103, in get_camera
    intr = torch.tensor([[self.meta["fl_x"], self.meta["sk_x"], self.meta["cx"]],
intr = torch.tensor([[self.meta["fl_x"], self.meta["sk_x"], self.meta["cx"]],KeyError
: 'sk_x'KeyError
: 'sk_x'
    intr = torch.tensor([[self.meta["fl_x"], self.meta["sk_x"], self.meta["cx"]],
KeyError: 'sk_x'
    intr = torch.tensor([[self.meta["fl_x"], self.meta["sk_x"], self.meta["cx"]],
KeyError: 'sk_x'

Environment

• Windows 10 64 bit
• WSL2 + Ubuntu
• Gerforce RTX 4090
• CUDO 12.1

By the way
for the data fox
there are some frames in the transform.json file, but there is no relative jpg file in the folder images. this will cause exceptions, so I removed the unused frames from transform.json. For e.g. image 0016.jpg is not in folder images, I removed the following frame from transform.json.

{
      "file_path": "images/0016.jpg",
      "sharpness": 15.184317835374785,
      "transform_matrix": [
        [
          0.34747494861304573,
          0.053380418010164174,
          0.9361685961577728,
          5.639885048014769
        ],
        [
          0.9356011023082821,
          0.04685455607320249,
          -0.3499359634577667,
          -2.679224620930034
        ],
        [
          -0.06254348426497747,
          0.9974744008232991,
          -0.033661993185310046,
          -0.6582774295652742
        ],
        [
          0.0,
          0.0,
          0.0,
          1.0
        ]
      ]
    },

[Bombaninha]

The transforms.json file format for Instant-NGP and Neuralangelo differs in some aspects.
Apparently, your file does not have information about skew.
Ideally, you should go through the preprocessing step of Neuralangelo to update this file in the correct format.
This can be seen in: neuralangelo/projects/neuralangelo/scripts/convert_data_to_json.py