Deferred rendering for pytorch backend

[jjennings955]

Hello, deferred rendering with the pytorch backend doesn't seem to work. I assume this is known, but it's not clear from the documentation or code that this is the case.

Deferred rendering is currently only implemented in the Scene3D class, and it's using some numpy functions on pytorch tensors. I expect it would have to be overriden in Scene3DPytorch with numpy stuff replaced with pytorch equivalents. I tried a bit myself but didn't quite figure it out, and was short on time.

For now, it might be a good idea to override render_deferred in Scene3DPytorch to raise an NotImplementedError, so it's clear that it's not supposed to be working yet.

I think the same would be true for render_depth, as well as render_deferred/render_depth in the tensorflow versions.

Here's a minimal example and the current error message.

import copy
import torch
import numpy as np
import trimesh
from deodr.pytorch import Scene3DPytorch, CameraPytorch
from deodr.pytorch.triangulated_mesh_pytorch import ColoredTriMeshPytorch as ColoredTriMesh

def get_camera(camera_center, width, height, focal=None):
    if focal is None:
        focal = 2 * width
    rot = np.array([[1, 0, 0], [0, -1, 0], [0, 0, -1]])
    trans = -rot.T.dot(camera_center)
    intrinsic = np.array(
        [[focal, 0, width / 2], [0, focal, height / 2], [0, 0, 1]]
    )
    extrinsic = np.column_stack((rot, trans))
    return CameraPytorch(
        extrinsic=extrinsic,
        intrinsic=intrinsic,
        width=width,
        height=height
    )

mesh_trimesh = trimesh.creation.icosphere(subdivisions=2, color=[0.8, 0.0, 0.0])
v_c = trimesh.visual.color.face_to_vertex_color(mesh_trimesh, mesh_trimesh.visual.face_colors)
mesh_trimesh.visual.vertex_colors = v_c

default_color = np.array([1.0, 1.0, 1.0])
default_light = {
    "directional": -np.array([0.0, 0.0, 0.0]),
    "ambient": np.array([1.0]),
}

height = width = 256
light_directional = torch.tensor(copy.copy(default_light["directional"]))
light_ambient = torch.tensor(copy.copy(default_light["ambient"]))
vertices_color = torch.rand(mesh_trimesh.visual.vertex_colors.shape)

camera_center = torch.tensor([0.0, 0.0, 15.0])
camera = get_camera(camera_center, width, height)

scene = Scene3DPytorch()
scene.set_light(light_directional=light_directional, light_ambient=light_ambient)
background_color = np.array([0.0, 0., 0., 1.0])
scene.set_background_color(background_color)
mesh = ColoredTriMesh(vertices=torch.tensor(mesh_trimesh.vertices, dtype=torch.float64, requires_grad=True),
                        faces=torch.tensor(mesh_trimesh.faces, dtype=torch.int64),
                        colors=vertices_color)

scene.set_mesh(mesh)
scene.sigma = 0
channels = scene.render_deferred(camera)

This is the error message

Traceback (most recent call last):
  File "D:\zzzzz\scratch\deferred_example.py", line 52, in <module>
    channels = scene.render_deferred(camera)
  File "C:\zzzz\lib\site-packages\deodr\differentiable_renderer.py", line 1181, in render_deferred
    vertices_luminosity = self.compute_vertices_luminosity()
  File "C:\zzz\lib\site-packages\deodr\differentiable_renderer.py", line 875, in compute_vertices_luminosity
    0, -np.sum(self.mesh.vertex_normals * self.light_directional, axis=1)
  File "<__array_function__ internals>", line 200, in sum
  File "C:\zzz\lib\site-packages\numpy\core\fromnumeric.py", line 2324, in sum
    return _wrapreduction(a, np.add, 'sum', axis, dtype, out, keepdims=keepdims,
  File "C:\zzz\lib\site-packages\numpy\core\fromnumeric.py", line 84, in _wrapreduction
    return reduction(axis=axis, out=out, **passkwargs)
TypeError: sum() received an invalid combination of arguments - got (axis=int, out=NoneType, ), but expected one of:
 * (*, torch.dtype dtype)
      didn't match because some of the keywords were incorrect: axis, out
 * (tuple of ints dim, bool keepdim, *, torch.dtype dtype)
 * (tuple of names dim, bool keepdim, *, torch.dtype dtype)

Thanks

[jjennings955]

Also appears the pytorch backend only supports vertex colors (no textures).