How to get viewmat from c2w and intrinsic?

[baoachun]

I'm currently trying to implement 3D rendering using this library, but I'm confused about how to set the viewmat. I have the c2w matrix and the intrinsic matrix . According to the official documentation, I initially thought that taking the inverse of c2w would give me the viewmat. However, the calculation method here does not directly involve taking the inverse of c2w. Can you provide a more detailed example to explain the process of calculating viewmat?

https://github.com/Harr7y/nerfstudio/blob/37948b360b108f8e30b2122d8eab220bf1443965/nerfstudio/models/splatfacto.py#L682

[maturk]

@baoachun the view matrix is the world to camera (w2c) transform. So if you have a c2w you can invert this to obtain a w2c. The method linked above is actually the same as taking the inverse of a 3x4 camera matrix. Note the c2w has both the rotation and translation component appended, which is the convention often used. Here the inverse is just computed manually: the transpose of the SO(3) rotation matrix component is taken probably for speed and numerical efficiency reasons (instead of calling inv(c2w[:3,;3)). The inverse of the translation component also follows from math.

[baoachun]

@maturk Hi, thank you for your reply. However, I'm now encountering another issue. The original 3DGS can converge, but it fails to converge when using gsplat. I saved the rendering results and found that they are all black. I suspect that the input parameters are not set correctly. Could you please help me check if the following settings are correct?

3DGS

class MiniCam:
    def __init__(self, c2w, intrinsic, height, width):
        self.width = width
        self.height = height
        self.FoVx = 2 * math.atan2(width, 2 * intrinsic[0, 0].item() * width)
        self.FoVy = 2 * math.atan2(height, 2 * intrinsic[1, 1].item() * height)
        self.znear = 0.01
        self.zfar = 100.0
        
        R = c2w[:3, :3]
        T = c2w[:3, 3]
        R_inv = R.T
        T_inv = -R_inv @ T
        w2c = torch.eye(4, device=R.device, dtype=R.dtype)
        w2c[:3, :3] = R_inv
        w2c[:3, 3] = T_inv
        
        self.world_view_transform = w2c.transpose(0, 1)
        proj_matrix = getProjectionMatrix(self.znear, self.zfar, self.FoVx, self.FoVy)
        self.projection_matrix = proj_matrix.transpose(0, 1).to(c2w)
        self.full_proj_transform = self.world_view_transform @ self.projection_matrix
        self.camera_center = c2w[:3, 3]
  
def getProjectionMatrix(znear, zfar, fovx, fovy):
    tanHalfFovY = math.tan((fovy * 0.5))
    tanHalfFovX = math.tan((fovx * 0.5))

    P = torch.zeros(4, 4)
    z_sign = 1.0
    P[0, 0] = 1 / tanHalfFovX
    P[1, 1] = 1 / tanHalfFovY
    P[3, 2] = z_sign
    P[2, 2] = z_sign * zfar / (zfar - znear)
    P[2, 3] = -(zfar * znear) / (zfar - znear)
    return P

gsplat

class MiniCamera():
    def __init__(self, c2w, intrinsic, height, width):
        self.fx = intrinsic[0, 0].item() * width
        self.fy = intrinsic[1, 1].item() * height
        self.cx = intrinsic[0, 2].item() * width
        self.cy = intrinsic[1, 2].item() * height
        self.height = height
        self.width = width
        
        R = c2w[:3, :3]
        T = c2w[:3, 3]
        R_inv = R.T
        T_inv = -R_inv @ T
        viewmat = torch.eye(4, device=R.device, dtype=R.dtype)
        viewmat[:3, :3] = R_inv
        viewmat[:3, 3] = T_inv
        self.viewmat = viewmat

def gsplat_render(camera, points, rgb, opacity, scale, rotation, background):
    block_width = 16
    proj_params = {'means3d': points, 
                   'scales': scale, 
                   'glob_scale': 1.0, 
                   'quats': rotation,
                   'viewmat': camera.viewmat,
                   'fx': camera.fx, 
                   'fy': camera.fy, 
                   'cx': camera.cx, 
                   'cy': camera.cy, 
                   'img_height': int(camera.height), 
                   'img_width': int(camera.width), 
                   'block_width': block_width}
    
    xys, depths, radii, conics, _, num_tiles_hit, _ = project_gaussians(**proj_params)
    ras_params = {'xys': xys,
                  'depths': depths,
                  'radii': radii,
                  'conics': conics,
                  'num_tiles_hit': num_tiles_hit,
                  'colors': rgb,
                  'opacity': opacity,
                  'img_height': int(camera.height),
                  'img_width': int(camera.width),
                  'block_width': block_width,
                  'background': background}
    image = rasterize_gaussians(**ras_params)[..., :3]
    return image.permute(2, 0, 1)

[ivanpuhachov]

@baoachun given the same .ply file do you have the same rendering results from 3dgs and gsplat? IIRC, 3DGS does exponential activation on scales while gsplat does not (by default), axis convention might also differ

[baoachun]

@ivanpuhachov The two methods are identical in all settings except for the rendering method. I suspect there might be an issue with the camera parameter settings, so I'd like to know how to obtain the relevant input parameters for gsplat from the c2w and intrinsic matrices.

[EnderTime]

I meet the similiar problem probably. I've checked my input model on another gaussian splatting implement and got the expected rendering result. But when I try on gsplat, it returns blank image with only bg color.

I check the returns of project_gaussians and find that conics are extremely small value, maybe caused the gaussian be unexpectedly culled.

I upload some code in my repository to compare the result between gsplat and my temporary implement. My installed gsplat version is 0.1.11.
my code

Appreciation for your attention!

[ivanpuhachov]

I meet the similiar problem probably. I've checked my input model on another gaussian splatting implement and got the expected rendering result. But when I try on gsplat, it returns blank image with only bg color.

I check the returns of project_gaussians and find that conics are extremely small value, maybe caused the gaussian be unexpectedly culled.

I upload some code in my repository to compare the result between gsplat and my temporary implement. My installed gsplat version is 0.1.11. my code

Appreciation for your attention!

I've run your code and I have a few comments:

• your gaussian appears to be outside of view frustum and gsplat skips the computations
• It seems you are missing projection matrix to transform camera coordinates to ND (see https://arxiv.org/abs/2312.02121, (1) and (2) ), replace your intrinsic
• conic are the inverse of sigma, computed in image pixel space

After these fixes the code gsplat and your manual check output the same values

[EnderTime]

I meet the similiar problem probably. I've checked my input model on another gaussian splatting implement and got the expected rendering result. But when I try on gsplat, it returns blank image with only bg color.
I check the returns of project_gaussians and find that conics are extremely small value, maybe caused the gaussian be unexpectedly culled.
I upload some code in my repository to compare the result between gsplat and my temporary implement. My installed gsplat version is 0.1.11. my code
Appreciation for your attention!

I've run your code and I have a few comments:

• your gaussian appears to be outside of view frustum and gsplat skips the computations
• It seems you are missing projection matrix to transform camera coordinates to ND (see https://arxiv.org/abs/2312.02121, (1) and (2) ), replace your intrinsic
• conic are the inverse of sigma, computed in image pixel space

After these fixes the code gsplat and your manual check output the same values

I fix my code and find that in my pre-trained model no gaussian is inside the view frustum. Maybe it's because of my incorrect data loading.

Thanks for your comments!

[baoachun]

@ivanpuhachov Hello, based on the discussion above, I've realized that my model also suffers from the issue of having no Gaussian points within the current view. How can I debug this situation? Is there a way to visualize the view under the current camera parameters? Thank you.