the function: grid_sample in pytorch

[scholltan]

hi，did you try the new function grid_sample in pytorch to implement the same effect?

[alwynmathew]

No, I have been going through grid_sample recently. I'll update you if I could find a way, please update me if you could crack it before me.

[scholltan]

I tried grid_sample and it works for me as follows:

def image_warp(img, depth, padding_mode='zeros'):
    
    # img: the source image (where to sample pixels) -- [B, 3, H, W]
    # depth: depth map of the target image -- [B, H, W]
    # Returns: Source image warped to the target image
   

    b, h, w = depth.size() 
    i_range = torch.autograd.Variable(torch.arange(0, h).view(1, h, 1).expand(1,h,w),requires_grad = False)# .type_as(depth)  # [1, H, W]  copy 0-height for w times : y coord
    j_range = torch.autograd.Variable(torch.arange(0, w).view(1, 1, w).expand(1,h,w),requires_grad = False)# .type_as(depth)  # [1, H, W]  copy 0-width for h times  : x coord

    pixel_coords = torch.stack((j_range, i_range), dim=1).float().cuda()  # [1, 2, H, W]
    batch_pixel_coords = pixel_coords[:,:,:,:].expand(b,2,h,w).contiguous().view(b, 2, -1)  # [B, 2, H*W]

    X = batch_pixel_coords[:, 0]  + depth.contiguous().view(b,-1) # [B, H*W]
    Y = batch_pixel_coords[:, 1]

    X_norm = 2 * X /(w-1) - 1  # FloatTensor  Normalized, -1 if on extreme left, 1 if on extreme right (x = w-1) [B, H*W]
    Y_norm = 2 * Y /(h-1) - 1  # Idem [B, H*W]

    pixel_coords = torch.stack([X_norm, Y_norm], dim=2)  # [B, H*W, 2]
    pixel_coords = pixel_coords.view(b,h,w,2)  # [B, H, W, 2]

    projected_img = torch.nn.functional.grid_sample(img, pixel_coords, padding_mode=padding_mode)

    return projected_img

[alwynmathew]

@scholltan how to make a field flow(grid) when the input image is 4D?

According to doc:
Input : N x C x ID x IH x IW
Grid: N x OD x OH x OW x 3

[scholltan]

@alwynmathew according to my understanding, pixel_coords need to be the shape of # [B, H, W, 3], that Z_norm is added into just like X_norm and Y_norm. I did not try it.

[alwynmathew]

Instead of making the grid in range (0, h) then converting to (-1,+1):

i_range = torch.autograd.Variable(torch.arange(0, h).view(1, h, 1).expand(1,h,w),requires_grad = False)# .type_as(depth)  # [1, H, W]  copy 0-height for w times : y coord
j_range = torch.autograd.Variable(torch.arange(0, w).view(1, 1, w).expand(1,h,w),requires_grad = False)# .type_as(depth)  # [1, H, W]  copy 0-width for h times  : x coord

why cant we make the grid in (-1,+1) range directly like this:

i_range = Variable(torch.linspace(-1.0,1.0,h).view(1, h, 1).expand(1,h,w),requires_grad = False)
j_range = Variable(torch.linspace(-1.0,1.0,w).view(1, 1, w).expand(1,h,w),requires_grad = False)

[rickgroen]

In line 14 of bilinear.py there might still be an error.
X = batch_pixel_coords[:, 0, :] + depth.contiguous().view(b,-1) # [B, H*W]
Here we add the disparities, which are in the range [0.0,1.0] to pixel coordinates in the range [-1.0,1.0]. We are thus off by a factor 2.

Possibly it could be corrected to:
X = batch_pixel_coords[:, 0, :] + depth.contiguous().view(b, -1) * 2

I have tried the above and now get similar results (at least visually). Numerically, I no longer get highly underestimated depth values.