Questions about ICON+Keypoint

[Jarvisss]

ICON/lib/net/HGPIFuNet.py

Lines 301 to 305 in 8919d96

	if self.prior_type == "keypoint":
	kpt_feat = self.sp_encoder.forward(
	cxyz=xyz.permute(0, 2, 1).contiguous(),
	kptxyz=self.smpl_feat_dict["smpl_joint"],
	)

I find the implementation of keypoint+ICON projects query points to camera space, but smpl joints are not projected.
But in original keypointNerf paper, the spatial encoding is calculated by the projected keypoints.

ICON/lib/net/HGPIFuNet.py

Lines 333 to 340 in 8919d96

	if self.prior_type == "keypoint":
	if "vis" in self.smpl_feats:
	point_local_feat = feat_select(self.index(im_feat, xy), smpl_feat[:, [-1], :])
	point_feat_list = [point_local_feat, kpt_feat, smpl_feat[:, :-1, :]]
	else:
	point_local_feat = self.index(im_feat, xy)
	point_feat_list = [point_local_feat, kpt_feat, smpl_feat[:, :, :]]

And SMPL features are still used in the keypoint version, is this what it supposed to be?

[Jarvisss]

@YuliangXiu

[YuliangXiu]

smpl_joints should be projected into camera space, you could visualize the keypoints when running infer.py