This repository implements frame transformation methods to interface and use the Apple Hypersim Evermotion data set with multi view geometry applications. For example this package can be used to label pixel correspondences for self-supervised key point detection and description as in KP2D
Pixel correspondences between two frames of the same scene can be found through the pixel world position in the source frame and the target camera pose and camera intrinsic calibration.
warp():
Pixel that are visible in the source frame but occluded in the target frame can be detected automatically.
Reflecting and transparent surface can be masked using the diffuse reflectance information of the Hypersim dataset.
Install the hypersim_multiview toolbox with
conda develop $HOME/hypersim_multiview
or
pip3 install -e $HOME/hypersim_multiview
The following files are required to compute the projection, the occlusion detection, and the reflectance detection:
|_detail
|--*
|images
|--scene_cam_CC_final_preview
|----frame.FFFF.color.jpg
|----frame.FFFF.diffuse_reflectance.jpg
|--scene_cam_CC_geometry.hdf5
|----frame.FFFF.position.hdf5
The minimum dataset is still 500 Gb, thus we have to download it through the official source. Use 99991's alternative downloader to download the reduced dataset:
python3 ./download.py -d /diretory/to/download/to --contains _detail --silent
python3 ./download.py -d /diretory/to/download/to --contains .color.jpg --contains final_preview --silent
python3 ./download.py -d /diretory/to/download/to --contains .diffuse_reflectance.jpg --silent
python3 ./download.py -d /diretory/to/download/to --contains .position.hdf5 --silent
The download script will first list all files that it is skipping. This may take 15 minutes. Then it starts downloading.
This work was created within the 3D Vision class student project Viewpoint Adaptation in a Synthetic Environment with Shengqu Cai (@HexagonPrime), Menelaos Kanakis (@menelaoskanakis) and Mihai Dusmanu (@mihaidusmanu) at ETH Zurich.