3D Pano Inpainting: Building a VR Environment from a Single Input Panorama
Shivam Asija, Edward Du, Nam Nguyen, Stefanie Zollmann, Jonathan Ventura
2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)
Abstract: Creating 360-degree 3D content is challenging because it requires either a multi-camera rig or a collection of many images taken from different perspectives. Our approach aims to generate a 360◦ VR scene from a single panoramic image using a learning-based inpainting method adapted for panoramic content. We introduce a pipeline capable of transforming an equirectangular panoramic RGB image into a complete 360◦ 3D virtual reality scene represented as a textured mesh, which is easily rendered on a VR headset using standard graphics rendering pipelines. We qualitatively evaluate our results on a synthetic dataset consisting of 360 panoramas in indoor scenes.
To get a local copy of the project up and running on your machine, follow these simple steps:
Before attempting to build this project, make sure you have Docker Engine installed on your machine.
Clone the repository
git clone https://github.com/jonathanventura/3d-pano-inpainting.git
cd 3d-pano-inpainting
- Place your panoramic images in the
datadirectory. The images should be in equirectangular format and the width of each image should be twice the height. You will also need to update the filenames indata/data.txt. - Execute the depth estimation step
sh depth-estimation/run_360monodepth.sh
- Execute the meshing and inpainting step
sh inpainting/run_3d_photo_inpainting.sh
- The results are placed in the
resultsdirectory. - Re-scale the mesh according to the known height of the camera off of the ground:
python mesh/estimate_scale_histogram.py <input mesh> <output mesh> [--camera_height <height>]
The inpainting code will resize the images to a fixed maximum side length determined by the longer_side_len parameter in inpainting/argument.yml.
Copy the resulting .glb file into docs/assets and update the path in docs/renderer.html accordingly.
To start the renderer you can use
python -m http.server
and then navigate to localhost:8000/renderer.html.
This project is distributed under the terms of the MIT license. See LICENSE for details and more information.