ScanNet or SUNRGBD datasets with BlenderProc

[wakasu]

Describe your feature request

I have used for creating datasets for COCO instance segmentation and Linemod. I really appreciate the work that the developers have done with this tool. I am working on a monocular 3D object detection project and I need to use either ScanNet or SUNRGBD datasets. However, it seems like these datasets are not supported by BlenderProc.

I would like to know if anyone has worked on this problem before and if they could share their approach or code with me. If not, I might start working on it myself if I have some spare time.

Describe a possible solution

No response

[cornerfarmer]

Hey @wakasu,

how exactly do you want to use these datasets? Both provide rgb-d sequences which are not directly suitable for blenderproc. Or do you want to load the provided 3D reconstructions? If yes, then this should be possible by using the bproc.loader.load_obj(path) method, where path points to e.g. the ply files from ScanNet.

[wakasu]

Hey @cornerfarmer

Currently, I would like to have the capability to export the dataset in SUNRGBD or ScanNet format. As for input data, I'm flexible and can use either my own .blend scenes or, if this project progresses further, I can also work on 3D reconstruction using indoor point cloud datasets such as S3DIS.

[cornerfarmer]

Ah okay, it was not clear to me that you want to export data in that formats. That should be easy to do. The blenderproc renderer returns a list of numpy arrays for each image, so you can export them in any format you want. SUNRGBD only seems to be a specific folder structure containing the explicit images. So here you just need to export the numpy arrays as images into the correct path. For ScanNet you need to write the images in a custom .sens format. Thats a bit more difficult, as you need to put them into a struct complementary to how the .sens readings works (see https://github.com/ScanNet/ScanNet/blob/master/SensReader/python/SensorData.py)

[wakasu]

Wow, thank you for the message and for providing the link to the code for reference. I will begin working on this shortly.