How to Render Replica Dataset Scenes with habitat-viewer?

[Liu-Jinxin]

Issue: Problems Rendering Replica Dataset Scenes with habitat-viewer

Description

I am trying to use Habitat Sim to visualize scenes from the Replica dataset but encountering rendering issues. Specifically, the visuals appear incorrect when executing the command for the Replica dataset, while rendering works as expected for the ReplicaCAD dataset.

Commands and Results

Replica Dataset Issue Command:

habitat-viewer --enable-physics --dataset data/Replica-Dataset/data/replica.scene_dataset_config.json --room_1

or

python examples/viewer.py --dataset data/Replica-Dataset/data/replica.scene_dataset_config.json --scene room_0

The scene does not render as expected.

The output

Renderer: NVIDIA GeForce RTX 4090/PCIe/SSE2 by NVIDIA Corporation
OpenGL version: 4.6.0 NVIDIA 525.147.05
Using optional features:
    GL_ARB_vertex_array_object
    GL_ARB_separate_shader_objects
    GL_ARB_robustness
    GL_ARB_texture_storage
    GL_ARB_texture_view
    GL_ARB_framebuffer_no_attachments
    GL_ARB_invalidate_subdata
    GL_ARB_texture_storage_multisample
    GL_ARB_multi_bind
    GL_ARB_direct_state_access
    GL_ARB_get_texture_sub_image
    GL_ARB_texture_filter_anisotropic
    GL_KHR_debug
    GL_KHR_parallel_shader_compile
    GL_NV_depth_buffer_float
Using driver workarounds:
    no-forward-compatible-core-context
    nv-egl-incorrect-gl11-function-pointers
    no-layout-qualifiers-on-old-glsl
    nv-zero-context-profile-mask
    nv-implementation-color-read-format-dsa-broken
    nv-cubemap-inconsistent-compressed-image-size
    nv-cubemap-broken-full-compressed-image-query
    nv-compressed-block-size-in-bits
PluginManager::Manager: duplicate static plugin StbImageImporter, ignoring
PluginManager::Manager: duplicate static plugin GltfImporter, ignoring
PluginManager::Manager: duplicate static plugin BasisImporter, ignoring
PluginManager::Manager: duplicate static plugin AssimpImporter, ignoring
PluginManager::Manager: duplicate static plugin AnySceneImporter, ignoring
PluginManager::Manager: duplicate static plugin AnyImageImporter, ignoring
[Sim] 
=====================================================
Welcome to the Habitat-sim Python Viewer application!
=====================================================
Mouse Functions ('m' to toggle mode):
----------------
In LOOK mode (default):
    LEFT:
        Click and drag to rotate the agent and look up/down.
    WHEEL:
        Modify orthographic camera zoom/perspective camera FOV (+SHIFT for fine grained control)

In GRAB mode (with 'enable-physics'):
    LEFT:
        Click and drag to pickup and move an object with a point-to-point constraint (e.g. ball joint).
    RIGHT:
        Click and drag to pickup and move an object with a fixed frame constraint.
    WHEEL (with picked object):
        default - Pull gripped object closer or push it away.
        (+ALT) rotate object fixed constraint frame (yaw)
        (+CTRL) rotate object fixed constraint frame (pitch)
        (+ALT+CTRL) rotate object fixed constraint frame (roll)
        (+SHIFT) amplify scroll magnitude


Key Commands:
-------------
    esc:        Exit the application.
    'h':        Display this help message.
    'm':        Cycle mouse interaction modes.

    Agent Controls:
    'wasd':     Move the agent's body forward/backward and left/right.
    'zx':       Move the agent's body up/down.
    arrow keys: Turn the agent's body left/right and camera look up/down.

    Utilities:
    'r':        Reset the simulator with the most recently loaded scene.
    'n':        Show/hide NavMesh wireframe.
                (+SHIFT) Recompute NavMesh with default settings.
                (+ALT) Re-sample the agent(camera)'s position and orientation from the NavMesh.
    ',':        Render a Bullet collision shape debug wireframe overlay (white=active, green=sleeping, blue=wants sleeping, red=can't sleep).
    'c':        Run a discrete collision detection pass and render a debug wireframe overlay showing active contact points and normals (yellow=fixed length normals, red=collision distances).
                (+SHIFT) Toggle the contact point debug render overlay on/off.
    'j'         Toggle Semantic visualization bounds (currently only Semantic Region annotations)

    Object Interactions:
    SPACE:      Toggle physics simulation on/off.
    '.':        Take a single simulation step if not simulating continuously.
    'v':        (physics) Invert gravity.
    't':        Load URDF from filepath
                (+SHIFT) quick re-load the previously specified URDF
                (+ALT) load the URDF with fixed base
=====================================================

replica.scene_dataset_config.json is as follows:

{
  "stages": {
    "paths": {
      ".json": [
        "*_[0-9]/habitat/replica_stage.stage_config.json"
      ]
    },
    "default_attributes": {
      "shader_type": "flat",
      "up": [
        0,
        0,
        1
      ],
      "front": [
        0,
        1,
        0
      ],
      "origin": [
        0,
        0,
        0
      ]
    }
  },
  "objects": {},
  "light_setups": {},
  "scene_instances": {
    "default_attributes": {
      "default_lighting": "no_lights"
    }
  }
}

For the ReplicaCAD dataset, I used:

habitat-viewer --enable-physics --dataset data/replica_cad/replicaCAD.scene_dataset_config.json -- apt_1

This renders the scene correctly.

[aclegg3]

We have removed support for ptex shader which was handling Replica scenes specifically. I imagine this is related to that deprecation.
I'm a bit surprised to see they are loading with what looks like base metallic color and no texture, though.
@jturner65 any thoughts on this?

[Liu-Jinxin]

We have removed support for ptex shader which was handling Replica scenes specifically. I imagine this is related to that deprecation. I'm a bit surprised to see they are loading with what looks like base metallic color and no texture, though. @jturner65 any thoughts on this?

Thank you for the insight on the deprecation of ptex shader support, given this context, could you please inform me about the last version or branch of Habitat Sim that fully supported normal rendering for the Replica dataset with the ptex shaders? I'm looking to work with the Replica dataset.

Thank you for your assistance.