Camera rendering in MuJoCo - How to take a picture from a simulated camera and still keeping the GUI active

[vmstavens]

Hello there,

I'm a research assistant and I'm trying to take pictures of my MuJoCo simulation using a simulated camera.

Currently, I have my main rendering loop running:

main render loop

def run(self) -> None:
  """
  Method for running the MuJoCo viewer main loop.
  
  Returns:
  - None.
  """
  with mj.viewer.launch_passive(model=self._model, data=self._data, key_callback=self.keyboard_callback) as viewer:
  
	  # load initial state
	  load_mj_state(self._model, self._data, "home", self._lock)
	  mj.mj_step(self._model, self._data)
  
	  # Toggle site frame visualization.
	  if self._args.show_site_frames:
		  viewer.opt.frame = mj.mjtFrame.mjFRAME_SITE
  
	  while viewer.is_running():
		  step_start = time.time()
  
		  # Step the simulation.
		  mj.mj_step(self._model, self._data)
  
		  viewer.sync()
		  time_until_next_step = self.dt - (time.time() - step_start)
		  if time_until_next_step > 0:
			  time.sleep(time_until_next_step)

while I have my keyboard_callback function defined as

`keyboard_callback`

def keyboard_callback(self, key) -> None:
  """
  Method for the keyboard callback. This method should not be called explicitly.
  
  Parameters:
  - key: Keyboard input.
  
  Returns:
  - None.
  """
  if key == glfw.KEY_COMMA:
	  print(self.cam.image)

When I call the cam.image attribute, the following function is run

get image attribute

@property
def image(self) -> np.ndarray:
  """Return the captured RGB image."""
  self._renderer.update_scene(self._data, camera=self.name)
  self._img = self._renderer.render()
  self._renderer.update_scene(self._data)
  return self._img

The self._renderer is here a mj.Renderer object defined by self._renderer = mj.Renderer(self._model, self._height, self._width) where self._model is the model object loaded from the scene file.

After I call the image attribute, the GUI becomes unresponsive while the main rendering loop still seems to be running just fine. Ideally, I would want to control the GUI after i have rendered a picture from the camera.

All feedback and/or ideas for how I could progress/solve this problem of mine are very much appreciated!

[qgallouedec]

Same issue.