MuJoCo Unity Plugin

[aadhithya14]

I am trying to set the qpos values to the assets in unity and render it in unity. The process I am following is, I am trying to run the mujoco simulation separately and get the qpos values over a ZMQ socket to unity and render the assets in unity. I wanted to know if there is any example code available to set the asset qpos in unity? @Balint-H

[Balint-H]

It's relatively straightforward. Depending on the kinds of constraints or guarantees you need it can be as simple as adding the following to a MonoBehaviour that manages the communication with the external process:

public void Start() { 
  MjScene.Instance.preUpdateEvent += UpdateWithRemoteQpos;
}

public unsafe void UpdateWithRemoteQpos(object sender, MjStepArgs e) { 
  double[] qposExternal = GetMostRecentQpos();
  for (int i=0; i<e.model->nq; i++)
  {
    e.data->qpos[i] = qposExternal[i];
  }
  for (int i=0; i<e.model->nv; i++)
  {
    e.data->qvel[i] = 0;
  }
}

I did not test this myself (just quickly typed it up, sorry for the potential typo or syntax error!), as I've not yet needed to pipe in simulation data from an external process to the Unity plugin. The expectation is that this should work. If it does, we could add an example component or scene somewhere that demonstrates it. If it doesn't, let me know and I'll try to fix it/troubleshoot.

You need to implement the "GetMostRecentQpos()" method yourself, which could just reuse values from the last communication if new ones have not arrived since then.

This does mean that you might preform more updates than needed (busy polling). This is unlikely to be the case, since you are using Unity for visualization only, and can set a high timestep without detriment to simulation quality, and achieve a stable smooth framerate (and you'll miss some qpos frames instead, which is no problem).

If you need to render every single frame reliably and therefore need precise manual control over stepping the physics, let me know, it is a reasonably simple addition.

You could also set some disable flags in Global options component in your Unity scene to make sure you are not wasting operations on dynamics/contacts etc as that is managed outside.

[aadhithya14]

Hi @Balint-H , Thanks for your quick resonse. I have a function GetMostRecentQpos() within my MonoBehavior which gets the qpos from the simulation through ZMQ. But when I try to add these lines of code I get an error because of MjStepEventArgs mentioning error CS0246: The type or namespace name 'MjStepEventArgs' could not be found (are you missing a using directive or an assembly reference?). I tried to look out in MjScene.cs file generated at runtime as well as MjActuator.cs file, I don't see this Event Handler. Could you let me know how I can include within my Monobehavior?

[Balint-H]

My bad, it is simply MjStepArgs. I'll update the original answer too.

[aadhithya14]

Thanks @Balint-H. I updated my changes you mentioned. But still I cannot render the assets realtime. I added my Monobehavior script to the asset game object after importing using Mujoco plugin. Attached image for reference.Here the clientobject subscribes to a port using NetMQ and updates the qpos and qvel. Am I doing something wrong ?

[Balint-H]

Could you drop a Debug.Log($"Cur qpos is {string.Join(", ", qposExternal)}") into UpdateWithRemoteQpos() to see if its being called and that the correct values are retrieved?

[aadhithya14]

Hi @Balint-H , thanks for your reply. I broke down the problem into two modules to debug the issue. I removed the communication and loaded a static asset using mujoco unity plugin and try to send different positions to it based on the code snippet you gave me. I am able to set positions after I pause the unity for sometime. I have set the Scene recreation at late update requested to be True. But when I render this in VR it still renders static asset in the first step. Can I not step it in the VR? Looking forward to hearing from you.

Screen.Recording.2024-05-10.at.7.07.45.PM.mov

[Balint-H]

I did a test of the sample script from the previous message above, with the following placeholder kinematic motion source (a sine varying over time on all DoFs):

  private unsafe double[] GetMostRecentQpos() {
    return Enumerable.Repeat((double)Mathf.Sin(Time.time * 0.4f)/2, MjScene.Instance.Model->nq).ToArray();
  }

Movie_002.mp4

https://github.com/google-deepmind/mujoco/assets/41113387/9e5961f7-3a15-482d-b3fe-a5e5b3187795
It seems to work as intended in an example scene with a humanoid, bypassing all joint constraints and contacts.

I also disabled all non-relevant simulation steps (e.g. contacts) in the GlobalOptions component. No need to do scene recreation on late update, unless you are adding/removing things from the scene, and should work without pausing.

Could you confirm in a minimum working example scene that only has mujoco components (e.g. no VR components that cause you exceptions) and your Qpos setter whether you are still having issues?

[aadhithya14]

Hi @Balint-H , thanks for your reply. I tried that before itself. I was able to get my Qpos setter working ( with no VR components) and was able to play the scene in unity. I wanted to know if I can get this working with the VR rendering. Has anyone tried this before?

Screen.Recording.2024-05-13.at.11.40.29.AM.mov

[Balint-H]

I have used the MuJoCo plugin in Unity successfully without issues in VR. That scene was running the physics as well, so simple kinematic control should be supported too.

What system are you running on? The playmode seems to be running on a very low framerate.

Have you set up a minimal scene only with VR/unity components (no MuJoCo) and confirmed you can have the desired features you need from the VR side working?

[aadhithya14]

I am running on a Mac Pro M2. My plugin version is 3.1.5. Unity version 2021.3.5f1
Yes I have my scene only with VR components and it is all working , it is publicly released as well(https://open-teach.github.io/).
Also one more thing , how do you set the MJGlobalOptions. I dont't have that set right now. That could be a reason it is running on a very low framerate

[Balint-H]

Add the MjGlobalSettings component to your scene if its not in it already (or search for it). In the inspector with it selected you can edit Global Options > Flag to disable simulation elements. That will speed things up but it is likely not the bottleneck unless you have a tiny timestep (<0.001).

You can also try to downgrade to 3.1.4 for a while, unfortunately you do need to downgrade at the moment if you want to use MjGlobalSettings.

Does your Unity-only scene run smoothly in VR? If yes, but not with MuJoCo, a troubleshoot step you can try is importing e.g. the humanoid.xml example scene and running it in VR, without any script, just raw mujoco physics and VR.

[aadhithya14]

Yes unity only scene runs smoothly in VR. My timestep is 0.02 so that is fine I think. I will test the humanoid.xml in VR and get back to you.

[aadhithya14]

Hi @Balint-H , I tested out Humanoid.xml example, it renders as a static scene in VR. Without any script if I try to play the unity file with humanoid, unity crashes immediately. Do you know why this could be happening?

[Balint-H]

And it runs no problem without VR? Any error messages when using VR?

[Balint-H]

Ah yes I think I know the current issue, its about using 3.1.5, the plugin on main is not stable due to a small bug, could you downgrade to MuJoCo 3.1.4 (plugin and DLL) or check out the Pull request fixing the issue with the "sensor noise" feature?

[aadhithya14]

Even without the VR it crashes. How do I downgrade the plugin ? Does altering package.json and loading it again works?

[Balint-H]

You can either download it at the 3.1.4 version (https://github.com/google-deepmind/mujoco/tree/3.1.4) or update it in the manifest.json file inside your Unity project folder (Packages > manifest.json) and add this line:

"org.mujoco": "https://github.com/google-deepmind/mujoco.git?path=unity#3.1.4",

Make sure that the mujoco.so library you are using is also 3.1.4 (https://github.com/google-deepmind/mujoco/releases/tag/3.1.4).
If you reference it with the git url like above Unity will download it for you, but you might have to add the library manually to your asset folder.

You likely also have to close and reopen the project to unload the old library.

[aadhithya14]

I get this error IOException: Error loading the model: XML Error: Schema violation: unrecognized attribute: 'sensornoise' on 3.1.4 when I play the unity.

[aadhithya14]

I was able to solve that problem. But I still have the issue when I play it in unity it works but I when I load the APK in the VR it just renders a static image.

[Balint-H]

Thanks for getting it to work (I apologize for directing you to 3.1.4, as the version without the bug was 3.1.3... Thanks for figuring it out)!

Could you click on the mujoco.so file and see what platforms are selected for it? Its possible you have to enable android there.

In the past when I used MuJoCo with VR it was for the Oculus Quest. A quick search online showed that it is possible to use shared libraries with it, but it is possible you need to recompile MuJoCo specifically for it. However, I expected that to be only a potential issue when making a build, not in the Editor.

@MyelinsheathXD did you end up succeeding in compiling MuJoCo for android? #1638

[aadhithya14]

I am using Oculus Quest as well. In the editor things work smoothly, only while making a build in the VR, the scene rendered is static. I will try to enable android on the mujoco.so and see if that works

[Balint-H]

In my example I used Quest Link to stream the app to the headset. Is that an option for you?

[aadhithya14]

yes I use Quest Link with a cable.

[MyelinsheathXD]

@Balint-H hey! Unfutunately, I am not able to build mujoco library for android yet.
Currently I am working on another project .
However , here the link for the engine’s 3rd party libraries that was build for android, if you like to build the mujoco itself for android using Android studio’s CMake
https://github.com/MyelinsheathXD/modelsLib

[aadhithya14]

I tried to reinstall everything from scratch on 3.1.3 and even mujoco so 3.1.3. I found my mujoco so was 3.1.4 and plugin was 3.1.3 which could be causing issues. But when I reinstall I get this error which I never got before. Do you know the fix for it? @Balint-H
DllNotFoundException: mujoco assembly: unknown assembly type:unknown type member:(null)
Mujoco.MjVfs..ctor () (at /Users/aadhithyaiyer/Downloads/mujoco-3.1.3/unity/Runtime/Tools/MjVfs.cs:88)

[aadhithya14]

Hi @Balint-H , do you know the fix to this issue?

[Balint-H]

My guess is the following issue:

Make sure the mujoco library is named libmujoco.so and not "libmujoco.so.3.1.3" . Annoyingly Unity will display it as "mujoco.so" even if its extension is longer, but will ignore it unless you truncate the extension to just so.

Can't guarantee that's the issue as I only used the plugin on Windows and Linux in the past (where this is a common cause of unrecognized .so files), but I suspect it could be related.

[aadhithya14]

Hi , I modified that already. I also tested things in windows, it plays in unity editor but the humanoid asset is static when build in VR. I feel this is a bug. Let me know if you care able to find a fix for it. Thanks.

[Balint-H]

The time I had MuJoCo running on the Quest, I was using Quest link via USB, but ran the app from the Editor (and streamed the VR scene in play mode to the headset). If you have not tried that, and only tried to make a stand-alone build, this approach could be a temporary solution for you. I'd expect an android compatible build of MuJoCo is needed for the stand-alone build to work, which could be a nice feature, but is not yet supported.