Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Custom GLContext python ? #1658

Open
GCrequer opened this issue May 13, 2024 · 0 comments
Open

Custom GLContext python ? #1658

GCrequer opened this issue May 13, 2024 · 0 comments
Labels
question Request for help or information

Comments

@GCrequer
Copy link

Hello everyone,

I'm a student in robotics and computer vision, and I'm trying to get into MuJoCo. I have some knowledge of OpenGL, enough to understand the basics, but not enough to be fully independent with all the subtleties.

My goal is to simulate cameras with strong distortions (fisheye, 360°, etc.) embedded on robots. To do this, I need to bypass MuJoCo's default OpenGL context. I've read that it's entirely possible to do so here and I've found the class template for it in the __init__.py files (for now, I'm just trying with mujoco.GLContext as it's integrated).

However, it's mentioned that once the context is created, the classic mjr_ functions should work. So, I don't know where my mistake is in the following simplified code:

import mujoco as mj
import numpy as np

from mujoco.glfw import glfw
from matplotlib import pyplot as plt

xml= """
<mujoco>
    <asset>
    <texture type="2d" name="groundplane" builtin="checker" mark="edge" rgb1="0.2 0.3 0.4" rgb2="0.1 0.2 0.3" markrgb="0.8 0.8 0.8" width="300" height="300"/>
    <material name="groundplane" texture="groundplane" texuniform="true" texrepeat="5 5" reflectance="0.2"/>
    </asset>
	<worldbody>
        <light pos="0 0 1.5" dir="0 0 -1" directional="true"/>
            <geom name="floor" size="0 0 0.05" type="plane" material="groundplane"/>
		<body pos="0 0 1">
			<joint type="free"/>
			<geom type="box" size=".1 .2 .3" rgba="0 .9 0 1"/>
		</body>
	</worldbody>
</mujoco>
"""

model = mj.MjModel.from_xml_string(xml)
data = mj.MjData(model)

cam = mj.MjvCamera()

opt = mj.MjvOption()
mj.mjv_defaultOption(opt)
scene = mj.MjvScene(model, maxgeom = 100)


### TO ADAPT : changing shaders, projection matrix, etc.
ctx = mj.GLContext(1200, 900)
ctx.make_current()
###


frame1 = np.zeros((900, 1200, 3), dtype=np.uint8)
while True:
    mj.mj_forward(model, data)
    mj.mj_step(model, data)
    
    viewport = mj.MjrRect(0, 0, 1200, 900)

    mj.mjv_updateScene(model, data, opt, None, cam,  mj.mjtCatBit.mjCAT_ALL.value, scene)
    mj.mjr_render(viewport, scene, ctx)
    mj.mjr_readPixels(frame1, None, viewport, ctx)
    plt.imshow(frame1)

    plt.pause(0.01)

plt.show()

I prefer to use my own context because making changes to mjrContext is not recommended here,and I believe that "everything" is possible with our own context, and that it's through this GLContext class that we can create it. I know that theses classes are only intended for offscreen rendering (#544), I will have to postprocess my frames so it is not a problem.

If you have any hints to give me, or an alternative to suggest (knowing that I'm already struggling to understand how to modify shaders and retrieve vertices in this system), it would be great to know !
Thanks in advance!
@GCrequer GCrequer added the question Request for help or information label May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Request for help or information
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@GCrequer