Physics interpolation: MultiMesh

[rburing]

Adds physics interpolation support to multimeshes, based on @lawnjelly's

• Fixed Timestep Interpolation (3D) (3.x) #52846
• Physics interpolation - Move out of Scenario #60147

This is running at 5 physics ticks per second, interpolating position and color:

multimesh-interpolation.mp4

Test project: godot-multimesh-test.zip and Godot 3 version for comparison: godot-3-multimesh-test.zip

This is a sizable self-contained chunk of the interpolation work, so I put it in its own PR.

To-do:

• Make it compile (port it to the new Godot API and rendering architecture).
• Make it work with the Forward+ and Mobile renderers.
• Make it work with the Compatibility renderer.

This PR is sponsored by My Spare Time™.

[rburing]

I ported more stuff like follow-up PRs, but I still didn't manage to make it fully work yet.

Here is a test project godot-multimesh-test.zip with a simple script:

extends MultiMeshInstance3D

const SPEED: float = 5.0

func _physics_process(delta: float) -> void:
        for i in multimesh.instance_count:
                multimesh.set_instance_transform(i, multimesh.get_instance_transform(i).translated(Vector3.UP * SPEED * (1 if i % 2 == 0 else -1) * delta))

        print(multimesh.get_instance_transform(0).origin)

It works when physics interpolation is disabled (and it moves the boxes up/down in visibly discrete steps as expected, at a low physics tick rate), but it doesn't work when physics interpolation is enabled (movement is stopped).

The same script in Godot 3 does work correctly: godot-3-multimesh-test.zip

Any help with finding what I missed is very welcome.

[fire]

I wasn't able to find the physical interpolation for MeshInstances was that forward ported?

[rburing]

@fire It is not yet ported. As far as I understand, MultiMesh physics interpolation should work independently from the rest (and it is a big chunk of the work, so I chose to do it first). For example, the following code does work already with this PR:

extends MultiMeshInstance3D

const SPEED: float = 5.0

var time: float = 0.0

func _physics_process(delta: float) -> void:
	time += delta
	for i in multimesh.instance_count:
		multimesh.set_instance_transform(i, Transform3D(Basis(), time * Vector3.UP * SPEED * (1 if i % 2 == 0 else -1) * delta))

It is interpolated. So what seems to be broken in this PR currently (possibly as a side-effect) is get_instance_transform.

Edit: The MultiMesh interpolation does work using the Forward+ and Mobile renderers, just not with Compatibility.

Edit 2: It seems the buffer layout of a MultiMesh is different when using the Compatibility renderer, and the physics interpolation code is currently assuming the other layout.

Edit 3: That's not exactly the issue; there is some compression/decompression that should take care of the difference already. However, there is an issue in the GLES3 MultiMesh cache (found with help from @clayjohn):

godot/drivers/gles3/storage/mesh_storage.cpp

Lines 1950 to 1951 in bdc0316

	//if we have a data cache, just update it
	multimesh->data_cache = multimesh->data_cache;

[fire]

I can be a rubber ducky, if you want to explain how you think it's broken. One of my friends is dependent on the addon interpolation and we could probably devise a testing strategy.

[rburing]

@fire Thanks, the issue has already been found and fixed by the PR linked just above.

This PR is ready for review now.

[rburing]

There is some probability of getting garbage data in the MultiMesh buffer on game start (e.g. in the MRP above with Forward+ and 5 physics ticks per second), probably due to data_prev being uninitialized. The same issue can be reproduced in Godot 3.5.3 (with the Godot 3 MRP above and 5 physics ticks per second). We can replace mmi->_data_prev.resize(size_in_floats) by mmi->_data_prev.resize_zeroed(size_in_floats) here, but I don't know if this is the best solution.

[fire]

Is there an identity init? Alternatively instead of interpolating pick one of the inputs. Like return the inputs.

[lawnjelly]

There is some probability of getting garbage data in the MultiMesh buffer on game start (e.g. in the MRP above with Forward+ and 5 physics ticks per second), probably due to data_prev being uninitialized.

Ah I'll see if I can double check this in 3.x if I get a moment. 👍

[lawnjelly]

I've had a little look through and it looks mostly okay from what I could see (a few differences from 3.x but I'm sure @rburing has figured them out). To some extent this kind of thing you have to go for and then see what bugs come up that might have been missed.

It probably needs a sensible review from someone on 4.x rendering team though, as architectural concerns are slightly different in 3.x (which is mostly maintenance) compared to 4.x (which is being rapidly iterated).

In particular the wrapper for the multimeshes I find a bit ugly (with the underscores), I don't know if anyone can think of a more elegant way.

[rburing]

@lawnjelly Thanks! I added a link to this PR in the notes for the rendering team meeting on 2024-05-23 along with two questions, which got the following answers:

Is the wrapper around multimeshes okay for 4.x? Is there any alternative approach? See lawnjelly's comment.

Might not be a more clean way to do this on the CPU side

rburing: Is it okay that the interpolator keeps its own copies of the buffers in the original format (for interpolation purposes), and then each frame RendererMeshStorage::update_interpolation_frame calls _multimesh_set_buffer which does buffer compression/packing in the GLES3 driver? This is one of the main differences with Godot 3.x, where _multimesh_set_buffer did not use compression/packing. Is there any alternative approach?

An alternative would be to interpolate the mesh on the gpu

Using 2 GPU buffers. Then interpolate on the GPU. Should scale better to larger scenes

Alternatively consider parallelizing update_interpolation_frame

I could probably parallelize RendererMeshStorage::update_interpolation_frame without too much difficulty, but interpolating on the GPU sounds like it would be more optimal. What do you think? Doing it on the GPU could perhaps be done as a follow-up, since I don't think this affects the user-facing API, and it would require doing all that transform interpolation on the GPU, which would be some significant extra effort (figuring out how to get this into the shader, etc.).

[lawnjelly]

I could probably parallelize RendererMeshStorage::update_interpolation_frame without too much difficulty, but interpolating on the GPU sounds like it would be more optimal. What do you think? Doing it on the GPU could perhaps be done as a follow-up, since I don't think this affects the user-facing API, and it would require doing all that transform interpolation on the GPU, which would be some significant extra effort (figuring out how to get this into the shader, etc.).

Sure, interpolating on GPU is always an option. 👍

Kind of considerations I would be thinking:

• What is the ratio of updates to renders?
  I.e. if the tick rate is 30 tps and render at 120fps, then you are getting 4 renders for each update, so even though you update 2x as much to GPU, it only uploads 1/4 as often.

At 60tps and 60fps, you are uploading 2x the data to save doing some small calculations on the CPU, it may be less clearcut.

Also for games, sometimes a constant cost per frame (even if higher) is better than cost that changes wildly frame by frame.

• Does the increased upload bandwidth outweigh the CPU interpolation costs?
  Lerping is likely pretty cheap, even on CPU, compared to loading / storing memory
• Code complexity / maintenance
  How intrusive do you want the interpolation to be? Is it going to be easier to maintain with interpolation in the shaders, or in the CPU code? Maybe not much difference.
• How many instances are there likely to be in the multimeshes? (With large numbers, then the bottlenecks might matter more than code complexity, and vice versa)
• Future compatibility (what happens if you want to add hermite interpolation? you then need to upload more data to GPU and things change)

It's hard to predict some of these things, and if you wanted to use GPU approach it might be good to start by doing some benchmark comparisons of both before committing, maybe GPU could be wildly faster and that might clinch it.

[Calinou]

• Future compatibility (what happens if you want to add hermite interpolation? you then need to upload more data to GPU and things change)

I doubt Hermite interpolation is viable for physics, given its requirement on an additional variable (velocity) and the fact it needs one more frame of latency to do its work. At 30 Hz physics, the latency from linear physics interpolation is already significant, so Hermite would make it even higher for limited visual benefit.

It can be viable for networking in MMOs and the like, but the physics interpolation system isn't designed to work for networking as it lacks any kind of buffering (it's only designed for client-side physics). For networking, a dedicated system should be created instead: godotengine/godot-proposals#7280