How do I make change a splat's position at runtime?

[GesusMist]

Hi I'm a student trying to implement blendshape for GaussuanSplats by mapping gaussian splattings' positions to a model's meshes'.
I found it complicated when I want to make changes to splats' positions at runtime and my current approach is

1. Update the centerArray in viewer.splatmesh.splatbuffer
2. call viewer.splatmesh.resetDataFromSplatBuffer()

No wonder this is a dumb, but easy to find way. Would bring huge performance problem.

I noticed that it seems to code information such as centerArray into textures to make it compute inside shader by GPU. However I am not yet professional enough to handle that.
Is there a faster while relatively simple way to do this?

[mkkellogg]

Unfortunately, right now there's not an easy built-in way to accomplish what you're trying to do in a performant way. What you're currently doing with these two steps:

1. Update the centerArray in viewer.splatmesh.splatbuffer
2. call viewer.splatmesh.resetDataFromSplatBuffer()

is really the only way to do it, and as you mentioned the performance will not be good.

Eventually I'd like to support something like dynamic transformations for each individual splat. It would be straight-forward to do, I imagine I'd have to store a transformation matrix for each splat that would be sent to the shader on every frame. The big down-side would be all the memory required to store those transforms. Conceivably, in order to save some memory (and depending on exactly what you're trying to do), you could implement something like skeletal animation with vertex skinning, so you wouldn't necessarily have to store a transformation for every splat, only for the "bone" that a splat is connected to.

[GesusMist]

Well, Thanks!
Great news. It seems I worked this out but with some artifects, could you please probably find the problem?
I managed to implement it in following way:

1. convertting morphtargets (i.e. Blendshapes) of the mesh to morphtargets of the splats' centers
2. Store them as datatexture and pass it to shader
3. Every frame I update the weight of each blendshape

Folloing are the codes I modified in vertexShaderSource

vec3 blendshape = vec3(0.0,0.0,0.0);
for (int i = 0; i < numBS; i++) {
        uvec4 sampledblendshape = texture(blendshapeTexture, getDataUV(numBS, i, blendshapeTextureSize));
         vec3 bs = uintBitsToFloat(uvec3(sampledblendshape.rgb));
          blendshape += bs * blendshapeStrength[i];
       }

vec3 splatCenter = uintBitsToFloat(uvec3(sampledCenterColor.gba));
splatCenter += blendshape;

Actually this is basicly the same thing that you mention.

https://github.com/mkkellogg/GaussianSplats3D/assets/119684095/a9ab793f-b58d-47de-955a-b19b34b75537
Here is a video of my work. You would notice artifacts at 0'49'' that the teeth and eyeballs were visibvle when eyes and mouth closed.

(↑ This is what my work looks like)
(↓This is what it should look like)

I guess it's some problem with opacity, but I have no idea how to fix this.

I'm sorry but I have some more questions.
Why did you scaled position.xy by 2.0 ( why 2.0, and why not axis z)
Should I modify vPosition?

[mkkellogg]

Sorry, I haven't really had time recently to look into this, hopefully I will soon. I'll let you know if can figure something out!

[mkkellogg]

I think for this issue I might need to see all of your code so I can try running it locally to debug it. It looks like your modifications should move the splat centers as intended. The video you uploaded is all black, could you try re-uploading it?

[GesusMist]

https://github.com/mkkellogg/GaussianSplats3D/assets/119684095/ddd5c441-d545-474b-89b1-577c10c910b4
↑↑↑ mine
↓↓↓ correct
https://github.com/mkkellogg/GaussianSplats3D/assets/119684095/1f7241be-58ef-499d-bd7e-6ed4b804204d

[mkkellogg]

My first guess would be that the splats are not being properly sorted. The offset to the splat center that is being applied in the vertex shader also needs be applied to the splat center when calculating the splat's distance from the camera for the sort. Is that happening?

[GesusMist]

Thanks! I think you're right, I did not apply the offset for calculating distence for the sort. I'm currently not very familiar with opengl coding thus I'm not really sure how to apply it. Does simply calling SplatMesh.updateCentersGPUBufferForDistancesComputation() work?

[mkkellogg]

Well it's a little more complicated than that. First, updateGPUCentersBufferForDistancesComputation() is only used when gpuAcceleratedSort is enabled. It updates the GPU buffer that stores the splats' center positions for pre-computing splat distances using a transform feedback operation. The shader attribute that is linked to this buffer is here:

GaussianSplats3D/src/SplatMesh.js

Line 1126 in 6298b67

in ivec4 center;

or here (depending on whether or not you're using an integer or floating-point based sort):

GaussianSplats3D/src/SplatMesh.js

Line 1148 in 6298b67

in vec3 center;

To be clear: this is not the GPU buffer that stores splat centers for rendering (In that case a data texture is actually used)

In your case the trasnform feedback operation would need to incorporate the blend shape data to properly calculate the splat distances, so I think you'd need to modify the setupDistancesComputationTransformFeedback() to modify the transform feedback setup to include blend shape data, and then add a new function to store the blend shape data in a GPU buffer similar to updateGPUCentersBufferForDistancesComputation(), maybe called something like: updateBlendShapeBufferForDistancesComputation().

[GesusMist]

Hi, thanks so much for helping, I did as your advice, everything works well except using a texture.

(There is a brief description of my problem at the end if you don't want/need to read these code)

This is the shader for distance computation after I modified. (My modification is Bolded)
`

            #version 300 es
            precision highp float;
            in ivec3 center;
            uniform ivec3 modelViewProj;
            flat out int distance;

            uniform highp usampler2D BSTexture;
            uniform float blendshapeStrength[51];
            in ivec3 IndexBsnum1;
            


            const vec4 encodeNorm4 = vec4(1.0 / 255.0, 1.0 / 255.0, 1.0 / 255.0, 1.0 / 255.0);
            const uvec4 mask4 = uvec4(uint(0x000000FF), uint(0x0000FF00), uint(0x00FF0000), uint(0xFF000000));
            const uvec4 shift4 = uvec4(0, 8, 16, 24);
            vec4 uintToRGBAVec (uint u) {
               uvec4 urgba = mask4 & u;
               urgba = urgba >> shift4;
               vec4 rgba = vec4(urgba) * encodeNorm4;
               return rgba;
            }

            vec2 getDataUV(in int stride, in int offset, in vec2 dimensions, in int splatIndex) {
                vec2 samplerUV = vec2(0.0, 0.0);
                float d = float(uint(splatIndex) * uint(stride) + uint(offset)) / dimensions.x;
                samplerUV.y = float(floor(d)) / dimensions.y;
                samplerUV.x = fract(d);
                return samplerUV;
            }
            void main(void) {
            
                vec3 blendshape = vec3(0.0,0.0,0.0);



                
                for (int i = 0; i < IndexBsnum1[1]; i++) {
                    uvec4 sampledblendshape = uvec4(texture(BSTexture, getDataUV(IndexBsnum1[1], i, vec2(4096,4096), IndexBsnum1[0])));
                    vec3 bs = uintBitsToFloat(uvec3(sampledblendshape.rgb));
                    blendshape += bs * blendshapeStrength[i];
                }

                ivec3 splatCenter = center.rgb;
                splatCenter.x += int(blendshape.x*1000.0);
                splatCenter.y += int(blendshape.y*1000.0);
                splatCenter.z += int(blendshape.z*1000.0);


                distance = splatCenter.x * modelViewProj.x + splatCenter.y * modelViewProj.y + splatCenter.z * modelViewProj.z;
                //distance = center.x * modelViewProj.x + center.y * modelViewProj.y + center.z * modelViewProj.z;
                //distance = 0*iii.x + center.x * modelViewProj.x + center.y * modelViewProj.y + center.z * modelViewProj.z;
            }
        `

And these setups
`

        gl.useProgram(this.distancesTransformFeedback.program);
        this.distancesTransformFeedback.centersLoc =
            gl.getAttribLocation(this.distancesTransformFeedback.program, 'center');
        this.distancesTransformFeedback.modelViewProjLoc =
            gl.getUniformLocation(this.distancesTransformFeedback.program, 'modelViewProj');
            //GesusMModified
        this.distancesTransformFeedback.BSTextureLoc =
            gl.getUniformLocation(this.distancesTransformFeedback.program, 'BSTexture');
        this.distancesTransformFeedback.IndexBsnum1Loc =
            gl.getAttribLocation(this.distancesTransformFeedback.program, 'IndexBsnum1');
        this.distancesTransformFeedback.blendshapeStrengthLoc = 
            gl.getUniformLocation(this.distancesTransformFeedback.program, 'blendshapeStrength');


        console.log("qwqwqwqwq",this.distancesTransformFeedback);
        if (rebuildGPUObjects || rebuildBuffers) {
            this.distancesTransformFeedback.centersBuffer = gl.createBuffer();
            gl.bindBuffer(gl.ARRAY_BUFFER, this.distancesTransformFeedback.centersBuffer);
            gl.enableVertexAttribArray(this.distancesTransformFeedback.centersLoc);
            gl.vertexAttribIPointer(this.distancesTransformFeedback.centersLoc, 3, gl.INT, 0, 0);


            this.distancesTransformFeedback.IndexBsnum1Buffer = gl.createBuffer();
            gl.bindBuffer(gl.ARRAY_BUFFER, this.distancesTransformFeedback.IndexBsnum1Buffer);
            gl.enableVertexAttribArray(this.distancesTransformFeedback.IndexBsnum1Loc);
            gl.vertexAttribIPointer(this.distancesTransformFeedback.IndexBsnum1Loc, 3, gl.INT, 0, 0);
        }

        if (rebuildGPUObjects || rebuildBuffers) {
            this.distancesTransformFeedback.outDistancesBuffer = gl.createBuffer();
        }
        gl.bindBuffer(gl.ARRAY_BUFFER, this.distancesTransformFeedback.outDistancesBuffer);
        gl.bufferData(gl.ARRAY_BUFFER, splatCount * 4, gl.DYNAMIC_COPY);

        if (rebuildGPUObjects) {
            this.distancesTransformFeedback.id = gl.createTransformFeedback();
        }
        gl.bindTransformFeedback(gl.TRANSFORM_FEEDBACK, this.distancesTransformFeedback.id);
        gl.bindBufferBase(gl.TRANSFORM_FEEDBACK_BUFFER, 0, this.distancesTransformFeedback.outDistancesBuffer);

        //console.log("paddedblendshapes",this.paddedblendshapes);
        gl.activeTexture(gl.TEXTURE0);
        this.texture = gl.createTexture();
        gl.bindTexture(gl.TEXTURE_2D, this.texture);
        
        gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA32UI, 4096, 4096, 0, gl.RGBA_INTEGER, gl.UNSIGNED_INT,this.paddedblendshapes);
        gl.generateMipmap(gl.TEXTURE_2D);
        gl.uniform1i(this.distancesTransformFeedback.BSTextureLoc, 0);


        if (currentProgram) gl.useProgram(currentProgram);
        if (currentVao) gl.bindVertexArray(currentVao);`

And below is computeDistancesOnGPU()
`computeDistancesOnGPU(modelViewProjMatrix, outComputedDistances) {

    if (!this.renderer) return;

    const iViewProjMatrix = this.getIntegerMatrixArray(modelViewProjMatrix);
    const iViewProj = [iViewProjMatrix[2], iViewProjMatrix[6], iViewProjMatrix[10]];

    // console.time("gpu_compute_distances");
    const gl = this.renderer.getContext();

    const currentVao = gl.getParameter(gl.VERTEX_ARRAY_BINDING);
    const currentProgram = gl.getParameter(gl.CURRENT_PROGRAM);

    gl.bindVertexArray(this.distancesTransformFeedback.vao);
    gl.useProgram(this.distancesTransformFeedback.program);

    gl.enable(gl.RASTERIZER_DISCARD);

    gl.uniform3i(this.distancesTransformFeedback.modelViewProjLoc, iViewProj[0], iViewProj[1], iViewProj[2]);



    // gl.activeTexture(gl.TEXTURE0);
    // gl.bindTexture(gl.TEXTURE_2D, this.texture);
    // gl.uniform1i(this.distancesTransformFeedback.BSTextureLoc, 0);
    
    console.log("blendshapeStrength",this.material.uniforms.blendshapeStrength.value);
    // console.log("blendshapeStrength",this.distancesTransformFeedback);
    gl.uniform1fv(this.distancesTransformFeedback.blendshapeStrengthLoc,this.material.uniforms.blendshapeStrength.value);

    gl.bindBuffer(gl.ARRAY_BUFFER, this.distancesTransformFeedback.centersBuffer);
    gl.enableVertexAttribArray(this.distancesTransformFeedback.centersLoc);
    gl.vertexAttribIPointer(this.distancesTransformFeedback.centersLoc, 3, gl.INT, 0, 0);

    console.log("indexbuffer", this.distancesTransformFeedback);

    gl.bindBuffer(gl.ARRAY_BUFFER, this.distancesTransformFeedback.IndexBsnum1Buffer);
    gl.enableVertexAttribArray(this.distancesTransformFeedback.IndexBsnum1Loc);
    gl.vertexAttribIPointer(this.distancesTransformFeedback.IndexBsnum1Loc, 3, gl.INT, 0, 0);
    

    gl.bindTransformFeedback(gl.TRANSFORM_FEEDBACK, this.distancesTransformFeedback.id);
    gl.bindBufferBase(gl.TRANSFORM_FEEDBACK_BUFFER, 0, this.distancesTransformFeedback.outDistancesBuffer);

    gl.beginTransformFeedback(gl.POINTS);
    gl.drawArrays(gl.POINTS, 0, this.getSplatCount());
    gl.endTransformFeedback();

    gl.bindBufferBase(gl.TRANSFORM_FEEDBACK_BUFFER, 0, null);
    gl.bindTransformFeedback(gl.TRANSFORM_FEEDBACK, null);

    gl.disable(gl.RASTERIZER_DISCARD);

    gl.bindBuffer(gl.ARRAY_BUFFER, this.distancesTransformFeedback.outDistancesBuffer);
    gl.getBufferSubData(gl.ARRAY_BUFFER, 0, outComputedDistances);
    gl.bindBuffer(gl.ARRAY_BUFFER, null);

    // console.timeEnd("gpu_compute_distances");

    if (currentProgram) gl.useProgram(currentProgram);
    if (currentVao) gl.bindVertexArray(currentVao);

}`

---

To be brief,

I added uniform highp usampler2D BSTexture; in the shader

Get its loc this.distancesTransformFeedback.BSTextureLoc = gl.getUniformLocation(this.distancesTransformFeedback.program, 'BSTexture');

and tried to upload data to GPU using the following code in setupDistancesTransformFeedback() ,

        `gl.activeTexture(gl.TEXTURE0);
        this.texture = gl.createTexture();
        gl.bindTexture(gl.TEXTURE_2D, this.texture);
        gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA32UI, 4096, 4096, 0, gl.RGBA_INTEGER, gl.UNSIGNED_INT,this.paddedblendshapes);
        gl.generateMipmap(gl.TEXTURE_2D);
        gl.uniform1i(this.distancesTransformFeedback.BSTextureLoc, 0);`

It does not work, google chrome would say
[.WebGL-00004F9802B51400] GL_INVALID_OPERATION: Mismatch between texture format and sampler type (signed/unsigned/float/shadow).
and FireFox browser would say
webGL warning: drawArraysInstanced:TEXTURE_2D at unit 0 is of type FLOAT, but the shader samples as UINT.
webGL warning: getBuffersubData: Reading from a buffer with usage other than *_READ Causes pipeline stalls. capy through a STREAM_READ buffer ,

If uncomment these lines in computeDistancesOnGPU()
// gl.activeTexture(gl.TEXTURE0); // gl.bindTexture(gl.TEXTURE_2D, this.texture); // gl.uniform1i(this.distancesTransformFeedback.BSTextureLoc, 0);
The error FireFox would output is
webGL warning: drawArraysInstanced: TEXTURE_2D at unit 0 is incomplete: Bad mipmap dimension or format.
webGL warning: getbuffersubData:Reading from a buffer with usage other than *_READ Causes pipeline stalls. copy through a STREAM_READ buffer .

I'm sure I got the gl.texImage2D() parameters right so I think its that the texture was not passed to the shader correctly. I checked IndexBSnum1 and it's correctly passed.

[GesusMist]

For why I'm using a texture:

There are about 50~100 blendshapes for my model, if we use vectors it means I would have to declare 100 uniforms in shader.
I can endore some influence on frame rate, but from my observation it seems the render frame rate is not affected still keeps a rate of 6ms per frame, I'm very confused.

[mkkellogg]

Sorry for the late replay, What I think might be happening is state conflict with three.js. Unfortunately WebGL does not provide a nice way to isolate state changes, so when you bind textures or activate texture units, it may conflict with what three.js thinks the state should be. I have tried to get around that to some extent by using a VAO (Vertex array object) for all state that is associated with the transform feedback calculations. The idea is that state for three.js is saved in its own VAO, and when I perform the transform feedback calculations I activate my own VAO for the calculations and then restore the VAO for three.js when they're done. The problem is that you can't save all WebGL state in a VAO, only certain state like vertex attribute bindings. So state like texture bindings shouldn't really be touched outside of three.js unless you know you can restore it after you've done what you need to do.

[mkkellogg]

Since GPU accelerated sorting may be difficult given the situation I described above, you could try the CPU-based sort for your project. I could point you to where you would need to make changes in order to push the blend-shapes data to the sort worker.