Gaussian instantiation out of memory

[oUp2Uo]

Hi, I have find some problem with Gaussian instantiation out of memory.

First time, I run a ply file ~2.1GB, and it worked well.
GPU memory usage went to ~8.5GB at first, then went to 9.4GB when the render result showed.

The next time, I run a ply file ~500MB, which is an edited version of the first file, by deleting some 'fog' points.
GPU memory usage also went to ~8.5GB at first, but then it came out Gaussian instantiation out of memory.

Why an edited small ply file causes out of memory?
The graphic card is Geforce RTX 2080 Ti 11GB.

Thanks.

[shg8]

Thanks for raising the issue. The reason is that each Gaussian needs to be initialized for each tile they appear in, so the theoretical maximum memory needed is (number of Gaussians * ceil(width / 16) * ceil(height / 16)). The renderer doesn't handle dynamic memory allocation right now, so everything is preallocated. You can try increasing the number here.

[oUp2Uo]

Thanks for the reply.
Change SORT_ALLOCATE_MULTIPLIER works!
Maybe memory limit could be enlarged automatically when the usage is close to the limit value.
Or just gave out warning, throw away the remaining points, and do not shut down the program.

Btw, key control now have 2 translation (left/right and front/near), maybe there could be one more pair for up/down?
For example,
change std::array<bool, 4> getKeys(); in Window.h to std::array<bool, 6> getKeys();.
and add

glfwGetKey(static_cast<GLFWwindow*>(window), GLFW_KEY_T) == GLFW_PRESS,
glfwGetKey(static_cast<GLFWwindow*>(window), GLFW_KEY_G) == GLFW_PRESS

in getKeys() function in Window.cpp.
Then add

if (keys[4]) {
    direction += glm::vec3(0.0f, -1.0f, 0.0f);
}
if (keys[5]) {
    direction += glm::vec3(0.0f, 1.0f, 0.0f);
}

in handleInput() function in Renderer.cpp.

And also, mouse control is not easy at some time (hard to rotate to the angle wanted).

[shg8]

Thanks for the reply. Change SORT_ALLOCATE_MULTIPLIER works! Maybe memory limit could be enlarged automatically when the usage is close to the limit value. Or just gave out warning, throw away the remaining points, and do not shut down the program.

Btw, key control now have 2 translation (left/right and front/near), maybe there could be one more pair for up/down? For example, change std::array<bool, 4> getKeys(); in Window.h to std::array<bool, 6> getKeys();. and add

glfwGetKey(static_cast<GLFWwindow*>(window), GLFW_KEY_T) == GLFW_PRESS,
glfwGetKey(static_cast<GLFWwindow*>(window), GLFW_KEY_G) == GLFW_PRESS

in getKeys() function in Window.cpp. Then add

if (keys[4]) {
    direction += glm::vec3(0.0f, -1.0f, 0.0f);
}
if (keys[5]) {
    direction += glm::vec3(0.0f, 1.0f, 0.0f);
}

in handleInput() function in Renderer.cpp.

And also, mouse control is not easy at some time (hard to rotate to the angle wanted).

Hey these are great suggestions. Controls are a bit wonky right now. How I envisioned it is that the window would capture the mouse on click using glfwSetInputMode(window, GLFW_CURSOR, GLFW_CURSOR_DISABLED). You can stop mouse capturing on pressing ESC. There should also be an status indicator for capture status in the Controls imgui window. Feel free to submit a PR if you're interested in implementing any of these.

[shg8]

Also the latest commit should add dynamic memory management. Feel free to give it a try.

[oUp2Uo]

Also the latest commit should add dynamic memory management. Feel free to give it a try.

The latest commit worked with the 2nd model I used yesterday.
And maybe, add checking for the max memory size limit before changing sortBufferSizeMultiplier is even safer. : )

About camera control,
maybe I feel a little weird just because mouse moving horizontal not just changing yaw and mouse moving vertical not just changing pitch of the camera now.
I am not sure if this is right, or what is a correct/comfortable mouse control method.

[shg8]

I added a few things that should bring better controls. The scene is not aligned right now and it's usually loaded up-side-down, so that's why it might feel a bit weird. I'll add some form of calibration in the future.

[oUp2Uo]

Mouse click and ESC release makes control feel better.

But I tried to move mouse left/right, and after some times, still the target model looks like rotating in roll direction.
Sometimes clockwise and other times anti-clockwise. I donot know why.

And I found it hard to change the camera to the other side of the model.
Which means if the start point is at the back of the object, it is hard to move to the front of the object and then turn 180 degree to watch the object.

[shg8]

There isn't really a good way to place the camera at the right position unless we read the training dataset as well. I might add that as an option.

[oUp2Uo]

There isn't really a good way to place the camera at the right position unless we read the training dataset as well. I might add that as an option.

I thought the rotation vector should not just be like x or y axis.
Should this rotation vector be calculated using the camera orientation?

          camera.rotation = glm::rotate(camera.rotation, static_cast<float>(translation[0]) * 0.005f,
                                        glm::vec3(0.0f, -1.0f, 0.0f));
          camera.rotation = glm::rotate(camera.rotation, static_cast<float>(translation[1]) * 0.005f,
                                        glm::vec3(-1.0f, 0.0f, 0.0f));

(Btw, add a option for on/off showing origin point (0, 0, 0) and xyz axes direction may help debugging, if possible.)

[shg8]

There isn't really a good way to place the camera at the right position unless we read the training dataset as well. I might add that as an option.

I thought the rotation vector should not just be like x or y axis. Should this rotation vector be calculated using the camera orientation?

          camera.rotation = glm::rotate(camera.rotation, static_cast<float>(translation[0]) * 0.005f,
                                        glm::vec3(0.0f, -1.0f, 0.0f));
          camera.rotation = glm::rotate(camera.rotation, static_cast<float>(translation[1]) * 0.005f,
                                        glm::vec3(-1.0f, 0.0f, 0.0f));

(Btw, add a option for on/off showing origin point (0, 0, 0) and xyz axes direction may help debugging, if possible.)

Good catch! I'll fix that.