Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracer / NeF: background values for additional channels #152

Open
orperel opened this issue May 2, 2023 · 0 comments
Open

Tracer / NeF: background values for additional channels #152

orperel opened this issue May 2, 2023 · 0 comments
Assignees
Labels
bug Something isn't working feature New features
Milestone

Comments

@orperel
Copy link
Collaborator

orperel commented May 2, 2023

The current PackedRFTracer implementation allows some control for bg_color to the RGB channel. We're missing this flexibility for additional channels (i.e. whose "clear color" may be random or some other non-zero value).

There are actually 3 entangled issues here that need to be addressed:

  1. Allow to set the background value for any channel type, not just RGB.
  2. The premultiplied alpha flag should be orthogonal to the bg color to avoid confusion, as i.e. Render Color Equation  #143
  3. Properly introduce neural fields with an env MLP that learns the bg color

The suggested fix is to extend BaseNeuralField with the concept of background functions one could register, similar to forward ones:

class BaseNeuralField:

    def _register_bg_function(self, fn, channels):
        """Registers a background function.

        Args:
            fn (function): Function to register.
            channels (list of str): Channel output names.
        """
        if isinstance(channels, str):
            channels = [channels]
        self._bg_functions[fn] = set(channels)

    @abstractmethod
    def register_background_functions(self):
         pass

class NeuralRadianceField:
    def register_background_functions(self):
        register_background_functions(self.bg_rgb, ["rgb"])
        register_background_functions(self.bg_depth, ["depth"])

    def bg_rgb(self, rays: Ray):
           ... # Return color according to bg param, or from some env decoder

    def bg_depth(self, rays: Ray):
           ... # Return color according to far plane value == rays.dist_max

The tracer should now be agnostic to background color - and instead fetch it from the neural field per channel, similar to how foreground color is obtained.

The premultiplied alpha flag will remain within the tracer, as it determines how blending should take place.

@orperel orperel self-assigned this May 2, 2023
@orperel orperel added bug Something isn't working feature New features labels May 2, 2023
@orperel orperel added this to the Wisp 0.1.3 milestone May 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working feature New features
Projects
None yet
Development

No branches or pull requests

1 participant