Issues with Color-correcting Kinect and RealSense recordings

[jackjansen]

We should color-correct our recordings made with offline Realsense or Kinect capturers, but that is currently impossible because of a number of distinct issues:

• We haven't captured a color reference, so we would have to make an educated guess as to what the correction should be. But we have captured the Aruco Origin target, which has areas of white, black, red, green, blue and yellow, so maybe we can use that for our educated guess? Or if that doesn't work we could do it "by taste" if we have serious software like BlackMagic DaVinci.

• If we want to use standard software like DaVinci I guess we have to isolate the color track out of our recording, process that in DaVinci, and then re-insert the color track into the recording. But this re-inserting is going to be a problem:

  • For the Kinect the recordings are made into.mkv files. I can use a very long ffmpeg command line to extract the RGB track. But I have not been able to find the command line to re-insert the track. The Azure Kinect MKV file is rather non-standard-complying. Help here would be needed. (and incidentally the ability to reconstruct Kinect MKV files would also help @ashutosh3308 with his VQEG experiments).
  • For the Realsense things are probably even worse, because the recording is made into a .bag format, which isn't really a video file format but more a stream of events. I don't think we have any chance of recreating this.

• An alternative would be to do the conversion in code, as we are reading the RGB data from the recording to construct the point clouds. I presume that we could multiply each (r, g, b) value by a 3x3 matrix (possibly after de-gamma-correcting it). But do do this I would need a reference to working code, or to an algorithm.

[cwidisgroup]

For reference here are 8 stills showing the Aruco origin marker (from 8 cameras):

[ashutosh3308]

@jackjansen:

Color Correction Using Aruco Markers
We could use OpenCV for color correction. Please use Link 1 . Possibly, Link 2 may help as well.

FFmpeg related things, please see this link. You can use MKVToolNIX to extract the tracks without any issues.

[jackjansen]

Thanks! I think https://stackoverflow.com/questions/70233645/color-correction-using-opencv-and-color-cards looks the most promising. I will look at it.

[jackjansen]

Some ideas on implementation. Let's assume 8 cameras for now.

1. We use the above method to compute the R, G and B curves for each camera for a specific well-known area. An option is to take a square around the Aruco marker, about twice the size of the marker (or maybe 3 times, or maybe just the aruco marker itself so we have just black and white, let's experiment).
2. From these 8 sets of 3 curves (R, G and B)we compute one "best set of curves". Maybe averaging, maybe something else.
3. Now, for each camera, we compute the 3 LUTs (lookup tables, of 256 entries each) that convert the source curves to the destination curves. Or we could store these conversion LUTs in a standard format like https://resolve.cafe/developers/luts/
4. We store the LUT filenames in cameraconfig.json, in the per-camera section.
5. Those LUT entries will be ignored by the low-level (C/C++) capturers, but the Python wrappers will recognize them. They will parse the LUT files and convert them to 3 arrays of 256 bytes. They will then use the new cwipc_tiledsource_auxiliary_operation() method to pass these 3 arrays to the low-level per-stream capturers.
6. The capturers can now do 3 cheap lookups for every pixel as it receives an RGB image.