De-fish

[abrock]

This pull-request introduces two changes to the perspective correction tool, previous discussion was in #6676

1. Add support for de-fishing by clicking a checkbox.
   The code uses the focal length provided by the user (already existing feature) and assumes a perfect equidistant fisheye model for computing the un-distortion operation. It works well together with the other perspective corrections.

Here's a perfect checkerboard rendered using blender with a equidistant camera model (and also a cat):
https://drive.google.com/drive/folders/1_DOu7Zm-_YjaRgid8okSnyvQ6rP6Gtm8?usp=share_link

The sidecar file already has the correct focal length set. If you change any perspective parameter it will always look like a checkerboard captured using a rectilinear lens, even though the image file shows a circular fisheye.

2. Add a scale option to the perspective adjustments.
   This is needed since de-fishing of an extreme fisheye lens means scaling the corners of the fisheye image. Without down-scaling we lose a large portion of the field of view.

An alternative (or addition) would be to allow the user to increase the resulting image size. This would have the benefit that the center doesn't have to be scaled down (or not as much) so the optical resolution in the center is preserved. I think the current state is quite useful already, so increasing the image size should be tackled separately later.

[Floessie]

@abrock Thanks for your contribution! 👍

I've left some comments for you to streamline the code. They are just some minor formatting issues that should be easy to fix.

Best,
Flössie

[Floessie]

@abrock @Lawrence37 @Thanatomanic I'm fine code-wise.

@Lawrence37 @Thanatomanic Please review functionality.

[abrock]

I found one bug I'm not sure what's causing it:
When I open an image I haven't opened before and enable "De-fish" the preview gets 100% black. As soon as I change the focal length it suddenly works as expected.

[Lawrence37]

De-fishing is not related to perspective, so it makes more sense to place the check box in the upper level or inside Distortion Correction.

Similarly, Scale applies to more than just perspective. I suggest combining it with Auto-fill, like how the radius in Capture Sharpening can be calculated automatically or set manually.

[abrock]

I agree that de-fishing is unrelated to perspective. At the moment there's a problem with moving the checkbox to distortion: The effect of de-fishing depends to a very high degree on the focal length setting in perspective.
Should I add a second focal length setting in the distortion correction section?

[Lawrence37]

Should I add a second focal length setting in the distortion correction section?

That is one option, or you can move the focal length to the upper level. The choice depends on if there is ever a situation where the focal lengths need to be different.

[abrock]

That is one option, or you can move the focal length to the upper level. The choice depends on if there is ever a situation where the focal lengths need to be different.

In theory the correct focal length for de-fishing is also the correct focal length for perspective, assuming a perfect fisheye. In practice I don't use the correct focal length, I use something that looks good.
After de-fishing a user might want to change the perspective so something that looks good with all the degrees of freedom provided by the perspective tool.

So it might make more sense to have separate tools without shared parameters.

[Beep6581]

I'm very happy to see de-fishing capability in RT!

A thought regarding the discussion on where to put this tool in the GUI and pipeline. De-fishing is a form of re-projection - you are changing the projection from equidistant fisheye to rectilinear. It is not a form of distortion correction. While in lay terms anything wonky is called distorted, there's more to it. Definitions vary, but one which serves well to clarify the matter is that a distortion is a deviation from the perfect projection, whatever that projection might be. A fisheye lens can itself suffer from distortion if the image produced by the lens is not perfectly projected (nothing is). Theoretically, a user might want to perform distortion correction on a fisheye image without re-projecting it into rectilinear.

With the tool the way it currently is, I cannot re-project a photo and fix perspective automatically - would be nice if that was possible.

Test image:
http://rawtherapee.com/shared/test_images/fisheye_samyang_8mm_shed.pef

[abrock]

I merged the dev branch into my branch, I hope I did it right. Still compiles and works for me.

[abrock]

[...] and the scale be tied to the auto-fill.

How does that work / which behavior do you want to see?

[Lawrence37]

I envision something like the the contrast threshold or radius in capture sharpening. The user can adjust it manually or choose automatic. In the current stable release, there is only an automatic option which scales the image by the smallest amount possible while filling the entire frame up with the source image. The scale also automatically updates as needed when the user tweaks the transform parameters. When combined with a slider, the user can override the scale. At that point, automatic selection of the scale is disabled until the user enables it again.

[Beep6581]

Great work @abrock Would be great to have this in 5.10.
I tested 87e96b3e9 on the same image and it seems to work fine. Am I correct in understanding that the "Focal length" in the "Distortion Correction" tool must be specified manually?

[Beep6581]

@Lawrence37 what are your thoughts on merging this? I'm leaning towards merging and leaving improvements for the future.

[abrock]

Great work @abrock Would be great to have this in 5.10. I tested 87e96b3e9 on the same image and it seems to work fine. Am I correct in understanding that the "Focal length" in the "Distortion Correction" tool must be specified manually?

Yes. In theory this could be automatically set from the focal length of the lens if that information is included in the exif data, but I only own 100% manual fisheye lenses without any electronics inside them.
Another possibility would be to have some smart detection of lines that are straight in reality within a fisheye image and then automatically adjust the focal length to make the de-fished lines as straight as possible.
When working with the tool I found that eye-balling the focal length is quick and easy. Usually I re-use the setting of one image for a whole series of images taken with the same focus setting.

[Lawrence37]

I will take a look a this soon. Just scanning the code, I see the scale is inside the camera-based perspective tool. That should be moved outside to make it more accessible. The scale is also independent of the auto-scale (or does auto-scale override it?). It could complicate backward compatibility if we decide to integrate it with the auto-scale before 6.0.

[abrock]

I will take a look a this soon. Just scanning the code, I see the scale is inside the camera-based perspective tool. That should be moved outside to make it more accessible.

I'm happy to change it if someone could make a mockup, or draw some arrows in a screenshot of what should go where.

The scale is also independent of the auto-scale (or does auto-scale override it?). It could complicate backward compatibility if we decide to integrate it with the auto-scale before 6.0.

With de-fishing disabled:
Setting a scale smaller than 1 does nothing. Setting a scale larger than 2 zooms in (I'm confused, I would have expected a scale larger than 1 to already zoom in). With Auto-Fill disabled, it works as I expected it, scale < 1 means zooming out and black borders appearing, scale > 1 means zooming in.

With de-fishing enabled:
Setting a scale smaller than 1 suddenly makes sense, it allows for a larger field of view to be visible in the rectified image. With auto-fill disabled and vertical field of view of the input < 180° Auto-fill prevents very small scales which would lead to some areas of the resulting image to be black. Without Auto-fill it's the user's job to crop it correctly

I think the current behavior makes sense.

[Lawrence37]

I think putting it at the first arrow shown in this comment makes sense.

The Auto-Fill behavior is due to it fighting the scale. There's probably a limit to how much Auto-Fill scales the image which explains why scale starts working at 2x. Disabling the scale when Auto-Fill is selected should make the behavior more intuitive. Making the slider inactive when disabled will show the user that it has no effect.