Add horizontal skew and vertical skew features

[guoyingtao]

https://www.hackingwithswift.com/articles/135/how-to-render-uiviews-in-3d-using-catransformlayer
https://dzone.com/articles/a-look-at-perspective-transform-correction-with-co
https://stackoverflow.com/questions/8235288/perspective-correction-of-uiimage-from-points
https://github.com/paulz/PerspectiveTransform

CATransform3D CATransform3DMakeRotation(CGFloat angle, CGFloat x, CGFloat y, CGFloat z);

[guoyingtao]

Possible steps

1. Rotate UIImageView by CATransform3DMakeRotation
2. Get corner point of rotated UIImageView
3. Use CIPerspectiveTransform to create the new image

[guoyingtao]

func CATransform3DMakePerspective(_ x: CGFloat, _ y: CGFloat) -> CATransform3D {
    var transform = CATransform3DIdentity
    transform.m34 = -1.0 / 1000.0
    transform = CATransform3DRotate(transform, y, 1, 0, 0)
    transform = CATransform3DRotate(transform, x, 0, 1, 0)
    return transform
}

[guoyingtao]

@objc func handleSliderValueChanged(_ slider: UISlider) {
    let value = CGFloat(slider.value)
    let inputImage = CIImage(image: imageView.image!)
    
    // Calculate the perspective correction values
    let inputWidth = inputImage.extent.width
    let inputHeight = inputImage.extent.height
    let topLeft = CGPoint(x: 0, y: 0)
    let topRight = CGPoint(x: inputWidth, y: 0)
    let bottomLeft = CGPoint(x: 0, y: inputHeight)
    let bottomRight = CGPoint(x: inputWidth, y: inputHeight)
    let correction = CIPerspectiveTransform(inputImage: inputImage,
                                             topLeft: topLeft,
                                             topRight: CGPoint(x: topRight.x + value * inputWidth, y: topRight.y),
                                             bottomLeft: bottomLeft,
                                             bottomRight: CGPoint(x: bottomRight.x + value * inputWidth, y: bottomRight.y))
    
    // Apply the perspective correction to the CIImage
    let outputImage = correction.outputImage
    let context = CIContext()
    let cgImage = context.createCGImage(outputImage, from: outputImage.extent)
    imageView.image = UIImage(cgImage: cgImage!)
}

In this modified function, we calculate the perspective correction values based on the value of the slider, and then apply the CIPerspectiveTransform filter to the CIImage to create the corrected output image. Finally, we create a UIImage from the corrected output image and set it as the image for the UIImageView.

Note that this implementation assumes that the image is being cropped to preserve the original aspect ratio. If you want to allow for the image to be resized to fit the screen, you will need to adjust the calculation of the topRight and bottomRight points.

[guoyingtao]

Made a little bit progress with the code below

        var transform = CATransform3DIdentity
        transform.m34 = -1.0 / 500.0
        transform = CATransform3DRotate(transform, totalRadians, 1, 0, 0)
        cropWorkbenchView.layer.transform = transform

[Karllas]

Hello,

Any plans of finishing this implementation?

[guoyingtao]

@Karllas
I haven't found a good way to build this feature yet, so I have no plans to finish it in the near future.

[rickshane]

The solution to this problem is fairly straightforward in a Core Image workflow:

There is a UX that presents two sliders to the user - a Horizontal Perspective adjustment slider with values ranging from -n to +n, centered at 0, and a Vertical Perspective adjustment slider with the same values.

Both sliders will provide input to a single function that calculates input to the Core Image CIPerspectiveTransform filter, which accepts four CIVectors that represent adjustments at the four corners of the image.

Only a single side of the image is adjusted for a given axis. A negative value for the horizontal adjustment results in the left side of the image being adjusted. A positive value for the horizontal adjustment results in the right side of the image being adjusted (see image below). As the adjustment in magnitude increases, both the x-axis and y-axis adjustment increases on the side of the adjustment (though not necessarily with equal magnitude - this is something to be tuned by the implementer). The same scenario applies for vertical adjustments.

NOTE: This depends on the design. Apple DOES adjust both sides for a given axis, but not in equal amounts and it depends on the value of the slider. The mechanics of this will need to be codified for Apple's implementation.

Regarding design of some popular tools: Photomator and Darkroom both adjust one side for a given axis. Apple adjusts both sides. Lightroom anchors in the center and pivots like a see-saw (which is similar to the demo video above).

This is a stateless solution - subsequent adjustments do not stack as transactions, they merely replace the previous value. All adjustments must be applied to the original full-sized image.

A flip transform will invert any non-zero adjustment along the same axis, i.e., if there is a negative horizontal adjustment, then flipping the image horizontally will invert the horizontal adjustment to a positive value of the same magnitude.

A 90 degree rotation will swap the horizontal and vertical slider values. It is important to still apply an inversion depending on the rotation state.

A straighten operation that adjusts the rotation degrees (such as performed by the RotationDial control) should NOT affect the slider values.

A single function taking as input the two slider values then calculates the four input values to set in the CIPerspectiveTransform filter for inputTopLeft, inputTopRight, inputBottomLeft, inputBottomRight. It is left to the implementer to determine how these calculations are generated.

NOTE: When both horizontal and vertical adjustments are made at the same time, then one of the inputs to CIPerspectiveTransform = f(horizontalAdjustment, verticalAdjustment), i.e., one of the corners adjusted will be determined by both slider values.

The most complex issue for Mantis is that of UX: how to design the user interface to accommodate the two sliders. Obviously the SlideControl may be used for both adjustments. They could be stacked in the area below the image where the rotation dial appears, or there could be a modal solution where only one control appears at a time.

[rickshane]

Here is the description of CIPerspectiveTransform from Core Image Filter Reference:

[rickshane]

[guoyingtao]

@rickshane
Thanks for the suggestion!
The challenging part is when rotating the image, how to adjust cropBox for 3d space. I haven't figured it out yet.

Update

It is not adjusting cropBox, it should be sometimes the image need to keep touching the cropBox corners while rotating. Looks like it needs more work to do for a skewed image.

RPReplay_Final1711317587.mov

[guoyingtao]

For UX part, we can directly use the design from Apple's Photos app which separated rotation with horizontal skew and vertical skew.

I have another project https://github.com/guoyingtao/Inchworm which servers the similar purpose. Once I solved this issue, I can borrow the Apple's design to Mantis.

[rickshane]

Very nice component!