Figure out Sony IBIS data #727

AdrianEddy · 2023-09-22T02:10:49Z

IBIS data in Sony files consists of a timestamp, angle of rotation and X/Y translation of the sensor.
The main unknown is timing (not sure when the first sample in the array happens exactly) and then how to use the rotation and shift in the stabilization kernel

Adding this to gyro_source.rs after Sony timing:

// IBIS
telemetry_parser::try_block!({
    use telemetry_parser::tags_impl::*;
    let ibis = tag_map.get(&GroupId::IBIS)?;

    let shift =  (ibis.get_t(TagId::Data) as Option<&Vec<TimeVector3<i32>>>)?;
    let angle =  (ibis.get_t(TagId::Data2) as Option<&Vec<TimeVector3<i32>>>)?;
    let mut xs = BTreeMap::<i64, f64>::new();
    let mut ys = BTreeMap::<i64, f64>::new();
    let mut th = BTreeMap::<i64, f64>::new();
    //dbg!(angle);


    let pixel_pitch = (8400.0, 8400.0);
    let e406 = 1000000000.0;

    assert_eq!(shift.len(), angle.len());
    for (s, a) in shift.into_iter().zip(angle.into_iter()) {
        let x = ((e406 / 1000000000.0) * (4096.0 / pixel_pitch.0)) * s.x as f64;
        let y = ((e406 / 1000000000.0) * (4096.0 / pixel_pitch.1)) * s.y as f64;

        if xx < 100 {
            dbg!((x, y, s.x, s.y, a.z));
            xx += 1;
        }
        xs.insert(s.t as i64, x);
        ys.insert(s.t as i64, y);
        th.insert(a.t as i64, a.z as f64);
    }

    let mut xs2 = Vec::new();
    let mut ys2 = Vec::new();
    let mut angles = Vec::new();
    for vid_y in 0..2160 {
        let x = telemetry_parser::util::interpolate_at_timestamp((vid_y as f64 * (8850.0 / 2160.0)).round() as i64, &xs);
        let y = telemetry_parser::util::interpolate_at_timestamp((vid_y as f64 * (8850.0 / 2160.0)).round() as i64, &ys);
        let t = telemetry_parser::util::interpolate_at_timestamp((vid_y as f64 * (8850.0 / 2160.0)).round() as i64, &th);
        angles.push(t / 1000.0);
        xs2.push((x / 4096.0));
        ys2.push((y / 4096.0));
    }

    md.per_frame_data.push(serde_json::json!({
        "translatex": xs2,
        "translatey": ys2,
        "angle": angles
    }));
});

And this to frame_transform.rs:

        let mut angles = Vec::new();
        let mut xs = Vec::new();
        let mut ys = Vec::new();

        (|| -> Option<()> {
            let frame_md = params.gyro.file_metadata.per_frame_data.get(frame)?;
            angles.extend(frame_md.get("angle")?.as_array()?.iter().map(|x| x.as_f64().unwrap() as f32));
            xs.extend(frame_md.get("translatex")?.as_array()?.iter().map(|x| x.as_f64().unwrap() as f32));
            ys.extend(frame_md.get("translatey")?.as_array()?.iter().map(|x| x.as_f64().unwrap() as f32));
            Some(())
        })();
        
        ....
        
        let matrices = (0..rows).into_par_iter().map(|y| {
            ....

            let th = *angles.get(y).unwrap_or(&0.0) as f64;
            let theta = Matrix3::new_rotation(th * (std::f64::consts::PI / 180.0));
            let sx = *xs.get(y).unwrap_or(&0.0) as f32;
            let sy = *ys.get(y).unwrap_or(&0.0) as f32;

           ....
           // then replace last 0.0, 0.0, 0.0 with
                sx, sy, th as f32

can get you the IBIS data all the way up to the kernel, so you can start experimenting

IBIS data needs to be interpolated for every row of the video (and used together with rolling shutter correction) - this is already interpolated in this sample code, but I think the timing is wrong and thus the point of reference (to the captured frame) is not correct

Sample file with IBIS: https://drive.google.com/file/d/1QgNEK9KtRxIRdS734IgfeH_UTi2Ian3_/view?usp=sharing
Sample file with IBIS and OIS: https://drive.google.com/file/d/15rlTkZfqrRYYCK_mcXel8jKCs-klPNaG/view?usp=sharing

The text was updated successfully, but these errors were encountered:

AdrianEddy · 2023-09-22T02:11:18Z

/bounty $1000

algora-pbc · 2023-09-22T02:11:21Z

💎 $1,000 bounty • Gyroflow

Steps to solve:

Start working: Comment /attempt #727 with your implementation plan
Submit work: Create a pull request including /claim #727 in the PR body to claim the bounty
Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Additional opportunities:

🔴 Livestream on Algora TV while solving this bounty & earn $200 upon merge! Comment /livestream once live

Thank you for contributing to gyroflow/gyroflow!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🟢 @aazam-gh	Sep 22, 2023, 6:33:15 PM	WIP
🟢 @uzmi1	Oct 29, 2023, 4:39:47 AM	WIP
🟢 @younghun1124	Feb 3, 2024, 6:06:37 PM	WIP

aazam-gh · 2023-09-22T18:33:14Z

/attempt #727

Options

Cancel my attempt

aazam-gh · 2023-09-22T18:41:28Z

Hi @AdrianEddy any tips for working on this issue in addition would be much appreciated :)

AdrianEddy · 2023-09-22T18:52:33Z

Hi, most of what I know is already in the description. Figuring this out will require a good understanding of how IBIS works and probably some knowledge about lens geometry will be needed as well.
Sony's own Catalyst Browse has support for this data, and reverse-engineering Catalyst Browse might be useful. I have a Ghidra project with some structs reconstructed, if you want to dive into it, but I have to warn you that it's an massive app with A LOT of math operations compiled to C++ so reversing that is a very difficult task.

For reading the metadata by itself, compile latest master of gyro2bb and you can use gyro2bb --dump C1386-ibis.MP4 to see the data. The same data is in let ibis = tag_map.get(&GroupId::IBIS)?; in the sample code

Catalyst Browse uses Catmull-Rom for interpolation, but I don't think it matters much. The point is to get an exact angle, shift_x and shift_y for every row of the video. Once we have that, we need to add that in the kernel somewhere (kernels are in OpenCL, WGSL, GLSL and Rust, easiest is to use OpenCL as it can be hot-reloaded, make sure to change Preview pipeline and Device for video processing in Advanced), maybe in rotate_and_distort but I'm not sure in which step it should be applied. it's then in the matrix[9], matrix[10] and matrix[11]

The main task here is to understand how to use that data, implementation itself is a small detail

aazam-gh · 2023-09-22T18:57:43Z

Thanks for the valuable insight. I shall start by diving into IBIS and take a look at Catalyst Browse first.

AdrianEddy · 2023-09-22T18:59:27Z

Feel free to contact me on Discord for any details about that or the Gyroflow itself, so you can focus on figuring out IBIS, instead of figuring out the Gyroflow code and where things are and what they do :)

aazam-gh · 2023-09-22T19:00:39Z

Noted. That approach seems the most effective in getting this done

AdrianEddy · 2023-09-22T19:05:42Z

Also, proper implementation (optimized and clean, in all kernel languages etc) is not necessary to claim the bounty. Working proof of concept is enough, and I'll take care of proper implementation, once the algorithm is figured out and proved that it works correctly.

aazam-gh · 2023-09-22T19:06:59Z

Very well. I shall keep that in mind as well.

uzmi1 · 2023-10-29T04:39:46Z

/attempt #727

Options

Cancel my attempt

algora-pbc · 2023-10-29T04:39:49Z

Note: The user @aazam-gh is already attempting to complete issue #727 and claim the bounty. If you attempt to complete the same issue, there is a chance that @aazam-gh will complete the issue first, and be awarded the bounty. We recommend discussing with @aazam-gh and potentially collaborating on the same solution versus creating an alternate solution.

uzmi1 · 2023-10-29T04:46:28Z

Hi Adrianaddy check this solution-

The provided code appears to integrate IBIS (In-Body Image Stabilization) data into a larger system. Let's break down the solution step by step:

Parsing IBIS Data in gyro_source.rs:
- The code extracts IBIS data (shift and angle) from telemetry tags in Sony files.
- It calculates x, y translations, and rotation angles based on pixel pitch and a constant.
- The results are stored in BTreeMaps for further processing.
Interpolating IBIS Data in frame_transform.rs:
- IBIS data (translations and angles) is retrieved from per-frame metadata.
- Interpolation is performed for each row of the video, generating translated and rotated coordinates.
- Matrices are constructed with the interpolated values and integrated into the processing pipeline.
IBIS Data Experimentation:
- The code supports experimentation by providing IBIS data up to the kernel.
- IBIS data needs to be interpolated for every row of the video, considering rolling shutter correction.
Timing Concerns:
- There's a note about potential timing issues, suggesting that the current timing might be incorrect.
- The timing issue may affect the point of reference to the captured frame.
Sample Files:
- Two sample files are provided for testing, one with IBIS and the other with both IBIS and OIS (Optical Image Stabilization).

To improve the code:

Verify and refine the timing logic to ensure accurate synchronization with the captured frames.
Consider testing with the provided sample files to validate the IBIS data integration.
Further experimentation may involve adjusting interpolation methods or exploring alternative approaches based on specific use cases.

For any precise debugging or modifications, thorough testing with sample files and detailed understanding of the specific Sony sensor characteristics would be essential.

rgaufman · 2023-12-02T21:02:55Z

This would be so badass as the physical IBIS stabilisation does reduce that shake/shimmer effect when using lower shutter speeds, this is the only thing preventing me from using GyroFlow as a permanent replacement to Catalyst Browse :)

digital-phoenix · 2024-01-12T00:53:28Z

@AdrianEddy is this issue still open I'd like to work on it.

AdrianEddy · 2024-01-12T01:20:26Z

@digital-phoenix it is, please join discord, and there's a lot of details discussed in this thread; https://discord.com/channels/797044698682228736/1155153866174767175

younghun1124 · 2024-02-03T18:06:35Z

/attempt #727

Options

Cancel my attempt

younghun1124 · 2024-02-03T18:13:24Z

IBIS data in Sony files consists of a timestamp, angle of rotation and X/Y translation of the sensor. The main unknown is timing (not sure when the first sample in the array happens exactly) and then how to use the rotation and shift in the stabilization kernel

Adding this to gyro_source.rs after Sony timing:

// IBIS
telemetry_parser::try_block!({
    use telemetry_parser::tags_impl::*;
    let ibis = tag_map.get(&GroupId::IBIS)?;

    let shift =  (ibis.get_t(TagId::Data) as Option<&Vec<TimeVector3<i32>>>)?;
    let angle =  (ibis.get_t(TagId::Data2) as Option<&Vec<TimeVector3<i32>>>)?;
    let mut xs = BTreeMap::<i64, f64>::new();
    let mut ys = BTreeMap::<i64, f64>::new();
    let mut th = BTreeMap::<i64, f64>::new();
    //dbg!(angle);


    let pixel_pitch = (8400.0, 8400.0);
    let e406 = 1000000000.0;

    assert_eq!(shift.len(), angle.len());
    for (s, a) in shift.into_iter().zip(angle.into_iter()) {
        let x = ((e406 / 1000000000.0) * (4096.0 / pixel_pitch.0)) * s.x as f64;
        let y = ((e406 / 1000000000.0) * (4096.0 / pixel_pitch.1)) * s.y as f64;

        if xx < 100 {
            dbg!((x, y, s.x, s.y, a.z));
            xx += 1;
        }
        xs.insert(s.t as i64, x);
        ys.insert(s.t as i64, y);
        th.insert(a.t as i64, a.z as f64);
    }

    let mut xs2 = Vec::new();
    let mut ys2 = Vec::new();
    let mut angles = Vec::new();
    for vid_y in 0..2160 {
        let x = telemetry_parser::util::interpolate_at_timestamp((vid_y as f64 * (8850.0 / 2160.0)).round() as i64, &xs);
        let y = telemetry_parser::util::interpolate_at_timestamp((vid_y as f64 * (8850.0 / 2160.0)).round() as i64, &ys);
        let t = telemetry_parser::util::interpolate_at_timestamp((vid_y as f64 * (8850.0 / 2160.0)).round() as i64, &th);
        angles.push(t / 1000.0);
        xs2.push((x / 4096.0));
        ys2.push((y / 4096.0));
    }

    md.per_frame_data.push(serde_json::json!({
        "translatex": xs2,
        "translatey": ys2,
        "angle": angles
    }));
});

And this to frame_transform.rs:

        let mut angles = Vec::new();
        let mut xs = Vec::new();
        let mut ys = Vec::new();

        (|| -> Option<()> {
            let frame_md = params.gyro.file_metadata.per_frame_data.get(frame)?;
            angles.extend(frame_md.get("angle")?.as_array()?.iter().map(|x| x.as_f64().unwrap() as f32));
            xs.extend(frame_md.get("translatex")?.as_array()?.iter().map(|x| x.as_f64().unwrap() as f32));
            ys.extend(frame_md.get("translatey")?.as_array()?.iter().map(|x| x.as_f64().unwrap() as f32));
            Some(())
        })();
        
        ....
        
        let matrices = (0..rows).into_par_iter().map(|y| {
            ....

            let th = *angles.get(y).unwrap_or(&0.0) as f64;
            let theta = Matrix3::new_rotation(th * (std::f64::consts::PI / 180.0));
            let sx = *xs.get(y).unwrap_or(&0.0) as f32;
            let sy = *ys.get(y).unwrap_or(&0.0) as f32;

           ....
           // then replace last 0.0, 0.0, 0.0 with
                sx, sy, th as f32

can get you the IBIS data all the way up to the kernel, so you can start experimenting

IBIS data needs to be interpolated for every row of the video (and used together with rolling shutter correction) - this is already interpolated in this sample code, but I think the timing is wrong and thus the point of reference (to the captured frame) is not correct

Sample file with IBIS: https://drive.google.com/file/d/1QgNEK9KtRxIRdS734IgfeH_UTi2Ian3_/view?usp=sharing Sample file with IBIS and OIS: https://drive.google.com/file/d/15rlTkZfqrRYYCK_mcXel8jKCs-klPNaG/view?usp=sharing

I have an opinion on this code. Even if the distance that the sensor has moved due to ibis is the same, the amount of correction of the image should vary depending on how much the sensor has moved outward from the center of the lens. This is because if the sensor moves in the part where the light is heavily compressed (the part where the density of light is due to the distortion of the lens), it will move more than in the part where it is not.

+This is a scenario when Sony cameras process files that have their own lens distortion correction. Rather than adjusting the amount of pixel shift according to the degree of lens distortion, it may be easier to undo the lens distortion correction of the camera itself, perform the pixel shift, and correct the lens distortion again.

digital-phoenix · 2024-02-03T18:16:37Z

@younghun1124 you should join the discord if you haven't already there is much more information there.

rgaufman · 2024-03-28T11:55:17Z

Thank you to those that are working on this, I think it will be game changing once this is added. The IBIS removes a lot of the blur and micro jitters, so not being able to have this on in the camera is what's stopping a lot of Sony users from using GyroFlow at the moment.

AdrianEddy added the enhancement New feature or request label Sep 22, 2023

AdrianEddy added this to the 1.6.0 milestone Sep 22, 2023

algora-pbc bot added the 💎 Bounty label Sep 22, 2023

AdrianEddy mentioned this issue Sep 26, 2023

Figure out Sony lens distortion parameters #44

Closed

AdrianEddy mentioned this issue Apr 26, 2024

Full Sony metadata support #825

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Figure out Sony IBIS data #727

Figure out Sony IBIS data #727

AdrianEddy commented Sep 22, 2023 •

edited

AdrianEddy commented Sep 22, 2023

algora-pbc bot commented Sep 22, 2023 •

edited

aazam-gh commented Sep 22, 2023 •

edited by algora-pbc bot

aazam-gh commented Sep 22, 2023

AdrianEddy commented Sep 22, 2023 •

edited

aazam-gh commented Sep 22, 2023

AdrianEddy commented Sep 22, 2023

aazam-gh commented Sep 22, 2023

AdrianEddy commented Sep 22, 2023

aazam-gh commented Sep 22, 2023

uzmi1 commented Oct 29, 2023 •

edited by algora-pbc bot

algora-pbc bot commented Oct 29, 2023

uzmi1 commented Oct 29, 2023

rgaufman commented Dec 2, 2023

digital-phoenix commented Jan 12, 2024

AdrianEddy commented Jan 12, 2024

younghun1124 commented Feb 3, 2024 •

edited by algora-pbc bot

younghun1124 commented Feb 3, 2024 •

edited

digital-phoenix commented Feb 3, 2024

rgaufman commented Mar 28, 2024

Figure out Sony IBIS data #727

Figure out Sony IBIS data #727

Comments

AdrianEddy commented Sep 22, 2023 • edited

AdrianEddy commented Sep 22, 2023

algora-pbc bot commented Sep 22, 2023 • edited

💎 $1,000 bounty • Gyroflow

Steps to solve:

Additional opportunities:

aazam-gh commented Sep 22, 2023 • edited by algora-pbc bot

aazam-gh commented Sep 22, 2023

AdrianEddy commented Sep 22, 2023 • edited

aazam-gh commented Sep 22, 2023

AdrianEddy commented Sep 22, 2023

aazam-gh commented Sep 22, 2023

AdrianEddy commented Sep 22, 2023

aazam-gh commented Sep 22, 2023

uzmi1 commented Oct 29, 2023 • edited by algora-pbc bot

algora-pbc bot commented Oct 29, 2023

uzmi1 commented Oct 29, 2023

rgaufman commented Dec 2, 2023

digital-phoenix commented Jan 12, 2024

AdrianEddy commented Jan 12, 2024

younghun1124 commented Feb 3, 2024 • edited by algora-pbc bot

younghun1124 commented Feb 3, 2024 • edited

digital-phoenix commented Feb 3, 2024

rgaufman commented Mar 28, 2024

AdrianEddy commented Sep 22, 2023 •

edited

algora-pbc bot commented Sep 22, 2023 •

edited

aazam-gh commented Sep 22, 2023 •

edited by algora-pbc bot

AdrianEddy commented Sep 22, 2023 •

edited

uzmi1 commented Oct 29, 2023 •

edited by algora-pbc bot

younghun1124 commented Feb 3, 2024 •

edited by algora-pbc bot

younghun1124 commented Feb 3, 2024 •

edited