Query in correspondence of samples across datasets

[pmkalshetti]

Hi @Chris10M,

Thank you for providing the synchronized RGB streams and the MANO parameters. I had a query regarding the number of samples in this data. Below are the number of samples for each subject.

Subject	Ev2Hands-R (subject_{id}_event.pickle)	MANO_Parameters (event_mano_params.pkl)	MANO_Parameters (rgb_mano_params.pkl)	RGB_Data (subject_{id}_rgb.mp4)
1	11384	11384	11383	11385
2	11777	11777	11777	11778
3	11921	11921	11921	11922
4	13275	13275	13275	13276
5	12386	12386	12385	12387

Could you tell me the correspondence across these datasets? Specifically, given a sample index in rgb_mano_params.pkl, what is the corresponding frame in subject_{id}_rgb.mp4? Also, for subjects 1 and 5, given a sample index in event_mano_params.pkl what is the corresponding sample index in rgb_mano_params.pkl?

[Chris10M]

Hi @pmkalshetti,

I'm sorry for not getting back to you sooner. I did not receive notifications for Github.

For the datasets, the start_frame is synchronised across the event and RGB streams with their corresponding GTs. Hence,
subject_{id}_event.pickle) , MANO_Parameters (event_mano_params.pkl) , MANO_Parameters (rgb_mano_params.pkl), RGB_Data (subject_{id}_rgb.mp4) all have synchronised starting index.

You can clip the exceeding frames or the frame correspondences at the end.

To verify this, I ran this on the MANO meshes obtained by both the streams,

transformed_event_mesh = event_mesh.apply_transform(event_to_rgb_transform); print(np.abs(transformed_event_mesh.vertices * 1000 - rgb_mesh.vertices * 1000).mean())
The error comes out to be 0.023 mm (approx).

Also, I checked the sync scripts for RGB and event streams with the multi-view studio (to obtain the Captury GTs) and they too have synchronised starting index.

Best,
Christen

[pmkalshetti]

That's great; I'll clip the frame indices at the end. Thank you for verifying :)