[FEATURE] Add support for GoPro timelapse videos

[fpokryvk]

It would be nice to have support for timelapse videos. This might be a bit tricky. If I am not wrong, GoPro is using variable video speed while taking timelapse in default setting (speed: auto), so distribution of GPX waypoints might not be uniform thoughout the video. But, it should be possible to pair every GPS fix timespamp from the video to the closest waypoint timespamp from GPX file and then distribute rest of GPX waypoints to the video frames between GPS marks in the video.

I see the comment in README stating some GPX interpolation is maybe planned, this seems a bit different direction, but might be related to that.

Thank you!

[progweb]

It's already supported.

Actually, gpx2video isn't able to detect time-lapse / time wrap video.

But you can use "--time-factor" option to read gpx and compute telemetry data.

[fpokryvk]

That would work for fixed factor time wrap videos, but with auto mode, the factor varies in time (at least I think so, I will check on some longer timelapse)...

[progweb]

Indeed, in this case, it can't work. I have to be able to parse time-lapse settings.

[fpokryvk]

Hmm, I am rendering some longer video with average time factor computed by hand, when I checked few random points from map recognizable in the video, it was giving roughly the same factor (computed by hand), so maybe time factor is constant per video, I will check the result, if there are some sections that are really off and let you know.

Anyway, thanks for --time-factor tip, I was not aware of it.

[fpokryvk]

I have tested on some hilly timelapse video and it is not matching, sometimes numbers seems to be ahead and sometimes behind the video, so it is seems to be not constant time factor. I will put some effort to decode GPS data from gopro, split video by these GPS waypoints and use different time factor for each part and then merge the video into one file again.

[fpokryvk]

I have created some simple python script using pgmf library, which outputs for each GPS entry in the video its frame, time, and difference from last time. In the beginning of the file it is inaccurate (even if reported fix and precision is high), but in timelapse basically every frame has GPS data assigned (not true for 60 fps standard video where GPS records are each second), so some gpx interpolation would be handy. Also, it would be nice feature to match timestamps from video to GPX and interpolate GPX values - however it might be a mess if a value changes each frame, so it should do some average value for few frames...

In timelapse video, the number of frames equals the number of GPS records, here is the beggining of a timelapse video:

frame, fix, precision, date, time, diff
1 3 4.12 2023-12-29 09:59:22.199 0.0
2 3 3.13 2023-12-29 09:59:33.699 11.5
3 3 3.13 2023-12-29 09:59:33.699 0.0
4 3 3.13 2023-12-29 09:59:33.799 0.1
5 3 3.13 2023-12-29 09:59:33.899 0.1
6 3 3.13 2023-12-29 09:59:34.899 1.0
7 3 3.13 2023-12-29 09:59:35.000 0.101
8 3 3.13 2023-12-29 09:59:35.000 0.0
9 3 3.13 2023-12-29 09:59:35.099 0.099
10 3 3.13 2023-12-29 09:59:36.299 1.2
11 3 3.13 2023-12-29 09:59:36.299 0.0
12 3 3.13 2023-12-29 09:59:36.399 0.1
13 3 3.13 2023-12-29 09:59:36.499 0.1
14 3 3.13 2023-12-29 09:59:37.599 1.1
15 3 3.13 2023-12-29 09:59:37.699 0.1
16 3 3.13 2023-12-29 09:59:37.699 0.0
17 3 3.13 2023-12-29 09:59:37.799 0.1
18 3 3.13 2023-12-29 09:59:38.899 1.1
19 3 3.13 2023-12-29 09:59:39.000 0.101
20 3 3.13 2023-12-29 09:59:39.099 0.099
21 3 3.13 2023-12-29 09:59:39.199 0.1
22 3 3.13 2023-12-29 09:59:40.299 1.1
23 3 3.13 2023-12-29 09:59:40.399 0.1
24 3 3.13 2023-12-29 09:59:40.399 0.0
25 3 3.13 2023-12-29 09:59:40.499 0.1
26 3 3.13 2023-12-29 09:59:41.599 1.1
27 3 3.13 2023-12-29 09:59:41.699 0.1
28 3 3.13 2023-12-29 09:59:41.799 0.1
29 3 3.13 2023-12-29 09:59:41.899 0.1
30 3 3.13 2023-12-29 09:59:43.000 1.101
31 3 3.13 2023-12-29 09:59:43.000 0.0
32 3 3.13 2023-12-29 09:59:43.199 0.199
33 3 3.13 2023-12-29 09:59:43.299 0.1
34 3 3.13 2023-12-29 09:59:43.299 0.0
35 3 3.13 2023-12-29 09:59:44.399 1.1
36 3 3.13 2023-12-29 09:59:44.599 0.2
37 3 3.13 2023-12-29 09:59:44.699 0.1
38 3 3.13 2023-12-29 09:59:44.699 0.0
39 3 3.13 2023-12-29 09:59:46.000 1.301
40 3 3.13 2023-12-29 09:59:46.099 0.099
41 3 3.13 2023-12-29 09:59:46.299 0.2
...

Maybe it is needlessly complex and constant time-factor timelapse is probably better idea anyway. Feel free to close this, if you do not plan to work on this. --time-factor should suffice for my future videos with constant time factor.

[progweb]

How do you link video frame number with time ? Can you share your script ?

Today, gpx2video doesn’t use gpmf stream except to sync video with gpx data. But I have to add support for time-lapse but accelerometer data too and more.

[fpokryvk]

Matching to the frames is the main challenge, but I have just compared the number of records that the first example here returned for my video and it was exactly the number of frames for timelapse (also the frequency of 1 second pause in GPS data is variable, that is probably decided by that auto setting). For standard 60 fps video the number of GPS records was around 1/6 of the number of frames. Also gpmf-parser states that hero 11 is recording GPS with 10Hz frequency, which matches nicely (but GPS time in the beginning of any video is inaccurate, even if fix is 3 and not bad precision, I guess it then shifts the internal clock once GPS fix is stable enough and it is fine later on). Also, gpmf-parser is in C, so it might help in this project too.

[fpokryvk]

Small update, I have recorded timewarp video, speed 15x and used --time-factor=15. Some portions of the videos matched exactly (stop on traffic lights, speed went to 0), some were 3 seconds of (so around 45 seconds of real time) - when uphill changed to downhill, time was delayed for 3 seconds... If I use my script to compute average time factor by frames (again, number of frames and the GPS records is the same number), in the beginning it varies a lot, after about 50 frames it stable in range 14-16.5 and ends up on value 15.9. So, GoPro seems to use variable time factor even if fixed value is set in the settings.

The only solution so far is to record the standard video and speed it up with ffmpeg and then --time-factor will work.

[progweb]

Thank you for your work

Time factor can be computed from gpmf data. We have to understand how is coded time-lapse factor,

Then I have to update gpx2video to read gpx AND gpmf stream during encoding.

[fpokryvk]

I am not sure if it is possible to read 2 streams (video and gpmf) as they go sequentially in MP4 file (if it was possible, it would be probably the best option).

If that is not possible, gpx2video could go through gpmf stream, each GPS5 entry contains timestamp (but first few entries are sometimes off, probably if the GoPro time was not GPS corrected yet), and then distribute the entries to frames uniformly. However, this approach would break if there are some realtime sequences in the timewarp video - I have no idea how to map video and gpmf in that case.

[fpokryvk]

Hmm, I have noticed if I export images to PNG, it goes roughly 2 FPS on my CPU. If I use tiff format, it goes 60 FPS. Then overlaying images to video with ffmpeg and QSV is around 20 FPS on my PC. If I use gpx2video to create video, it goes around 2 FPS - so faster pipeline for me is to write simple script that executes gpx2video image command and then ffmpeg.

[fpokryvk]

I managed to overlay timelapse video and finally, widget values match the video! I realized, that timelapse video has variable framerate during the process, so it was a challenge to map images to frames. Several passes over the video are required:

1. run gpx2video image command on some long enough dummy video (without created_timestamp set) with properly set --start-time and --duration
2. parse GPS data from gpmd stream and match frames to overlay images from gpx2video (create symlinks).
   GoPro timestamp in GPS records are 10s-15s off, depending on video, except the first record which matches. Probably some bug in firmware that incorrectly syncs internal clock with GPS. So, I use STMP record instead of GPS5, which carries a number of real microseconds since start of the file.
3. compute concat file of the original video - outputs duration of each frame using showinfo2concat.py as described here (first command in the Solution section)
4. finally overlay images over video, use concat data to produce image sequence of matching variable framerate (second command in the Solution section)

So, what I noticed, in timelapse video, it's possible to rely on STMP records in gpmd stream of the video, and it would be great if --start-time was overriding creation_time, even if creation_time is set in the video (it is usually a few seconds off in my case, so working override would be handy). Then, for each video frame, next STMP value can be fetched from the gpmd stream and used to sync with gpx.

I am not sure if this process would work for standard video (maybe frequency of STMP doesn't match framerate), I plan to investigate it more tomorrow.

[progweb]

Great work, I'll study and integrate it it as I'll find a moment.

[fpokryvk]

I have checked, and in standard time video, the GPS data are evenly distributed evenly every second, so there is no need to sync using metadata. I would add special option, something like --timelapse-metadata-sync which would turn on metadata parsing. But no rush, I have it now in single file and now I am around 1/5x speed :) (each second of video takes roughly 5s to process in total). Thank you for support!

[progweb]

Are you able to use hardware acceleration?

[fpokryvk]

Yes. I am using HEVC_QSV codec, but not on very powerfull CPU (laptop i7).

[progweb]

Hi,

Finally I have started to work about timelapse and mainly about GPMF parsing during video rendering.

If you run gpx2video tool in command line with options :

./gpx2video -v -g test.gpx -m test.MP4 -l layout.xml --telemetry 3 --output test.mp4 video

I have added new informations :

Time synchronization...
PACKET: 0 - PTS: 0 - TIMESTAMP: 0 ms - TIME: 2022-08-03 06:15:01 - GPS FIX: 3 - GPS TIME: 2022-08-03 06:15:01.584 - OFFSET: 0
Video stream synchronized with success (offset: 0 s)
Download map from OpenStreetMap I...
  Download tile 65 / 65 [##################################################] DONE      
Build map...
Rendering...
FRAME: 0 - PTS: 0 - TIMESTAMP: 0 ms - TIME: 2022-08-03 08:15:01
  [0] GPMF Timestamp: 81 ms - Timelapse factor: 1
  [20] GPX Time: 2022-08-03 08:16:18 Distance: 0.000 km in 0 seconds, current speed is 0.000 (valid: true)
FRAME: 1 - PTS: 3003 - TIMESTAMP: 33 ms - TIME: 2022-08-03 08:15:01
  [1] GPMF Timestamp: 415 ms - Timelapse factor: 10
  [20] GPX Time: 2022-08-03 08:16:18 Distance: 0.000 km in 0 seconds, current speed is 0.000 (valid: true)
FRAME: 2 - PTS: 6006 - TIMESTAMP: 66 ms - TIME: 2022-08-03 08:15:01
  [2] GPMF Timestamp: 749 ms - Timelapse factor: 10
  [20] GPX Time: 2022-08-03 08:16:18 Distance: 0.000 km in 0 seconds, current speed is 0.000 (valid: true)
FRAME: 3 - PTS: 9009 - TIMESTAMP: 100 ms - TIME: 2022-08-03 08:15:01
  [3] GPMF Timestamp: 1082 ms - Timelapse factor: 9
  [20] GPX Time: 2022-08-03 08:16:18 Distance: 0.000 km in 0 seconds, current speed is 0.000 (valid: true)

As you can read, for each frame, I print GPMF extract information. And I can compute timelapse factor (without using GPS data since removed into the GoPro 12)

I'm going to continue to work about it. Now I should be able to render each widget from GPMF or GPX (in adding "source" param for each widget inside the layout file). And we can compare computed speed from GPMF or GPX.

I need to test timelapse factor... is it possible to have "9" as value, or only 0.5 ; 1 ; 2 ; 5 ; 10 ; 30....

I need to test timelaspse with auto value too. Then, gpx2video take in count the timelaspse factor value.

Note: code will be re-factor soon.

[fpokryvk]

Great, thank you for the effort, I currently have python wrapper over gpx2video and ffmpeg. It works fine, but sometimes I need to tune the starting time manually by eye (+- 20 seconds). Not sure why that happens, even the GPS time with POSITION FIX seems to be noticeably off in some videos. Maybe it might be caused by external GPX, I didn't try GPMF stream from GoPro (I believe more my Garmin watch, but maybe it is better to use GPMF).

[progweb]

As far as, I'm using mainly garmin device to generate GPX file. But I have noticed that I can have 6 to 8 seconds offset between garmin and gopro. So I can't sync garmin GPX with video in using time information.

So as I render a video, I'm using gpx2video to extract gpx from gopro.
With the timesync feature, I deduce the start time (since my gopro isn't always at the good time).
Then from the lat & lon of the first point, I have the start position.
At last, I search the same position in the garmin gpx to compute the offset time between garmin and gopro.

I haven't yet the time to implement auto sync by geoposition in gpx2video, but it's coming.

Note: to look the time issue I have just recorded a gopro video in watching the garmin device.

To sum up, gopro has the good time and garmin has the good and more accurate position.

[progweb]

So, I have added a new option value:

gpx2video --time-factor auto

But need to work again for timelapse variable value. It seems to work as expected for constant value. But there are some issues with variable value :(

[progweb]

Now seems work.

I have to fix widget computed values (speed, avg speed...) but time seems ok.