Reporting progress of get_significant_points_gdf

[marklit]

If I feed up to 100K points into MP, get_significant_points_gdf will finish in a few minutes. I'm keen to potentially feed upwards 40M points. Is there some way that get_significant_points_gdf could be passed a function and it would report its progress?

I usually use https://rich.readthedocs.io/en/stable/progress.html for tracking progress in long-running Python scripts.

[bamacgabhann]

Use tqdm?

https://github.com/tqdm/tqdm

[marklit]

tqdm's API looks much like rich's. The issue is how to tie this into MP's aggregation calls. There is no iterator exposed that I could use to keep track of what's happening and I can't pass an object into the aggregation call to report on its progress either.

It took all night for my 2020 MBP to generate the generalised route maps for the top 25 airlines in ADSB.lol's dataset. It would be great to know roughly how long is left on jobs like this. https://tech.marksblogg.com/global-flight-tracking-adsb.html#generalising-routes

[anitagraser]

Nice blog post, @marklit.

I'm keen to potentially feed upwards 40M points.

Heads up: The TrajectoryCollection from these 40M points has to fit into your RAM. The script will crash otherwise.

Looking at the current implementation, a meaningful progress indicator will be challenging.
The first 50% could be showing the progress of

movingpandas/movingpandas/trajectory_aggregator.py

Line 63 in 6a060c3

self.significant_points = self._extract_significant_points()

the remaining 50% would be

movingpandas/movingpandas/trajectory_aggregator.py

Line 70 in 6a060c3

self.flows = self._compute_flows_between_clusters()

Both of these happen on init

[marklit]

Is there anywhere deeper when records are iterated over one at a time? This could be a place to add a hook to a progress counter.

[anitagraser]

Records are iterated for each trajectory individually. It would be hard to keep track of the overall progress for the whole trajectory collection