Please also add pokestops as they also move around or get deleted some times.

It would also be nice a wh notification when a gym or pokestop is removed.

@friscoMad pokestop dont have a last_scanned so removal this way is not possible

@elincognito that is not the scope of this PR

But the feature works and I saw old gyms removed at first database scrub

Ok some new thoughts pokestops have the last_updated field but the problem is that stops are not constantly updating like gyms so deleting based on this field would likely lead to false positives, they are only updated when a lure appears.

Anyway we could delete them anyway and let the map add them again, due to the long range for gyms/stops I don't see it a real problem but I would set the delta to at least 1 week to avoid doing this every day, anyway that would lead to having a lot of missing stops for some minutes just 1 week after the map was first started.

And on another issue, raids, gym members and gym pokes should also be cleaned with gym/gym details

Temporarily on hold while we discuss some SQL/SQLite things on Discord that are related to this PR.

Also, when removing a gym we should also remove the pokemon assigned to those gyms. I am seeing orphaned rows in gymmember and gympokemon.

I think there should be at least an option to enable/disable gym-details removal.
I for example run gym-info only once for every new area to retrieve gym names. This would require a new gym-info scan every 24 hours.

@acassani91 gym names are from GMO not gym_info you will get those if you scan within 1000m of a gym. This PR is to remove gym details that dont get scanned because you no long scan the same area or a gym is removed.

I think the only issue is with pokestops as we are not inserting them continously as we do with everything else, we would need to change the code to upsert everything instead of querying and only inserting the new ones. But I think it would be better to change the code to remove the missing ones instead of updating all and them removing the one that are not three anymore in a thread. That way we will avoid a lot of DB writes for no reason.

There is a tool out there called Worldpoole that queries the RM database to generate interesting reports using the data. I just noticed that this shows historical data from the database and so if we prune it after 7 days it would impact that tool and the people that use it. This PR has moved from just hiding gyms that are no longer in the game towards more database pruning. Need to ensure we consider all aspects of those decisions.

Here is a snippet from the trainer report:

@tomballgithub Solved with #2305. If someone doesn't want data to be cleaned, they shouldn't enable it.

Those who want more fine-grained control should look into setting up automated backups or MySQL triggers or a few options to control data removal (e.g. age of data).

This lets RM take care of its own needs (removing old data to reduce/limit performance impact and deadlocks) while the host has other options if they want to keep all old data.

my problem with this:

Users have asked me to have a static map of the gyms available in the country.
I did a hex scan to find them and deleted the trainers|pokemon inside to have static points showing unassigned gyms.

would it be a horrible idea to have some kind of config switch to not delete old gyms if user opts for it?

Can a static map not be e.g. an image file?

I also have a much larger, static area with stops and gyms. It's about 2 counties, 50 000 square kilometers. A simple static picture wont really do it justice. Not to mention people who wanna geek out and see gym names.

I accept that when pokestops go become gyms, a cleaner needs to take care and remove such things (same when things get removed). But I'm scared to loose my "static" info that I scanned once and forgot.

Some of these last comments are interesting. Maybe the solution would be to have configurable timeframes for the pruning of each item (although I realize there is opposition to having more configuration options)

I don't see the issue here if you want static maps just create another DB with them and do whatever you want with them.
There is no reason reason to mix static and dynamic data in the same db, just enable clean in the dynamic instance and we are all fine.

If someone wants to do things like update gyms/stops once a months and then disable gym scanning/stop scanning during regular work while using the same db for everything, then just the solution is to not enable db-clean and do the cleanup by hand, trigger, cron job or whatever, he should be able to do so.

Adding a lot of new config options just for fringe use cases makes no sense.

@friscoMad I don't agree with you that making a separate RM web instance and database to have static pokestop and gym info is trivial. ("just make a copy of the db").

That said, I can agree with you that there's maybe little to no practical use for having a bunch of more options just for some fringe use case like myself and @KartulUdus are discussing.

I am giving this PR the thumbs up, after a rebase.

This PR is being closed in favor of #2383 but the discussion is still relevant. #2383 adds a few options for better control of data removal (including an option to disable certain cleaners).