Fix NooBaa usagereports and endpointgroupreports collections

[tangledbytes]

Explain the changes

This PR:

1. Adds cleanup for endpointgroupreports.
2. Fixes invalid endpointgroupreports data being stored in the tables which was slowing down the queries.
3. Fixes the outdated indexes. NOTE: Even after trying quite a bit, I cannot get PG to reliably use the index. Sometimes it does, sometime it does not.

• Doc added/updated
• Tests added

[tangledbytes]

NOTE: I have split the PR into multiple commits as I am not sure what can we backport and what we cannot. Somethings are crucial for backporting IMO like cleanups and fixing the invalid data being stored in the table.
Indexes can be skipped because they affect the performance, not the correctness.

[dannyzaken]

I don't have any comments from a technical perspective.

In general, I really don't like the way we handle and store stats. Some of the stats we store are never even used anymore (e.g., leftovers from UI screens). Many are exported as metrics that I'm not sure are used by any actual user.

We should rethink and better design the way we handle stats. Of course, it's out of the scope of this PR, but we should start taking small steps and remove unnecessary information.
For example, endpoints_group_report - I did a quick search in the code, and it seems it isn't used at all:

get_endpoint_group_reports is called in 2 flows:

1. get_endpoints_history - This was used for one of the UI graphs, and AFAIK it isn't used anymore
2. _get_endpoint_groups - this is only called by read_system. There are a few internal flows in noobaa-core that call read_system (this by itself should be reconsidered), but it doesn't seem to use endpoint_groups field.

@tangledbytes @nimrod-becker @guymguym I think it's safe to remove endpoints_group_report. WDYT?

[tangledbytes]

@dannyzaken wow, you are right. I stopped my search at read_system call. Yes, it doesn't seem like we utilize this.

I have two questions:

1. Can we remove the collection and backport the removal as well?
2. If we can't backport the removal then should this be merged and backported (so that the cleanup is there) and then a new PR be created and merged which removes the collection from master consequently future NooBaa releases?

cc: @nimrod-becker @guymguym

[dannyzaken]

@tangledbytes I think that there shouldn't be a problem to backport it. the changes should be fairly small
@nimrod-becker @guymguym, any opinions?

[tangledbytes]

@dannyzaken @nimrod-becker @guymguym @jackyalbo, I want to revive this PR. I think I am seeing a related issue again with the same customer.

[nimrod-becker]

@dannyzaken @nimrod-becker @guymguym @jackyalbo, I want to revive this PR. I think I am seeing a related issue again with the same customer.

OFC, lets have it reviewed, merged to master and we can work on backporting up to 4.14

[github-actions]

This PR had no activity for too long - it will now be labeled stale. Update it to prevent it from getting closed.

[nimrod-becker]

keep alive