Remember referenced repositories as system repositories instead of user

[laeubi]

Currently there is a quite surprising behavior, that if a user adds an
updatesite that contains a referenced repository, that after an
update-check additional repositories are visible. Even worse these are
now used additionally to check for updates and if they contain other
references these are also added and so on. This can result not only in a
long list of sites the user never has added and has no clue where they
are coming from but even pulling in unwanted or conflicting updates.

Before this change

Here is an example with the m2e update site (https://download.eclipse.org/technology/m2e/snapshots/latest/):

In the first step there is only this single site:

Now I go to Help > Check For Updates and after this completes I end up with the following list (10 sites):

Now I repeat and perform Help > Check For Updates and after this completes I end up with the following list (12 sites):

After this changes

This now distinguishes two cases:

1. A repository is discovered by the RepositoryAction it is handled as a
   user added repository and becomes visible
2. A repository is discovered by a reference in that case it is handled
   as a system repository and not becomes visible

that way the list of user visible repositories stay clean from
referenced repositories and unexpected side effects on update checks.

FYI @merks @HannesWell @jonahgraham

[github-actions]

Test Results

  375 files  ±0    375 suites  ±0   41m 35s ⏱️ -55s
1 893 tests ±0  1 890 ✅ ±0  3 💤 ±0  0 ❌ ±0 
6 679 runs  ±0  6 670 ✅ ±0  9 💤 ±0  0 ❌ ±0 

Results for commit 6ac40fb. ± Comparison against base commit 4ed9c3e.

♻️ This comment has been updated with latest results.

[merks]

I see that org.eclipse.equinox.internal.p2.updatechecker.UpdateChecker.checkForUpdates(String, IQuery) calls org.eclipse.equinox.internal.p2.updatechecker.UpdateChecker.getAvailableRepositories() which calls org.eclipse.equinox.p2.repository.IRepositoryManager.getKnownRepositories(int) with argument org.eclipse.equinox.p2.repository.IRepositoryManager.REPOSITORIES_ALL. As such, is accurate to conclude that not showing the repositories to the user actually has no impact on how Check for Updates will behave after the repo is added (discovered), regardless of whether the repo is marked system or not? If so, this change does not actually address the "Even worse..." aspect described in the first comment. If not, what am I overlooking?

It's rather disconcerting that there are repositories that will be contacted and used but you can't see , remove, nor disable them....

[laeubi]

I can't tell much about UpdateChecker and it is possible that there are more issues lurking around, but this is about "Check For Updates" function in the IDE what uses UpdateHandler (!) and that one is defiantly consistent with the UI and calls it with flags = 2 (REPOSITORIES_NON_SYSTEM) I even verified with the debugger that there is no other call involved.

If one wants to see "everything" I'm sure the UI can be enhanced to support that use-case but this really focus on the part that I don't get new user repositories as a side effect of performing a check for updates.

[laeubi]

A possible UI might look like this (or as part of the filter bar):

but as said that's nothing I'd like to address with this PR.

[merks]

If you checked in the debugger then I’m reassured and this looks like a nice change. I had the same idea about a check box to show system repositories; it would help debug.

[merks]

In case someone/something relies on the old behavior, which I doubt, one might use a system property to specify true versus the default of false.

[laeubi]

If such thing is required I think it should (again) be an UI thing so when I add the site I can select to also add references and maybe select what I want to have added.

[merks]

It’s not required. It’s just a thought.

[laeubi]

It’s not required. It’s just a thought.

I now added p2.metadata.repository.reference.system that defaults to true

[merks]

Kudos for investing your time on these improvements! 🏅

[laeubi]

No problem, this was an issue for too long now and seem to regularly cause confusions.