Restructure releng work

[akurtakov]

This idea comes after all the drill from javadoc builds like #1439 (comment) and #1470 (comment) which show current state is simply unmaintainable.
The way to automate it and still have Platform/PDE/JDT isv docs is to build javadoc for all bundles (non-test/examples/snippets) in the Maven aggregator build and generate javadoc for all non-internal packages.
There is the issue of JDT and PDE bundles being in the same maven aggregator build with Platform bundles which drives this idea as no-go for now. If JDT and PDE projects build p2 repos on their own which Platform build references when aggregating the products and p2 repo that becomes a possiblity without changing the current aggregator output.
PDE is a single repo so it looks like best candidate to try such approach and split its build out of the platform.aggregator one to prove this as working path forward.
This would have the extra benefit of projects being able to drive their releng/reports/etc. up to their own standards.

[akurtakov]

@HannesWell @laeubi @merks WDYT?

[HannesWell]

I didn't follow the java-doc issue in detail, but could you explain how making PDE or JDT 'standalone' would help to mitigate them?

While I agree that making the projects more Independent is in general good I belive that this would be a huge task to make it right since many are coupled more than they should.
And with the current state I agree with Ed that future problems would potentially be detected later.

[akurtakov]

I didn't follow the java-doc issue in detail, but could you explain how making PDE or JDT 'standalone' would help to mitigate them?

Look at https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/eclipse.platform.common/bundles/org.eclipse.pde.doc.user/pdeOptions.txt . There are multiple problems:

• All the dependencies are manually listed with paths, which is truly fragile
• All the packages to build javadoc for are manually listed (is anyone sure that all non internal packages are listed here?)
• Javadoc is rebuilt by another job thus breakage can not be detected by verification job

The idea is to have o.e.pde.doc.user be part of pde git repo and have javadoc being generated like explained here https://maven.apache.org/plugins/maven-javadoc-plugin/examples/aggregate.html so Maven/Tycho supplies the dependencies and instead of manually listing what to include we go the other way and exclude plugins to not generate javadoc for (https://maven.apache.org/plugins/maven-javadoc-plugin/examples/exclude-package-names.html) e.g. *.internal

While I agree that making the projects more Independent is in general good I belive that this would be a huge task to make it right since many are coupled more than they should. And with the current state I agree with Ed that future problems would potentially be detected later.

PDE should already have its own download space (if not that's a bug) so having a p2 site module and such a javadoc module in it could be done without disrupting the SDK at all. Only when these are in good shape change and properly published change to aggregator to refer rather than build could be done.

[vogella]

I would like to see PDE and JDT independent of platform. That would give pde and JDT more flexibility. And if we could leverage language server protocol for certain tasks in pde or jdt that would be awesome.

If we go into this direction it would be IMHO great if the integration could use standard Maven functionality instead of P2 repos. Most of my RCP clients try getting rid of this Eclipse specific technology and I believe using common Maven standards would also help the Eclipse project.

[mickaelistria]

+1, it would also allow Platform and JDT to release at their own pace, as JDT has some external constraints (Java release schedule) that makes that the usual SimRel calendar followed by Platform may not be best. This would allow to release JDT normally without making a patch release.

[HannesWell]

If we go into this direction it would be IMHO great if the integration could use standard Maven functionality instead of P2 repos. Most of my RCP clients try getting rid of this Eclipse specific technology and I believe using common Maven standards would also help the Eclipse project.

Are the artifacts already published to maven-Central not sufficient?

[akurtakov]

@HannesWell One more reason to have PDE as separate project #1476 - to prevent the cyclic dependency between Platform and LSP4E and still allow PDE to use LSP editors.

[akurtakov]

Now that I remember that LSP4J uses guava things are becoming even funnier. EMF is very lean on external deps (none IIRC?) which makes things significantly easier.

[laeubi]

What I'm always wondered: If guava is that problematic, why not remove guava at all, what is the real benefit of it for lsp4x? Guava is Apache-2.0 license so even if there is "something" one cannot replace, maybe make a copy in a private package of it would be better alternative?

[merks]

This looks beyond scary to me:

#1476

[akurtakov]

I have spend enough time removing guava from many projects (non-eclipse.org ones too) as part of Fedora Java stack work. This is very timeconsuming job due to small behavior differences here and there, projects wanting to keep supporting ancient Java versions, inactive communities, if it's not broken don't touch it and what not.
I still do that when I spot it but overall I lost hope that getting rid of guava in general is easy task to achieve.

[HannesWell]

I assume that lsp4j/e is active enough to replace guava. :)
with modern Java there are IMHO only a few use-cases left where guava is irreplaceable.

But yes avoiding the cycle makes sense. Especially since lsp4e/j is probably by far not mature and stable as EMF is.

[HannesWell]

In general I think this is doable with PDE and/or JDT having its own target (containing the latest snapshots/I-builds of everything else in the SDK third party libs) and an own repo project.

Jenkins also allows to trigger a build if another one completes(not sure If this works out of the box for different Jenkins Controllers, but there will certainly be solutions), so that could be used to check if everything still compiles after a TLP on which PDE/JDT depends has changed. Or if this leads to too many builds PDE/JDT could at least be build once a night.

We could then also rethink the I-builds.
Eventually they could be changed to only collect the artifacts from equinox/Platform/p2/JDT/PDE and only run all the integration-tests.
Given we have a nice acyclic tree of dependencies between the projects (then we could also Trigger the nightly builds of each project one after another accordingly to the topological sorting).

@merks could you provide a graph of the dependencies between the Eclipse TLPs? IIRC there are some cycle over repo boundaries between platform and p2 or equinox?
It would be interesting to know If there are more or if we have a nice tree, which would IMHO be very imporant for this effort.

[mickaelistria]

I've submitted eclipse-pde/eclipse.pde#826 as a very first baby step for PDE.