Dokka Gradle Plugin: Implement Gradle Worker Daemon for Dokka generation

[aSemy]

Refactor the way dokka-gradle executes a Dokka generation task to use a Gradle Worker Daemon. The Worker is configured to use the same classpath as the existing reflection-based process.

Using a Worker Daemon completely replaces the existing 'reflection magic'. I think this is a lot more simple and easier to understand than the previous method of using reflection to create an instance.

This PR is a further improvement upon #2709, which required a main method. Using the Gradle Worker API means the dokka-core code can be called idiomatically. However, I am not sure about the compatibility with Gradle versions. #2709 might be more compatible...

Benefits:

• idiomatic Kotlin
• idiomatic Gradle
• no reflection magic
• easier to configure/debug/test
• no API changes for users - this should 'just work'

I haven't tested extensively so there might be some breaking changes. Of particular note:

• is the logging output still present and as expected
• is the Worker API compatible with the versions of Gradle that Dokka must support?

This PR includes

• apply kotlin-dsl plugin #2702
• Gradle Plugin: Expose finalizeCoroutines as Gradle task option #2759

[qwwdfsad]

Thanks!
The contribution indeed looks very promising (reflection magic is something we strive to get away from), but unfortunately I can build neither coroutines nor Ktor with this change:

> A failure occurred while executing org.jetbrains.dokka.gradle.workers.DokkaGeneratorWorker
   > org.jetbrains.kotlin.cli.jvm.compiler.KotlinCoreEnvironment$Companion.createForProduction(Lcom/intellij/openapi/Disposable;Lorg/jetbrains/kotlin/config/CompilerConfiguration;Lorg/jetbrains/kotlin/cli/jvm/compiler/EnvironmentConfigFiles;)Lorg/jetbrains/kotlin/cli/jvm/compiler/KotlinCoreEnvironment;

I've also verified that without your patch, coroutines can be built with Dokka from master branch. Could you please take a look?

[qwwdfsad]

It seems like Gradle Worker is missing some dependencies

[aSemy]

Thanks for trying it! Yes it's possible that dependencies might be missing now that the Dokka worker executes in a separate process. It should be easy to add them back in though.

(Missing dependencies isn't a big problem, even if it's annoying. Since Dokka creates a Gradle Configuration anyone would be able to fix this themselves, should a similar problem arise.)

It's been a while since I've looked at this.. I can try figuring out what's missing. Do you have the full stack trace? Are there any clues in there?

UPDATE: Ah I see, it's in the GitHub Actions

• coroutines https://github.com/Kotlin/dokka/actions/runs/3567172380/jobs/5994535855#step:4:4592
• serialization https://github.com/Kotlin/dokka/actions/runs/3567172380/jobs/5994536797#step:4:4032

[qwwdfsad]

I've checked it manually

Stacktrace:

Caused by: java.lang.NoSuchMethodError: org.jetbrains.kotlin.cli.jvm.compiler.KotlinCoreEnvironment$Companion.createForProduction(Lcom/intellij/openapi/Disposable;Lorg/jetbrains/kotlin/config/CompilerConfiguration;Lorg/jetbrains/kotlin/cli/jvm/compiler/EnvironmentConfigFiles;)Lorg/jetbrains/kotlin/cli/jvm/compiler/KotlinCoreEnvironment;
        at org.jetbrains.dokka.analysis.AnalysisEnvironment.createCoreEnvironment(AnalysisEnvironment.kt:107)
        at org.jetbrains.dokka.analysis.EnvironmentAndFacadeKt.createEnvironmentAndFacade(EnvironmentAndFacade.kt:30)
        at org.jetbrains.dokka.analysis.KotlinAnalysisKt.KotlinAnalysis(KotlinAnalysis.kt:13)
        at org.jetbrains.dokka.analysis.KotlinAnalysisKt.KotlinAnalysis$default(KotlinAnalysis.kt:11)
        at org.jetbrains.dokka.base.DokkaBase$defaultKotlinAnalysis$2$1.invoke(DokkaBase.kt:191)
        at org.jetbrains.dokka.base.DokkaBase$defaultKotlinAnalysis$2$1.invoke(DokkaBase.kt:43)
        at org.jetbrains.dokka.plugability.LazyEvaluated.get$core(LazyEvaluated.kt:8)
        at org.jetbrains.dokka.plugability.DokkaContextConfigurationImpl.single(DokkaContext.kt:159)
        at org.jetbrains.dokka.base.translators.psi.DefaultPsiToDocumentableTranslator.<init>(DefaultPsiToDocumentableTranslator.kt:818)
        at org.jetbrains.dokka.base.DokkaBase$psiToDocumentableTranslator$2$1.invoke(DokkaBase.kt:69)
        at org.jetbrains.dokka.base.DokkaBase$psiToDocumentableTranslator$2$1.invoke(DokkaBase.kt:43)
        at org.jetbrains.dokka.plugability.LazyEvaluated.get$core(LazyEvaluated.kt:8)
        at org.jetbrains.dokka.plugability.DokkaContextConfigurationImpl.actions(DokkaContext.kt:164)
        at org.jetbrains.dokka.plugability.DokkaContextConfigurationImpl.get(DokkaContext.kt:147)
        at org.jetbrains.dokka.base.generation.SingleModuleGeneration.translateSources(SingleModuleGeneration.kt:113)
        at org.jetbrains.dokka.base.generation.SingleModuleGeneration.access$translateSources(SingleModuleGeneration.kt:21)
        at org.jetbrains.dokka.base.generation.SingleModuleGeneration$createDocumentationModels$1$invokeSuspend$$inlined$parallelMap$1$1.invokeSuspend(parallelCollectionOperations.kt:17)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)

Coroutines branch (note: Dokka is published locally via ./gradlew publishToMavenLocal with the corresponding version): https://github.com/Kotlin/kotlinx.coroutines/tree/dokka-update, command: ./gradlew cleanDokkaHtmlMultiModule dokkaHtmlMultiModule --no-daemon --stacktrace

Dokka is published from your PR, with two changes -- version for publishing and commented out Dispatchers.shutdown

[aSemy]

I get an error when looking at AnalysisEnvironment about missing dependencies...

I am very confused why this might be. I don't think this PR changes the dependencies.

UPDATE: This error is confusing, but unrelated. I think it's caused by the kotlin-analysis subproject only depending on the 'shadow' configuration, which does not expose the API dependencies. The error would probably go away if I did a full rebuild, then the shadowed jar would contain the missing dependencies.

dokka/kotlin-analysis/build.gradle.kts

Line 11 in f7691fe

api(project("compiler-dependency", configuration = "shadow"))

[aSemy]

@qwwdfsad I think it's fixed now, can you try again please?

The problem was that I didn't add the Dokka plugins to the worker's classpath.
c829043

[qwwdfsad]

The error would probably go away if I did a full rebuild, then the shadowed jar would contain the missing dependencies.

Yeah, you have to explicitly reimport your project into IDEA