Gradle Kotlin script plus other stuff #3167

elect86 · 2024-05-08T12:39:08Z

Description

This PR convert Gradle Groovy scripts to Kotlin, which is also recommended by Gradle, being a statically typed language

this change is backward compatible

Interesting edge cases to note here:

removed any cross-project configuration (allprojects and subprojects), which add complexity and burden, preventing also optimizations, like parallel project builds and configuration on demand (more here and here)
using buildSrc to abstract imperative logic and share logic via pre-compiled script plugins
centralized management of dependencies in gradle/libs.versions.toml
version catalog
safe project builders

There a couple of TODOs which I still have to figure it out, but the 95% work is done and there for you to start looking at it so that I can already get some early feedbacks from you, folks

Also, why are you disabling the Gradle Metadata generation?

# Conflicts: # engines/pytorch/pytorch-jni/build.gradle # engines/pytorch/pytorch-native/build.gradle # gradle.properties # gradle/wrapper/gradle-wrapper.properties

frankfliu · 2024-05-08T19:03:06Z

docs/README.md

@@ -63,16 +63,16 @@ Note: when searching in JavaDoc, if your access is denied, please try removing t
 ## Extensions and utilities

 - [Android support](../android/README.md)
- [Audio support](../extensions/audio/README.md)


Why we need to rename extensions folder?

because the typesafe project accessor clashes with a getter called extensions, so we either rename that directory or we disable the typesafe accessor

Let's disable this feature for now. Since this project is experimental, we can create a ticket to gradle and see if they can adda flag to change "extension" folder name

frankfliu

Let's focus on gradle changes in this PR. Would you please remove these Kotlin tests?

elect86 · 2024-05-13T13:39:37Z

Let's focus on gradle changes in this PR. Would you please remove these Kotlin tests?

Absolutely, sorry, I already meant to do that, but I forgot

# Conflicts: # extension/tokenizers/src/main/python/arg_parser.py # extension/tokenizers/src/main/python/djl_converter/__init__.py # extension/tokenizers/src/main/python/djl_converter/arg_parser.py # extension/tokenizers/src/main/python/djl_converter/fill_mask_converter.py # extension/tokenizers/src/main/python/djl_converter/huggingface_converter.py # extension/tokenizers/src/main/python/djl_converter/huggingface_models.py # extension/tokenizers/src/main/python/djl_converter/metadata.py # extension/tokenizers/src/main/python/djl_converter/model_zoo_importer.py # extension/tokenizers/src/main/python/djl_converter/question_answering_converter.py # extension/tokenizers/src/main/python/djl_converter/safetensors_convert.py # extension/tokenizers/src/main/python/djl_converter/sentence_similarity_converter.py # extension/tokenizers/src/main/python/djl_converter/shasum.py # extension/tokenizers/src/main/python/djl_converter/text_classification_converter.py # extension/tokenizers/src/main/python/djl_converter/token_classification_converter.py # extension/tokenizers/src/main/python/djl_converter/zip_utils.py # extension/tokenizers/src/main/python/fill_mask_converter.py # extension/tokenizers/src/main/python/huggingface_converter.py # extension/tokenizers/src/main/python/huggingface_models.py # extension/tokenizers/src/main/python/metadata.py # extension/tokenizers/src/main/python/model_zoo_importer.py # extension/tokenizers/src/main/python/question_answering_converter.py # extension/tokenizers/src/main/python/safetensors_convert.py # extension/tokenizers/src/main/python/sentence_similarity_converter.py # extension/tokenizers/src/main/python/setup.py # extension/tokenizers/src/main/python/shasum.py # extension/tokenizers/src/main/python/text_classification_converter.py # extension/tokenizers/src/main/python/token_classification_converter.py # extension/tokenizers/src/main/python/zip_utils.py # extensions/tokenizers/src/main/python/arg_parser.py # extensions/tokenizers/src/main/python/fill_mask_converter.py # extensions/tokenizers/src/main/python/huggingface_converter.py # extensions/tokenizers/src/main/python/huggingface_models.py # extensions/tokenizers/src/main/python/metadata.py # extensions/tokenizers/src/main/python/model_zoo_importer.py # extensions/tokenizers/src/main/python/question_answering_converter.py # extensions/tokenizers/src/main/python/safetensors_convert.py # extensions/tokenizers/src/main/python/sentence_similarity_converter.py # extensions/tokenizers/src/main/python/shasum.py # extensions/tokenizers/src/main/python/text_classification_converter.py # extensions/tokenizers/src/main/python/token_classification_converter.py # extensions/tokenizers/src/main/python/zip_utils.py # gradle.properties

elect86 · 2024-05-13T14:07:14Z

I re-sync'ed with the upstream and re-ordered the versions in libs.versions.toml alphabetically

frankfliu · 2024-05-13T14:26:41Z

settings.gradle.kts

@@ -0,0 +1,22 @@
+rootProject.name = "djl"
+
+val reserved = listOf("gradle", "buildSrc")


I prefer list all project here, it can also treat as an project index file.

I prefer list all project here

Ok, as you wish

it can also treat as an project index file.

What do you mean, exactly?

frankfliu · 2024-05-13T14:27:29Z

gradle/libs.versions.toml

+#
+# This file has been generated by Gradle and is intended to be consumed by Gradle
+#
+[metadata]


Since this is generated file, do we need to check in this file?

it was generated, but then I modified to fill in the needed libraries, I'll clean up that original comment

frankfliu · 2024-05-13T14:28:39Z

gradle.properties

-mxnet_version=1.9.1
-pytorch_version=2.2.2
-tensorflow_version=2.10.1
+#djl_version=0.28.0


Where these version will be defined?

in the libs.versions.toml above :)

# Conflicts: # engines/pytorch/pytorch-engine/build.gradle # extension/tokenizers/src/main/python/djl_converter/model_converter.py

- cleaned libs.versions.toml comment - removed, for the moment, type-safe projects accessors due to bug

elect86 · 2024-05-14T08:39:57Z

Addressed last wishes plus re-sync

frankfliu · 2024-05-14T15:28:32Z

I really appreciate your contribution. But I ran into several failures:

not able to compile on Mac M1 machine and windows: https://github.com/deepjavalibrary/djl/actions/runs/9076365418/job/24956766906?pr=3167#step:6:42
Not able compile bom project:

cd bom
gradle build

> Configure project :
e: file:///Users/lufen/source/djl/bom/build.gradle.kts:18:13: Unresolved reference. None of the following candidates is applicable because of receiver type mismatch: 
public val TaskContainer.projects: TaskProvider<ProjectReportTask> defined in org.gradle.kotlin.dsl
...

continuous build failed: https://github.com/deepjavalibrary/djl/actions/runs/9076365418/job/24956767288?pr=3167#step:15:29

fixed integration:compileJava compilerArgs removal

# Conflicts: # integration/build.gradle

frankfliu · 2024-05-14T18:20:10Z

@elect86

Still facing a few issues:

Usage of layout.buildDirectory is wrong, it point to DirectoryProperty instead of Directory
still got invalid flag: --release error

elect86 · 2024-05-14T21:01:49Z

the sourceCompatibility issue is fixed now, I switched from the toolchain to release option and seems fine (more here)

the error we are seeing now it's another one, which I'm gonna hunt down next

elect86 · 2024-05-15T10:03:25Z

so, I can reproduce locally

interesting to notice, I ran gradlew clean, then ./gradlew -Dai.djl.default_engine=OnnxRuntime :integration:test

same error:

elect@elect-NUC11PHi7:~/IdeaProjects/djl$ ./gradlew -Dai.djl.default_engine=OnnxRuntime :integration:test

Task :extensions:tokenizers:processResources
prebuilt or cached file found for win-x86_64/libwinpthread-1.dll
prebuilt or cached file found for win-x86_64/libgcc_s_seh-1.dll
prebuilt or cached file found for win-x86_64/libstdc%2B%2B-6.dll
prebuilt or cached file found for win-x86_64/tokenizers.dll
prebuilt or cached file found for linux-x86_64/libtokenizers.so
prebuilt or cached file found for linux-aarch64/libtokenizers.so
prebuilt or cached file found for osx-x86_64/libtokenizers.dylib
prebuilt or cached file found for osx-aarch64/libtokenizers.dylib
Downloading PyTorch model zoo metadata: fill_mask
Downloading PyTorch model zoo metadata: question_answer
Downloading PyTorch model zoo metadata: text_classification
Downloading PyTorch model zoo metadata: text_embedding
Downloading Rust model zoo metadata: text_embedding
Downloading PyTorch model zoo metadata: token_classification

Task :engines:ml:xgboost:processResources
Downloading https://publish.djl.ai/xgboost/2.0.3/jnilib/linux/aarch64/libxgboost4j.so

Task :engines:ml:xgboost:jar FAILED

FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':engines:ml:xgboost:jar'.
Cannot expand ZIP '/home/elect/IdeaProjects/djl/api/build/classes/java/main' as it is not a file.

I ran the test one another time and now I get

elect@elect-NUC11PHi7:~/IdeaProjects/djl$ ./gradlew -Dai.djl.default_engine=OnnxRuntime :integration:test

Task :engines:mxnet:jnarator:compileJava
/home/elect/IdeaProjects/djl/engines/mxnet/jnarator/build/generated-src/antlr/main/ai/djl/mxnet/jnarator/parser/CParser.java:190: warning: [this-escape] possible 'this' escape before subclass is fully initialized
_interp = new ParserATNSimulator(this,_ATN,_decisionToDFA,_sharedContextCache);
^
/home/elect/IdeaProjects/djl/engines/mxnet/jnarator/build/generated-src/antlr/main/ai/djl/mxnet/jnarator/parser/CLexer.java:160: warning: [this-escape] possible 'this' escape before subclass is fully initialized
_interp = new LexerATNSimulator(this,_ATN,_decisionToDFA,_sharedContextCache);
^
2 warnings

Task :engines:onnxruntime:onnxruntime-engine:processResources
Downloading model zoo metadata: fill_mask
Downloading model zoo metadata: question_answer
Downloading model zoo metadata: text_classification
Downloading model zoo metadata: text_embedding
Downloading model zoo metadata: token_classification

Task :engines:pytorch:pytorch-jni:processResources
Downloading https://publish.djl.ai/pytorch/2.2.2/jnilib/0.28.0/linux-x86_64/cpu/libdjl_torch.so
Downloading https://publish.djl.ai/pytorch/2.2.2/jnilib/0.28.0/linux-x86_64/cpu-precxx11/libdjl_torch.so
Downloading https://publish.djl.ai/pytorch/2.2.2/jnilib/0.28.0/linux-aarch64/cpu-precxx11/libdjl_torch.so
Downloading https://publish.djl.ai/pytorch/2.2.2/jnilib/0.28.0/osx-aarch64/cpu/libdjl_torch.dylib
Downloading https://publish.djl.ai/pytorch/2.2.2/jnilib/0.28.0/win-x86_64/cpu/djl_torch.dll
Downloading https://publish.djl.ai/pytorch/2.2.2/jnilib/0.28.0/linux-x86_64/cu121/libdjl_torch.so
Downloading https://publish.djl.ai/pytorch/2.2.2/jnilib/0.28.0/linux-x86_64/cu121-precxx11/libdjl_torch.so
Downloading https://publish.djl.ai/pytorch/2.2.2/jnilib/0.28.0/win-x86_64/cu121/djl_torch.dll
Downloading https://publish.djl.ai/pytorch/2.2.2/jnilib/0.28.0/osx-x86_64/cpu/libdjl_torch.dylib

Task :integration:test FAILED

Gradle suite > Gradle test > ai.djl.CoverageTest > test FAILED
java.lang.NoClassDefFoundError at CoverageTest.java:34
Caused by: java.lang.ClassNotFoundException at CoverageTest.java:34

Gradle suite > Gradle test > ai.djl.integration.IntegrationTests > runIntegrationTests FAILED
java.lang.NoClassDefFoundError at IntegrationTests.java:56
Caused by: java.lang.ClassNotFoundException at IntegrationTests.java:56

elect86 · 2024-05-15T10:04:34Z

running ml:xgboost:jar after a clean lead to the same not a file error

elect86 · 2024-05-15T10:21:47Z

Ok, I fixed the first error

now I have both local and CI crashing with the same last error above

codecov-commenter · 2024-05-20T16:47:38Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 63.53%. Comparing base (5c691c0) to head (f9abfc2).
Report is 20 commits behind head on master.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3167      +/-   ##
============================================
- Coverage     68.41%   63.53%   -4.89%     
+ Complexity     7027     6451     -576     
============================================
  Files           697      697              
  Lines         32759    32790      +31     
  Branches       3406     3416      +10     
============================================
- Hits          22413    20833    -1580     
- Misses         8737    10380    +1643     
+ Partials       1609     1577      -32

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

elect86 added 3 commits May 8, 2024 14:14

first sketch

979f32f

Merge remote-tracking branch 'origin/master'

6c793ec

# Conflicts: # engines/pytorch/pytorch-jni/build.gradle # engines/pytorch/pytorch-native/build.gradle # gradle.properties # gradle/wrapper/gradle-wrapper.properties

merged

7c0ddd5

elect86 requested review from zachgk, frankfliu and a team as code owners May 8, 2024 12:39

frankfliu reviewed May 8, 2024

View reviewed changes

frankfliu reviewed May 10, 2024

View reviewed changes

elect86 added 2 commits May 13, 2024 15:41

removed kotlin prototyping

e55b2de

frankfliu reviewed May 13, 2024

View reviewed changes

elect86 added 2 commits May 14, 2024 10:36

Merge remote-tracking branch 'upstream/master'

60e3213

# Conflicts: # engines/pytorch/pytorch-engine/build.gradle # extension/tokenizers/src/main/python/djl_converter/model_converter.py

- re-sync

878cc9b

- cleaned libs.versions.toml comment - removed, for the moment, type-safe projects accessors due to bug

elect86 added 4 commits May 14, 2024 19:15

removed deprecated URL constructor

bd830eb

fixed integration:compileJava compilerArgs removal

included bom

01c10a8

Merge remote-tracking branch 'refs/remotes/upstream/master'

4b93ae9

# Conflicts: # integration/build.gradle

sync'ed

c22fc0e

elect86 added 7 commits May 14, 2024 20:29

fixed testing:compileJava

3d5df34

fixing buildDirectory

6a3e5dd

fixing sensitive arch issue, thanks @frank

beef9e3

trying to fix CI

f7378a5

trying to fix CI

7fad8d0

trying to fix CI

3f1f5e0

trying to fix CI

f62daa9

elect86 added 3 commits May 15, 2024 07:48

fixing release for mxnet:jnarator

13e6dc4

Test ONNX Runtime stacktrace

6200833

testing CI

d0eb5c8

testing CI

f9b1cad

elect86 and others added 11 commits May 15, 2024 13:27

testing CI

d40f272

cleaner check.gradle.kts

40e5290

fixed, thanks to @jjohannes for spotting the manifest issue

64248ce

fixed checkstyle error, one missing doc

3640fa0

Fixes OnnxRuntime unit test

d5d0095

Fixes code format version

818af54

Fixes checkstyle issues

3498b85

[ci] Pickup version from libs.versions.toml

f07dd90

debugging duplicate file destination

cab9457

Updates libs.versions.toml

96d5d5d

Fixes spotbugs, checkstyles and PMD errors

f9abfc2

frankfliu force-pushed the master branch from f3431d3 to f9abfc2 Compare May 20, 2024 16:10

frankfliu force-pushed the master branch 3 times, most recently from 9a25da0 to ace47ca Compare May 21, 2024 14:13

Reformat build.gradle.kts file and finish migration

897c1cf

frankfliu force-pushed the master branch from ace47ca to 897c1cf Compare May 21, 2024 15:29

frankfliu and others added 3 commits May 21, 2024 09:26

Merge branch 'master' into kotlin

7abca06

migrate release.gradle to kotlin

94b139a

removed useless accessors imports

3aceff5

frankfliu approved these changes May 27, 2024

View reviewed changes

frankfliu merged commit 97c6a0d into deepjavalibrary:master May 27, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gradle Kotlin script plus other stuff #3167

Gradle Kotlin script plus other stuff #3167

elect86 commented May 8, 2024 •

edited

frankfliu May 8, 2024

elect86 May 13, 2024 •

edited

frankfliu May 13, 2024

frankfliu left a comment

elect86 commented May 13, 2024

elect86 commented May 13, 2024

frankfliu May 13, 2024

elect86 May 14, 2024

frankfliu May 13, 2024

elect86 May 14, 2024

frankfliu May 13, 2024

elect86 May 14, 2024

elect86 commented May 14, 2024

frankfliu commented May 14, 2024

frankfliu commented May 14, 2024

elect86 commented May 14, 2024 •

edited

elect86 commented May 15, 2024

elect86 commented May 15, 2024

elect86 commented May 15, 2024

codecov-commenter commented May 20, 2024

		@@ -0,0 +1,22 @@
		rootProject.name = "djl"

		val reserved = listOf("gradle", "buildSrc")

Gradle Kotlin script plus other stuff #3167

Gradle Kotlin script plus other stuff #3167

Conversation

elect86 commented May 8, 2024 • edited

Description

Choose a reason for hiding this comment

elect86 May 13, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frankfliu left a comment

Choose a reason for hiding this comment

elect86 commented May 13, 2024

elect86 commented May 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elect86 commented May 14, 2024

frankfliu commented May 14, 2024

frankfliu commented May 14, 2024

elect86 commented May 14, 2024 • edited

elect86 commented May 15, 2024

elect86 commented May 15, 2024

elect86 commented May 15, 2024

codecov-commenter commented May 20, 2024

Codecov Report

elect86 commented May 8, 2024 •

edited

elect86 May 13, 2024 •

edited

elect86 commented May 14, 2024 •

edited