New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(crd-generator): Add CRD-Generator Maven plugin #5979
base: main
Are you sure you want to change the base?
Conversation
fa9d221
to
dc50235
Compare
@manusa Can you have second look on this? Is this now going into the right direction? The logic to collect and load custom resource classes is now included in an own module "crd-generator-collector" intended to be shared between maven-plugin, gradle-plugin, cli etc. TBD:
A prototype for the CLI which is using the same base can be found here: baloo42#9 |
@shawkins, @andreaTP Maybe you can have a look on this too. Marc has mentioned that he is not sure about the final / future purpose of the new module (me too), so I'll try to explain a few thoughts I had. To use the CRD-Generator from CLI, Maven, Gradle etc. we need in general two main features which are not implemented in the api-v2 module:
The new crd-generator-collector module implements both features via the main component CustomResourceCollector which can be used in the following way: CustomResourceCollector customResourceCollector = new CustomResourceCollector()
.withClasspathElements(<additional classpath elements>)
.withFilesToIndex(<e.g. directory with class files or jar file with class files>)
CustomResourceInfo[] customResourceInfos = customResourceCollector.findCustomResources();
CRDGenerator crdGenerator = new CRDGenerator()
.customResources(customResourceInfos)
.inOutputDir(outputDirectory);
crdGenerator.detailedGenerate() In conclusion this approach is some kind of "pre-processor" for the CRDGenerator. Another approach would be to implement it as a facade ("processor"):
This approach is nicer from a user perspective but we need to passthrough every option (or use some kind of nested builder pattern). A third approach would be to implement both features in the api-v2 module itself and make jandex an optional dependency (or allow to extend it by using a service loader).
This approach might be the nicest but it would also be the most advanced: the api-v2 module has now to deal somehow with additional dependencies like Jandex and the code base of the api-v2 would grow. Which approach do you prefer? Other thoughts? |
I'd like to keep the core api as simple as possible - whatever is needed to satisfy Quarkus should be good. The first option seems good enough. I don't really see much need for coupling between the collector and the crd generator - it doesn't really need to provide CustomResourceInfo, just classes. I mentioned on a pr somewhere that the collector to me just looks like something that you could provide a list of interfaces / base classes, and a list of annotations, and it will just find everything that matches that. |
Yes, you are right. Providing only classes could basically do the job but then we cannot implement high level filtering (e.g. by group or version). Do we need high level filtering? If yes: High level filtering could also be implemented in api-v2 so that we can stay by the strategy to return only classes instead of CustomResourceInfo's.
Yes, at the moment the base interface (HasMetadata) and the additional required annotations (Version, Group) to identify a custom resource class are hardcoded. But this can be easily changed to be configurable (with sane defaults) if you think this is useful. Do we need to search for other Interfaces/Annotations? |
Completely agree with what @shawkins said, let's keep the core API as simple as possible. |
Right I'd vote for putting that into the api if needed. It would probably be good to consider whether having a public constructor on CustomResourceInfo makes sense - that opens up a different usage model for the CRDGenerator that isn't bound to the typical annotations.
Not yet. I was just highlighting that it doesn't really need to be seen as that specific to CRs. |
Thanks for the feedback. To sum it up, I have the following todo's so far:
|
Don't worry about this step just yet, what you currently have should work fine for the first iteration. |
… of CustomResourceInfo's
3705e74
to
2842f0d
Compare
return customResourceClasses; | ||
} | ||
|
||
public Class<? extends HasMetadata>[] findCustomResourceClasses() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This method is required because the CRDGenerator accepts only an array.
Can we extend the CRDGenerator with an additional method?
public CRDGenerator customResourceClasses(Collection<Class<? extends HasMetadata>> crClasses) {
return customResources(crClasses.stream().map(CustomResourceInfo::fromClass).toArray(CustomResourceInfo[]::new));
}
* | ||
* @see JandexCustomResourceClassScanner#findCustomResourceClasses(IndexView) | ||
*/ | ||
private Index createBaseIndex() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can be optimized if required.
Existing indices of the fabric8 modules can be used or a specialized index can be created at build time.
Do we need further optimization within this PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree that we should reuse the existing index is possible - since users wil need to modify their pom / gradle file to use these plugins, it doesn't seem out of the question to just document also requiring the jandex plugin be configured as well for projects that want to create crds.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm currently writing some usage docs about the different use cases, including when additional configuration is required. This should help to get an overview from the user perspective.
Some notes on a more technically perspective:
- Jandex is only required to "scan" for Custom Resource classes. If the list of classes is given, no scanning at all is required. (The classloader does the heavy lifting)
- The implementation detects existing indices in the scan targets automatically and prefers those. Only if a scan target does not contain an index, an own in-memory index will be created. This means having an existing index is just optimization, never a requirement.
- Custom Resource classes must be loaded before they can be used by the CRDGenerator itself. Any dependency which is required to load those classes must be in the classpath.
- A scan target must be also in the classpath. But not every classpath element is a scan target.
- Additional indexed classes beside the "base-index" are only required, if the user wants to scan a target with Custom Resource classes which don't implement the CustomResource or HasMetadata interface directly. In this case, the user must ensure that the used intermediate class is included somehow in the underlying composite index. For example, he can add the dependency which includes the class to the list of scan targets, so that an index is automatically created if needed.
In conclusion, a user doesn't need to know anything about Jandex. Even for the edge case from the last bullet point, the user only has to care about including all "relevant" classes to the list of scan targets and not Jandex specifically.
(But if his project already uses Jandex, the scans will be a little bit faster.)
Keeping this in mind, optimizing the creation of the base index is mostly about performance.
Only if there are other "intermediate" classes like "CustomResource", then we should add them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wrote an integration test for the "intermediate class" case (last bullet point from above):
ddc3bec#diff-0488fc4e2f91d3e88ce94d75d1317631a14f2854194a2cf68354be4712e9c2cb
In this case, the user must add the dependency, which contains the intermediate class, to the list of dependencies to scan. Under the hood the classes are now added to the composite index and Custom Resources Classes will be found by the collector.
|
||
private int maxJarEntries = 100000; | ||
private long maxBytesReadFromJar = 100000000; // 100 MB | ||
private long maxClassFileSize = 1000000; // 1 MB |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need limits?
import static org.junit.jupiter.api.Assertions.assertTrue; | ||
import static org.mockito.Mockito.when; | ||
|
||
@EnabledForJreRange(min = JRE.JAVA_17) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we want to run these tests in Java 17 and above only? Is it possible to remove it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I get some mockito related errors and havn't figured out yet why this is only the case on a lower versions than 17. (see https://github.com/fabric8io/kubernetes-client/actions/runs/9052896736/job/24871094873#step:4:36770)
Any ideas?
But in general I'm not really happy with both tests in the maven-plugin module. A lot stuff must be mocked, only to test some simple things...
I'm open to rewrite both tests enterily if someone can point me into the right direction. e.g. writing e2e tests with tool x. (The java-generator maven plugin doesn't include any own tests, so I cannot use it as template)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you think you can use maven invoker plugin here? We use it for JKube integration tests to verify generated YAML files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added the first integration tests, is this the right direction?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In my opinion, it looks okay, but we might want to get opinions from other team members.
Maybe we can make the verification check more strict (it might help us find the differences in generated YAML). In Eclipse JKube we compare whole YAML manifests in expected/
directory with generated output.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added more integration tests and now the content is also verified.
Should I keep the unit tests or remove them?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's up to you. If you think they're redundant feel free to remove them.
@@ -0,0 +1,180 @@ | |||
# CRD-Generator - Maven Plugin | |||
|
|||
Maven plugin for the CRD-Generator: Generate CRDs from Java model during Maven builds. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This document should be either linked in https://github.com/fabric8io/kubernetes-client/blob/main/doc/CRD-generator.md or we should move it in doc/
folder like the rest of the documentation files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, we havn't talked about where the documentation for the generator-v2 should be placed, how it should be structured and especially how it should be distinguished to generator-v1.
My intention was to write first for each generator-v2 extension (maven-plugin, gradle-plugin, cli) an own README until at least two of them are merged.
Afterwards, it should be easier to write some common notes about generator-v2 including migration paths and deprecations / roadmap into the central docs. And of course the content from the README's can be moved again, too.
If I should do this within this PR, I need some guidence on how to introduce generator-v2. Otherwise we should probably create a dedicated issue for the common part of the generator-v2 docs.
cc4ec43
to
b79bae8
Compare
Quality Gate failedFailed conditions |
Description
Add CRD-Generator Maven plugin (using api-v2).
Fixes #5944
Examples: https://github.com/baloo42/crd-generator-maven-examples
Type of change
test, version modification, documentation, etc.)
Checklist