Bug Report: Fails to deploy CAR module

[mkarg]

Brief Summary

Our EAR contains a CAR module and deployed without problem on Payara 5.
On Payara CE Full 6.2024.3 it fails to deploy but instead prints a stack trace.

Expected Outcome

Deploys successfully.

Current Outcome

The log contains the following error description:

[2024-03-20T10:07:45.471+0100] [Payara 6.2024.3] [SCHWERWIEGEND] [] [javax.enterprise.system.core] [tid: _ThreadID=271 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1710925665471] [levelValue: 1000] [[
  A MultiException has 2 exceptions.  They are:%%%EOL%%%1. com.sun.enterprise.module.ResolveError: Failed to start OSGiModuleImpl:: Bundle = [fish.payara.server.internal.appclient.server.appclient-server-core [17]], State = [NEW]%%%EOL%%%2. java.lang.IllegalStateException: Could not load descriptor SystemDescriptor(%%%EOL%%%implementation=org.glassfish.appclient.server.core.CarHandler%%%EOL%%%  name=car%%%EOL%%%       contracts={org.glassfish.appclient.server.core.CarHandler,org.glassfish.api.deployment.archive.ArchiveHandler}%%%EOL%%%    scope=jakarta.inject.Singleton%%%EOL%%% qualifiers={}%%%EOL%%%  descriptorType=CLASS%%%EOL%%%   descriptorVisibility=NORMAL%%%EOL%%%       metadata=Bundle-SymbolicName={fish.payara.server.internal.appclient.server.appclient-server-core},Bundle-Version={6.2024.3}%%%EOL%%%       rank=0%%%EOL%%% loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [fish.payara.server.internal.appclient.server.appclient-server-core [17]], State = [NEW],1900774040)%%%EOL%%%        proxiable=null%%%EOL%%% proxyForSameScope=null%%%EOL%%% analysisName=null%%%EOL%%%      id=33%%%EOL%%%  locatorId=0%%%EOL%%%       identityHashCode=1803569378%%%EOL%%%    reified=false)%%%EOL%%% : A MultiException has 2 exceptions.  They are:
1. com.sun.enterprise.module.ResolveError: Failed to start OSGiModuleImpl:: Bundle = [fish.payara.server.internal.appclient.server.appclient-server-core [17]], State = [NEW]
2. java.lang.IllegalStateException: Could not load descriptor SystemDescriptor(
        implementation=org.glassfish.appclient.server.core.CarHandler
        name=car
        contracts={org.glassfish.appclient.server.core.CarHandler,org.glassfish.api.deployment.archive.ArchiveHandler}
        scope=jakarta.inject.Singleton
        qualifiers={}
        descriptorType=CLASS
        descriptorVisibility=NORMAL
        metadata=Bundle-SymbolicName={fish.payara.server.internal.appclient.server.appclient-server-core},Bundle-Version={6.2024.3}
        rank=0
        loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [fish.payara.server.internal.appclient.server.appclient-server-core [17]], State = [NEW],1900774040)
        proxiable=null
        proxyForSameScope=null
        analysisName=null
        id=33
        locatorId=0
        identityHashCode=1803569378
        reified=false)

com.sun.enterprise.module.ResolveError: Failed to start OSGiModuleImpl:: Bundle = [fish.payara.server.internal.appclient.server.appclient-server-core [17]], State = [NEW]
        at org.jvnet.hk2.osgiadapter.OSGiModuleImpl.start(OSGiModuleImpl.java:193)
        at org.jvnet.hk2.osgiadapter.OsgiPopulatorPostProcessor$1.loadClass(OsgiPopulatorPostProcessor.java:54)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.loadClass(ServiceLocatorImpl.java:2254)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.reifyDescriptor(ServiceLocatorImpl.java:433)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.narrow(ServiceLocatorImpl.java:2316)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.igdCacheCompute(ServiceLocatorImpl.java:1192)
        at org.jvnet.hk2.internal.ServiceLocatorImpl$8.compute(ServiceLocatorImpl.java:1186)
        at org.jvnet.hk2.internal.ServiceLocatorImpl$8.compute(ServiceLocatorImpl.java:1183)
        at org.glassfish.hk2.utilities.cache.internal.WeakCARCacheImpl.compute(WeakCARCacheImpl.java:105)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetDescriptor(ServiceLocatorImpl.java:1266)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:765)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:732)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:719)
        at org.glassfish.javaee.full.deployment.EarHandler.getArchiveHandlerFromModuleType(EarHandler.java:594)
        at org.glassfish.javaee.full.deployment.EarHandler.expand(EarHandler.java:202)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.getContext(ApplicationLifecycle.java:2239)
        at com.sun.enterprise.v3.server.ApplicationLifecycle$DeploymentContextBuidlerImpl.build(ApplicationLifecycle.java:2101)
        at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:508)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAs(Unknown Source)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAs(Unknown Source)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755)
        at org.glassfish.admin.rest.resources.admin.CommandResource.executeCommand(CommandResource.java:409)
        at org.glassfish.admin.rest.resources.admin.CommandResource.execCommandMultInMultOut(CommandResource.java:258)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:240)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:697)
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:367)
        at org.glassfish.admin.rest.adapter.JerseyContainerCommandService$3.service(JerseyContainerCommandService.java:179)
        at org.glassfish.admin.rest.adapter.RestAdapter.service(RestAdapter.java:189)
        at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
        at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:174)
        at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:153)
        at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:196)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:83)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:101)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.osgi.framework.BundleException: Unable to resolve fish.payara.server.internal.appclient.server.appclient-server-core [17](R 17.0): missing requirement [fish.payara.server.internal.appclient.server.appclient-server-core [17](R 17.0)] osgi.wiring.package; (osgi.wiring.package=jdk.security.jarsigner) Unresolved requirements: [[fish.payara.server.internal.appclient.server.appclient-server-core [17](R 17.0)] osgi.wiring.package; (osgi.wiring.package=jdk.security.jarsigner)]
        at org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:4398)
        at org.apache.felix.framework.Felix.startBundle(Felix.java:2308)
        at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:1006)
        at org.jvnet.hk2.osgiadapter.OSGiModuleImpl.startBundle(OSGiModuleImpl.java:224)
        at org.jvnet.hk2.osgiadapter.OSGiModuleImpl.start(OSGiModuleImpl.java:185)
        ... 74 more
]]

Reproducer

Unfortunately our software is closed source, so we must not publish the EAR here. If it is helpful, we can set up an empty CAR module, or we can provide the closed source EAR to Team Payara by directly.

Operating System

Windows 10 Pro

JDK Version

Zulu JDK 17

Payara Distribution

Payara Server Full Profile

[fturizo]

Greetings @mkarg, I'm afraid that to fix this issue we're going to need a self-contained reproducer that can help us troubleshoot the problem, so we'd appreciate it if you could prepare a sample source repository that creates an EAR assembly along with the CAR module so that we can test it out.

Unless we have this reproducer, I'm afraid that we cannot provide any assistance.

[github-actions]

Greetings,
It's been more than 5 days since we requested more information or an update from you on the details of this issue. Could you provide an update soon, please?
We're afraid that if we do not receive an update, we'll have to close this issue due to inactivity.

[mkarg]

I think I can invest more time into Payara 6 testing by end of this month, so it should be possible to provide a reproducer then.