Bug Report: Client EJB context lookup fails: SerialContextProvider.lookup(): prvdr is null

[kitlei-robert]

Brief Summary

Client side attempt for EJB lookup fails with Cannot invoke "com.sun.enterprise.naming.impl.SerialContextProvider.lookup(String)" because "prvdr" is null.

Expected Outcome

Here is a very simple client code that tries to access an EJB.

import java.util.Properties;

import javax.naming.InitialContext;

public class Main {
    public static void main(String[] args) throws Exception {
        var props = new Properties();
        var ctx = new InitialContext(props);
        ctx.lookup("java:global/fake/fake/fake!Fake");
    }
}

Baseline: this works using Payara Server 6.2022.2 (Full).

My setup: Eclipse IDE for Enterprise Java and Web Developers, Latest (2023-09) edition installed by way of the Eclipse Oomph installer. No special options. In it, I configure a Payara Server using Payara Tools installed from the Eclipse Marketplace. No special options. The server starts up as expected and in a sense, it's working well; for example, I can publish servlets onto it and they run just fine.

My even simpler setup: I just run asadmin start-domain domain1. The following is the same either way.

When I run my client program (see details below) for the first time, the server logs show several InvocationTargetExceptions. To avoid running into them, I ignore module usage with the following fakery:

asadmin create-jvm-options --add-opens=java.base/java.lang=ALL-UNNAMED
asadmin create-jvm-options --add-opens=java.base/java.io=ALL-UNNAMED
asadmin create-jvm-options --add-opens=java.management/javax.management=ALL-UNNAMED
asadmin create-jvm-options --add-opens=java.management/javax.management.openmbean=ALL-UNNAMED

Then I restart the server because the log now claims, Addition of: '--add-opens=java.management/javax.management=ALL-UNNAMED' cannot take effect without server restart.

Once the server is up again, I run the program like this, using either Java 20 or the recent Java 21 (specifically, the Azul Zulu distribution) on Windows 11. I show the command line version here; the equivalent execution in Eclipse behaves similarly.

java -cp .;payara6.2022.2\glassfish\lib\gf-client.jar --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.management/javax.management=ALL-UNNAMED --add-opens=java.management/javax.management.openmbean=ALL-UNNAMED -Dorg.glassfish.gmbal.no.multipleUpperBoundsException=true Main

If the gf-client.jar I use is the one in Payara Server 6.2022.2 (Full), it throws an exception like it should with this message:

Exception in thread "main" javax.naming.NamingException: Lookup failed for 'java:global/fake/fake/fake!Fake' in SerialContext[myEnv={java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl, java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.url.pkgs=com.sun.enterprise.naming} [Root exception is javax.naming.NameNotFoundException: fake]
    at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:496)
    at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:442)
    at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
    at Main.main(Main.java:15)
Caused by: javax.naming.NameNotFoundException: fake
    at com.sun.enterprise.naming.impl.TransientContext.resolveContext(TransientContext.java:299)
    at com.sun.enterprise.naming.impl.TransientContext.lookup(TransientContext.java:207)
    at com.sun.enterprise.naming.impl.TransientContext.lookup(TransientContext.java:208)
    at com.sun.enterprise.naming.impl.SerialContextProviderImpl.lookup(SerialContextProviderImpl.java:70)
    ...

That is, it manages to contact the server, it apparently is able to get a provider from it, then concludes that fake is indeed fake. So far so good...

Current Outcome

However, using Payara Server 6.2023.8 (Full) or the recent 6.2023.9 (Full) in a similar fashion (with the asadmin steps, and the client having gf-client.jar of the corresponding server's payara6/lib folder on its classpath now), the error message is quite different:

Exception in thread "main" javax.naming.CommunicationException: Communication exception for SerialContext[myEnv={java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl, java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.url.pkgs=com.sun.enterprise.naming} [Root exception is java.lang.NullPointerException: Cannot invoke "com.sun.enterprise.naming.impl.SerialContextProvider.lookup(String)" because "prvdr" is null]
    at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:518)
    at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:442)
    at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
    at Main.main(Main.java:15)
Caused by: java.lang.NullPointerException: Cannot invoke "com.sun.enterprise.naming.impl.SerialContextProvider.lookup(String)" because "prvdr" is null
    at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:483)
    ... 3 more

The same happens even if I include all the jar files under both payara6/lib/ and payara6/modules/ on the classpath.

This issue is clearly related to #5510 and #6214, as both ultimately failed with the prvdr is null bit. I couldn't find a fix either in them or in #6338 referenced by the latter.

Reproducer

See the code and the commands above.

Further details and investigations.

1. If the server is not up, the program (using gf-client.jar from Payara 6.2023.9) appropriately indicates that it cannot connect to the server, there's no message about prvdr is null.
2. I tried manually specifying environment details as follows, even though that shouldn't be necessary. Needless to say, it didn't bring about any change.

        props.setProperty("java.naming.factory.initial", "com.sun.enterprise.naming.SerialInitContextFactory");
        props.setProperty("java.naming.factory.url.pkgs", "com.sun.enterprise.naming");
        props.setProperty("java.naming.factory.state", "com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl");
        props.setProperty("org.omg.CORBA.ORBInitialHost", "127.0.0.1");
        props.setProperty("org.omg.CORBA.ORBInitialPort", "3700");

3. Even if I actually deploy an EAR (which the server allows me to do) and I try to access that rather than fake, the outcome is the same (prvdr is null).
4. If I run the client without the --add-opens fakery, it throws an InvocationTargetException. This is not the main question, but this is quite inelegant, and I'd be happy to run the program without them; can you suggest how? Also, I suspect that prvdr is null might be caused by some more illegal access down the line; probably the corresponding error message gets lost.
5. The server log shows the following INFO messages, in case they may be related to the problem, I include them here. The first two show up during Payara start, the second two when I deploy an EAR with an EJB.

• Cannot find the resource bundle for the name com.sun.logging.enterprise.system for class org.glassfish.enterprise.iiop.api.ORBLazyServiceInitializer using fish.payara.server.core.orb.connector
• Cannot find the resource bundle for the name com.sun.logging.enterprise.system.container.ejb for class com.sun.ejb.containers.EjbContainerUtilImpl using fish.payara.server.internal.ejb.ejb-container
• Cannot find the resource bundle for the name com.sun.logging.enterprise.system.container.ejb for class org.glassfish.ejb.security.application.EjbSecurityComponentInvocationHandler using fish.payara.server.internal.ejb.ejb-container
• Cannot find the resource bundle for the name com.sun.logging.enterprise.system.container.ejb for class com.sun.ejb.EJBUtils using fish.payara.server.internal.ejb.ejb-container

Conclusion:

Obviously, something in the context lookup mechanism got broken between Payara versions 6.2022.2 and 6.2023.8.
How do I force my command line client to properly connect to the EJB deployed on the Payara 6.2023.9 server?

Operating System

Windows 11

JDK Version

Zulu 20 and/or 21

Payara Distribution

Payara Server Full Profile

[artur-mal]

Hi kitlei-robert.
I am sorry for not coming back to you earlier.

Please note that JAVA 21 is not supported on the Payara Community Server yet. Java 20 will never be supported, because this release is not LTS.
I couldn't reproduce the prvdr is null error on JAVA 17, however, I failed to execute JNDI Lookup in my remote client.
Remote EJB JNDI Lookup doesn't work as it should.

I have escalated this to the platform development team with internal JIRA FISH-7946
I will let you know once we are able to proceed with the development of the fix.

Regards,
Artur

[kitlei-robert]

Addendum: the most recent version of Payara (6.2023.10) also exhibits the prvdr is null behaviour on my system. The most recent version of GlassFish (7.0.9) does not, it gets to NameNotFoundException: fake and if an EJB is actually deployed, it can look it up as expected. I used Java 21, so the JDK's version number probably does not matter all that much for this purpose.

[kitlei-robert]

The issue remains with the newest releases, Payara 6.2023.11 still fails with prvdr is null, GlassFish 7.0.10 is doing OK.

It seems to me that there's some difference inside appserv-rt.jar (referenced by gf-client.jar) in Payara vs GlassFish that causes the different behaviour: using the latter's appserv-rt.jar, the client can connect remotely to an EJB deployed in the former server but not vice versa.