Support for Virtual Threads - JDK21

[mjagus]

Basic information

• Axon Framework version: 4.7.4
• JDK version: OpenJDK 20 with --enable-preview switch
• Complete executable reproducer if available (e.g. GitHub Repo): Haven't prepared anything, but I think it's still worth reporting

I've been experimenting recently with project Loom by replacing various thread pools with virtual thread executors and running my application with -Djdk.tracePinnedThreads=full JVM option. The stack trace below popped up after I replaced my Quartz scheduler thread pool and QuartzDeadlineManager triggered my saga's @DeadlineHandler method:

org.axonframework.modelling.saga.repository.jpa.JpaSagaStore.loadSaga(JpaSagaStore.java:157)
org.axonframework.modelling.saga.repository.AnnotatedSagaRepository.doLoadSaga(AnnotatedSagaRepository.java:231)
org.axonframework.modelling.saga.repository.AnnotatedSagaRepository.lambda$doLoad$1(AnnotatedSagaRepository.java:111)
java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) <== monitors:1
org.axonframework.modelling.saga.repository.AnnotatedSagaRepository.doLoad(AnnotatedSagaRepository.java:110)
org.axonframework.modelling.saga.repository.AnnotatedSagaRepository.doLoad(AnnotatedSagaRepository.java:56)
org.axonframework.modelling.saga.repository.LockingSagaRepository.load(LockingSagaRepository.java:64)
org.axonframework.modelling.saga.AbstractSagaManager.send(AbstractSagaManager.java:235)
org.axonframework.deadline.quartz.DeadlineJob.lambda$executeScheduledDeadline$4(DeadlineJob.java:183) 

The thread pinning happens because of synchronized block in ConcurrentHashMap.computeIfAbsent method. That block wraps a call to mapping function which happens to do I/O when underlying SagaStore implementation is anything different than InMemorySagaStore (in my case it is JpaSagaStore).

At first this may look like something that should be reported to OpenJDK guys, but after quick skimming of JDK's mailing list it turns out that they are aware of this, but the fix is nowhere near close (according to Doug Lee himself), so it might be worthwhile to fix it in the framework itself before Loom goes GA (which happens in a few months apparently). The obvious solution would be to replace ConcurrentHashMap with HashMap + ReadWriteLock combo.

Expected behaviour

Virtual thread is not pinned to carrier thread when a saga is first loaded into managedSagas map in AnnotatedSagaRepository.

Actual behaviour

Virtual thread is pinned, which might lead to a potential deadlock in the JVM when all carrier threads become blocked.

[smcvb]

This is extremely helpful of you to share with us, @mjagus!

Was the only spot you faced issues with this on using Virtual Threads?
We tried Virtual Threads some time ago for our Streaming Event Processors, which seemed to work fine.
But if there are more spots you have encountered, that would be helpful.

Do note that I do not think we are dealing with a bug here, if I am honest with you.
It is more of an enhancement request to support a new feature from a JDK version thatAxon Framework is not built on.
Granted, providing that support is fair at some stage.
I will, however, remove the bug field and rename the ticket to Virtual Thread support.

[mjagus]

Hi Steven,
This is the only example of thread pinning I encountered during runtime. Every other part of the framework that I used seemed to work flawlessly with virtual threads enabled.

I also briefly looked at framework's source code to see if there are any obvious code patterns that may lead to thread pinning and I noticed EventProcessorTask.run method that has a synchronized block inside of which it may indirectly call event handlers:

    @Override
    public void run() {
        synchronized (runnerMonitor) {
            boolean mayContinue = true;
            int itemsAtStart = taskQueue.size();
            int processedItems = 0;
            while (mayContinue) {
                processNextTask();
                processedItems++;
                // Continue processing if there is no rescheduling involved and there are events in the queue, or if yielding failed
                mayContinue = (processedItems < itemsAtStart && !taskQueue.isEmpty()) || !yield();
            }
        }
    }

Since event handlers are allowed to do I/O, this code will most likely cause the virtual thread executing this code to be pinned. EventProcessorTask seems to be only used by a subscribing event processor when AsynchronousEventProcessingStrategy is configured. Since I'm using tracking event processors in production, this has not popped up during my experiments with Loom.

[smcvb]

Thanks for taking another look, @mjagus; your effort is much appreciated.
As stated before, I'll leave the issue open for others to find and for us to remember taking a look at this in the future.

[smcvb]

Minor update on this issue, mainly concerning the milestone.
I attached this issue to 4.9.0 with a two-fold desire:

1. Further validate AF works with JDK21 and keep validating - covered through PR Add JDK21 to GitHub Actions #2866
2. Investigate and adjust use of synchronized blocks to more nicely align with Virtual Thread support.

"Desire" one will be covered in release 4.9.0, but desire two cannot.
We simply do not have the resources to work on this at the moment, hence we'll move this effort towards a later point in time.
@mjagus, if you so desire, you're free to provide a pull request for your specific suggested points.

With the above, I think it's clear why I'll remove this issue from milestone 4.9.0.