Introduce VirtualThreadExecutorService for Virtual Threads Support #2185
Conversation
|
Though this is a good move I have two concerns. Luckily, grizzly allows easily to use of custom executor service as a worker thread pool. This allows us to play with VirtualThreads without making it a strong dependency right away. |
|
Ok, I have to correct here myself, as 21 is of course expected to be LTS. However the second concern is still valid. |
Just a small nit, but Java 21 itself is certainly not an LTS. Core Java SE doesn't have any concept of LTS. The distribution offered by Oracle may be LTS. |
|
@kofemann Can you make it so the build can use Java 11 and exclude these changes as well as Java 21 ea and include these changes. |
|
Hi, @kofemann,
The impact of using ThreadLocal in virtual threads is only if it's used like a cache for threads in a thread pool. Meaning that it caches data created by one task and used by subsequent tasks executed on the same thread later. Otherwise it effectively works like a ScopedValue within the scope a virtual thread, and thus within the scope of a single task executed on the thread. Switching to virtual threads might incur a performance penalty because objects previously cached for all tasks on the same thread would be cached only for a single task and then thrown away. But all should work, with this possibly negligible penalty. If the penalty is not negligible, ScopedValue wouldn't help. It would be necessary to make the cache global and thread-safe. |
|
@mz1999 I was also thinking of adding some limitation on the maximum virtual threads running at a time, similar to maxPoolSize for a platform thread pool. This would allow to limit the load on the server and allow for back pressure, slowing down communication from clients. Greg Wilkins from Webtide writes about this in this article https://webtide.com/if-virtual-threads-are-the-solution-what-is-the-problem/ (in The Cost of Waiting section). I think it would be enough to add a semaphore in the execute method and allow only X parallel executions of the What do you think? |
I completely agree with you. Although virtual threads are very lightweight and can be easily created in large numbers, request processing is not just about creating virtual threads, it is also about consuming system resources (CPU, memory, I/O, etc.) or external resources (e.g. databases) while executing the processing tasks. As the number of concurrently executing tasks increases, there comes a point where one or more resources become bottlenecks, leaving additional virtual threads idle while waiting for access to those resources. Therefore, imposing a limit on the maximum number of concurrent virtual threads is indeed a prudent strategy. To address this, I've introduced a semaphore to control the maximum number of concurrent tasks. When this limit is exceeded, a |
|
Generally, I have nothing against VirtualThread and we are looking in our project to make use of them as well. Nonetheless, I am less optimistic about using them in grizzly, which internally uses ThreadLocals in some places: Most of them are trivial to fix. This probably has to be done before VirtualThreads is made as a standard option. |
@kofemann, I agree with you that thread locals should be reviewed. But this new VirtualThreadExecutorService is not going to be the default option, users will have to enable it. Release notes and javadoc can mention that this feature is experimental and should be used with caution. Or do you suggest that Grizzly shouldn't provide even any optional support for virtual threads until all threadlocals are addressed? |
| poolSemaphore = new Semaphore(cfg.getMaxPoolSize()); | ||
| } else { | ||
| poolSemaphore = new Semaphore(Integer.MAX_VALUE); |
There was a problem hiding this comment.
The Semaphore should be constructed with the true value for the second parameter to enable fairness - guarantee the order of tasks.
|
|
||
| @Override | ||
| public void execute(Runnable command) { | ||
| if (poolSemaphore.tryAcquire()) { |
There was a problem hiding this comment.
I would also introduce a queue for the tasks that can't be immediately executed. It could be implemented by another semaphore wrapping the poolSemaphore, with the queueSize + maxPoolSize permits, like this:
Semaphore poolSemaphore = new Semaphore(maxPoolSize, true);
Semaphore queueSemaphore = new Semaphore(queueSize + maxPoolSize, true);
if (queueSemaphore.tryAcquire() {
internalExecutorService.execute(() -> {
try {
poolSemaphore.acquire();
command.run();
} finally {
poolSemaphore.release();
queueSemaphore.release();
}
});
} else {
throw new RejectedExecutionException("Too Many Concurrent Requests");
}
The queue semaphore should be acquired on the kernel thread but the pool semaphore would be acquired on the virtual thread. Virtual threads might be blocked on acquiring the pool semaphore but kernel thread should continue working.
There was a problem hiding this comment.
Following your latest advice, I've added a second semaphore (queueSemaphore) to further refine the task management in VirtualThreadExecutorService.
Your guidance has been instrumental in evolving the functionality and robustness of this service. I'm looking forward to any additional thoughts or feedback you might have!
|
Although I'm not a committer, this looks good to me now. However, I think it's not OK that this PR moves the Java baseline to Java 21. All future versions of Grizzly would require Java 21 to run. I think it's better to move the VT code to a separate maven module which is built with Java 21 as the baseline, and the core module is built with Java 21 but with Java 11 as the source and target version. Then it would still be possible to use Grizzly with Java 11 with the default executor. On top of this, there are 2 outstanding things to address but they could be addressed later:
Until these 2 are addressed, this executor can be released as an experimental executor for virtual threads. After they are addressed, it can be a generally available VT executor, and even might become the default one for Java 21 some day. More details about ThreadLocalsAcording to the previous comment #2185 (comment), there are a few ThreadLocal variables used as cache: Thread cache:
Some variables potentially used for caching - if not used for caching, may remain as is:
Thread-unsafe formatters cached for performance purposes - these should be replaced by global
|
This PR introduces the
VirtualThreadExecutorService, extending the capabilities of Grizzly's thread pool to support Java's virtual threads. This new executor service leverages the lightweight, user-mode threads introduced in recent Java versions, offering potential performance improvements for certain workloads.grizzly has four IO Strategies, and each IOStrategy will involve two types of thread pools:
VirtualThreadExecutorServiceis more suitable for worker-thread IOStrategy scenarios where Selector threads use Platform threads and the worker ThreadPool uses virtual threads.Here's how you create a worker ThreadPool for grizzly in glassfish:
Similarly, the
GrizzlyExecutorServicecan be replaced withVirtualThreadExecutorServiceso that the worker thread uses a virtual thread.