Server only accepts limited requests

[mrtechit]

The server only accepts limited requests due to blocking of listeners.

The below code in abstractiso8583 connector decides the number:
protected EventLoopGroup createWorkerEventLoopGroup() {
return new NioEventLoopGroup(Runtime.getRuntime().availableProcessors());
}

How can we the server non blocking to accept more connections ?

There is 1 option to increase worker threads is what i see, :
return new NioEventLoopGroup(10);

or maybe use netty pipelinefactory ?

[kpavlov]

Changed to default number of threads. You may extend 'createWorkerEventLoopGroup()' method in subclass. That's a quick fix, not necessary the best one.

[mrtechit]

How much will the default be ?

[kpavlov]

See io.netty.channel.MultithreadEventLoopGroup#DEFAULT_EVENT_LOOP_THREADS

DEFAULT_EVENT_LOOP_THREADS = Math.max(1, SystemPropertyUtil.getInt(
                "io.netty.eventLoopThreads", NettyRuntime.availableProcessors() * 2));

Maybe, you should tune boss EventLoopGroup ?com.github.kpavlov.jreactive8583.AbstractIso8583Connector#getBossEventLoopGroup

[mrtechit]

As per netty boss eventlookgroup should always be 1.
The workers have to be tuned in non blocking mode ?
With this current change, the server will accept maximum 4 or 6 depending on the CPU.

[kpavlov]

To limit the number of concurrent connections to your server - on a UNIX system - you can set the ulimit on file descriptors before running the application, and, once all descriptors are open, other clients will be unable to connect.

To get the number of allowed file descriptors, exec $ ulimit -n as the user running the process. See man ulimit for more info.

https://stackoverflow.com/a/19129335/3315474

[mrtechit]

Hi,

I think I did not make the issue very clear (Apologies for that),
The issue does not exist in terms of number of concurrent connections coming at OS LEVEL .
It lies on the handling of concurrent threads at API LEVEL.

The OS would always have a limitation, however usually what the API does is that, on the same port it keeps accepting requests with a unique identifier and works in non blocking mode.

Expected scenario :
Client thread 1, Client thread 2, Client thread 3 -> j8583 sever thread 1 (All requests accepted via non blocking code)

Current scenario :
Client 1 -> j8583 server thread 1
Client 2 -> j8583 server thread 2
Client 3 -> j8583 server thread 3
Client 4 -> BLOCKED until previous one clears up.

There is an API called jPOS which has implemented the expected scenario, we can have detailed a discussion if you are free?

[kpavlov]

@mrtechit, Can you share your test please?

[kpavlov]

@mrtechit, I added test but can't reproduce it. Can you help me to reproduce?

[mrtechit]

@kpavlov

Hi,
Setup 1 debug point anywhere in :
public boolean onMessage(ChannelHandlerContext ctx, IsoMessage isoMessage) {

Then,
Just hit with multiple client 1 by 1, you would see that the server accepts limited connections based on below method in AbstractIso8583Connector.java:

protected EventLoopGroup createWorkerEventLoopGroup() {
return new NioEventLoopGroup(10);
}

Let me know if you need more information.

[mrtechit]

Hi,

Any luck with this ?

[mahmood8664]

Hi @kpavlov,
Has this problem been resolved? I have the same issue, doing a blocking job in IsoMessageListener.onMessage(), so it would block until the task is finished. But I want to accept more messages meanwhile the previous thread is blocked.
What is your solution?

[kpavlov]

If you do a blocking job - it deplets the thread pool and will eventually explode. Make sure the blocking job is executed on worker group event loop and increase # of workers as a workaround

[mahmood8664]

I increased the worker group count, but nothing changed, it seems every listener has one dedicated thread:

2022-08-06 12:34:34,849 INFO  ID:[nioEventLoopGroup-6-1, , ] eu.nets.banknet.Configuration$2.140: client: received: 000000007392

2022-08-06 12:34:34,849 INFO  ID:[nioEventLoopGroup-6-1, , ] eu.nets.banknet.Configuration$2.140: client: received: 000000007393

2022-08-06 12:34:34,849 INFO  ID:[nioEventLoopGroup-6-1, , ] eu.nets.banknet.Configuration$2.140: client: received: 000000007394

2022-08-06 12:34:34,850 INFO  ID:[nioEventLoopGroup-6-1, , ] eu.nets.banknet.Configuration$2.140: client: received: 000000007395

2022-08-06 12:34:34,850 INFO  ID:[nioEventLoopGroup-6-1, , ] eu.nets.banknet.Configuration$2.140: client: received: 000000007396

2022-08-06 12:34:34,850 INFO  ID:[nioEventLoopGroup-6-1, , ] eu.nets.banknet.Configuration$2.140: client: received: 000000007397

If you have an example I would appreciated if you share it with me, by the way I can turn my blocking job to non-blocking by using CompletableFuture.
Thanks for your answer.

[Vostan]

@kpavlov Hi,

Thanks for this library we have used it in our Production for a couple of years now,
Is there some practical way we can contribute to fixing this issue?