Guidelines on custom IORuntime

[ppurang]

IORuntime comes with this recommendation:

Alternatively, you can create an explicit IORuntime value and put it in implicit scope.
This may be useful if you have a pre-existing fixed thread pool and/or scheduler which you
wish to use to execute IO programs. Please be sure to review thread pool best practices to
avoid unintentionally degrading your application performance.

Is this recommendation/guideline enough?

1. Should projects depend on IORuntime.global or create their own? Recommendation is to rely on global unless "pre-existing fixed thread pool and/or scheduler". When should applications, web services, etc. need an own IORuntime, apart from when they already had their own custom thread pools?

2. Would a single project ever need many IORuntimes? IORuntime is an end-of-the-world runner so why would one need more than one.

While creating a custom IORuntime, some facilities aren't accessible:

1. IORuntime private constructor takes more than the public apply methods allow. Is this by design?

2. The WorkStealingThreadPool isn't accessible. For me this is an oversight. Why shouldn't a third party be able to use this goodness to configure their own IORuntime? The only way to access it is through the createDefaultComputeThreadPool. This hinders setting a custom threadCount.

To see what might be a good canonical way of creating a custom IORuntime to ensure custom thread naming -

def unsafeCreateIORuntime(
  computePoolThreadNamePrefix: String, 
  blockingECThreadNamePrefix: String, 
  schedulerThreadNamePrefix: String): IORuntime = {

lazy val global: IORuntime = {
  val (compute, compDown) = createDefaultComputeThreadPool(global, computePoolThreadNamePrefix)
  val (blocking, blockDown) = createDefaultBlockingExecutionContext(blockingECThreadNamePrefix)
  val (scheduler, schedDown) = createDefaultScheduler(schedulerThreadNamePrefix)
  IORuntime(
    compute,
    blocking,
    scheduler,
    () => {
      compDown()
      blockDown()
      schedDown()
    }
  )
}
global
}

This is a copy-paste of IORuntime.global except for the inaccessible private things.

[djspiewak]

Great question!

First off, the fact that IORuntime doesn't allow the same constructor options externally as it has internally is (mostly) an oversight and should be fixed. The only exception to this is the WorkStealingPool note, which I'm willing to hear counter-arguments on. Backing up…

The circumstances under which you want a custom IORuntime are very niche. In fact, they're so niche that I almost want to just remove that guideline from the @implicitNotFound text. It's easier to enumerate scenarios in which you don't want a custom runtime…

• Anything that uses IOApp. The global pool is pretty much forcibly used within IOApp. We could probably adjust this, but I'm not sure I see a reason to do so. And if you have the global pool, then you're much better off making the rest of your application use this pool rather than trying to make IO use something configured by the rest of the application.
• Anything that behaves optimally with the global pools. This is going to be most applications.
• Any application where you don't have some other compute thread pool being forced on you by another runtime system of some sort. Very few runtime systems force thread pools. Netty and Java's NIO2 are obvious counter-examples, but their pools are kind of magical in other ways anyway, so you generally wouldn't want to try to shove your entire application into them. Meaning you would want a separate thread pool anyway, and global gives that to you.

The two main reasons that it's possible to build custom IORuntimes at all are testing and the rare domineering framework. If you have a pre-existing application wherein you're attempting to introduce IO, and that application has a notion of a compute thread pool which is difficult to replace with the global from IORuntime, then you should probably make a custom IORuntime which wraps that pool rather than using the default global. The long-term goal though in this case should still be to replace the pre-existing compute pool with IORuntime rather than vice versa.

The reason for this is simply that IORuntime's pool is as good or better than any pre-existing pool that you might be working with, even for non-IO code. The only exception to this would be if you're pervasively blocking on your compute pool, which is a situation that the default ExecutionContext.global would handle more gracefully than IORuntime. But… please don't do that.

There's another scenario hinted at by the @implicitNotFound text which I honestly don't think is valid once we make global more configurable (#1238). This other scenario is where your application has highly specialized needs from its pools and you've measured and tuned your thread counts to be something different from the default. Honestly, the right approach here would be to be able to tune the global pool more effectively, rather than trying to create a custom IORuntime, but right now that's a bit limited. In this scenario, you should be able to just reconfigure global to meet your needs.

It's worth noting that the global pools are highly elastic in ways which are hard to achieve (or impossible) in third-party pools. As an example, we've observed that the WorkStealingPool is effectively impossible to over-provision on thread count, since it will simply not use excess threads when they don't increase throughput (due to the way the stealing works). You would still end up with excess GC roots, but that's about it. In other words, global is already self-tuning in a way that you won't get out of other pools, even something like ForkJoinExecutor.

So to sum it all up: I don't think there are very many circumstances at all in which you would want a custom IORuntime outside of testing. The most likely scenario would be introducing IO into a large application where IOApp can't be used and where there is already a strong and tuned notion of a compute pool. This is very rare.

[vasilmkd]

To add to this, the WorkStealingPool will be as safe to block on as ExecutionContext.global is, soon ™️.

[andrethobu]

Is this already the case as of version 3.0.2?

[vasilmkd]

Yes, this is available even before 3.0.0 final.

[ppurang]

Thanks @djspiewak for the detailed answer. Glad that we can rely on global to have our backs.

Also checked out the accompanying issue #1238 and #1568. @vasilmkd Looking forward to the changes.

[bblfish]

Hi, I am looking for guidance on how to use IORuntime with Akka.
I guess the answer will depend very much on how deeply one uses cats-effect in Akka.

Right now I am just needing it very minimally: I am using IO in the routes layer, to fetch data on a public key.
That really just requires one (akka.http client) call to a remote resource which in Akka is done using a function
returning a Future. By using IO it looks like I need to have an IORuntime, with a thread pool then.

As one uses cats-effect more, things will likely end up looking very different.
I checked Evolution Gaming's Akka Effect library but could not find (using grep) any settings for IORuntime there. It would be interesting perhaps to have a discussion on how people are using cats-effects with akka productively.

At some point I may move to using cats effect without akka on the server (especially if there is a server using http3), and I am already using cats-effect with http4s on the client.

I am writing libraries that can be used on the client and server (including node-js), which is why I have been looking to use the cats abstractions. Two good examples are bblfish/bobcats used by bblfish/httpSig, an implementation of "Signing Http Messages", which I would like to help being used as widely as possible, from client to server on any JS or Java platform.

Perhaps this is a case where using MonadError[Future], Clock[Future] and bobcats.Verifier[Future] are just pragmatically better solutions than setting up a whole threadpool... see discussion on Discord cats-effect Thursday 24 Feb 2022.

[bblfish]

See issue bblfish/httpSig#9 of httpSig and the PR referenced there to see how I used IO.

[alcortaj]

Thank you for the great discussion @ppurang and @djspiewak.

I have a use case where I need to create my own runtime to limit the number of threads that will hit a third-party API since it has QPS limitations. Does this sound like a reasonable use of a custom runtime?

Also, maybe this belongs to a different discussion but I'll throw it out there. What's is the expected behavior here:

IO(...)
  .evalOn(myExecutionContext)
  .unsafeRunSync() // This requires the implicit IORuntime

I understand that unsafeRunSync shouldn't be called on prod code and instead you can rely on IOApp, however, let's say that this is a scenario that some of us face :). If I want my IO to be evaluated on myExecutionContext, what's the guidance about the unsafeRunSync, if the global runtime is defined on the scope, will it override myExecutionContext? Should a runtime be created with myExecutionContext to ensure that this pool is used?

Thanks!

[armanbilge]

I have a use case where I need to create my own runtime to limit the number of threads that will hit a third-party API since it has QPS limitations. Does this sound like a reasonable use of a custom runtime?

Can you use a semaphore here instead?
https://typelevel.org/cats-effect/docs/std/semaphore

[armanbilge]

If I want my IO to be evaluated on myExecutionContext, what's the guidance about the unsafeRunSync, if the global runtime is defined on the scope, will it override myExecutionContext?

No, that would be unlawful :) everything in your IO(...) will run on myExecutionContext, although anything outside of that would run on the compute pool defined in the IORuntime.

[armanbilge]

To clarify: if you have

val foo = IO(...)
val bar = IO(...)
val baz = IO(...)

val program = (foo *> bar).evalOn(myEC) *> baz

then foo *> bar would run on myEC and baz would run on the IORuntime compute pool.

[armanbilge]

You may also find upperbound useful for QPS limits: https://github.com/systemfw/upperbound

[alcortaj]

Thank you @armanbilge, your suggestion and explanations are super helpful!