Heavy usage of ThreadLocals when using default ExecutionModel

[ghik]

While profiling our application, we have noticed that Monix creates a ton of ThreadLocal instances.

After some analysis, we found that a ThreadLocal is created every time a Task is run with one of the impure run* methods (e.g. .runToFuture()). Unfortunately, our codebase has a lot of these invocations, having evolved from something that was not written in purely functional style.

Specifically, these ThreadLocal instances are created in monix.eval.internal.FrameIndexRef.Local and this only happens when ExecutionModel.BatchedExecution (the default one) is used. After switching to ExecutionModel.SynchronousExecution, the ThreadLocals were gone and we measured a performance improvement - visibly less GC pressure and CPU usage.

It might seem weird that ThreadLocals generate more GC pressure than any other heap object. However, if one looks closely into how the JVM manages thread locals (ThreadLocal.ThreadLocalMap), one will notice that while ThreadLocals themselves are kept in weak references, their values are stored in strong references and reclaimed later, in some more "manual" way rather than simply letting GC do its work. Therefore I hypothesize that thread local values live in memory much longer than expected and are likely to be promoted to older GC generations, therefore unnaturally increasing GC pressure. Our performance tests seem to be consistent with this hypothesis.

I'm not sure why JDK is doing this but it may be the case that ThreadLocals were designed to be long-lived (e.g. kept in static fields) rather than frequently created in large numbers.

This isn't necessarily a bug in Monix and the problem may be specific to our codebase but it is an interesting observation about ThreadLocals nevertheless - which suggests they should be used sparingly.

[rfkm]

I am suffering from a similar problem. After running my application for a long period of time (about a few weeks), ThreadLocalMap becomes bloated and performance degrades. Eventually, I have to restart the application.

Most of ThreadLocals keep integer, so I think our problem is also probably due to monix.eval.internal.FrameIndexRef.Local.

I have not yet identified which code is causing the problem. My application does not directly call the run* method in large numbers, but it does use methods such as Task.start. Also, I don't know if this is relevant, but we are using blocking context sometimes. (Since there seem to be several recent reports of blocking context issues, including #1676 that I reported, I am going to avoid using blocking context and see what happens. Hopefully that will solve the problem.)