WriterT[IO.. losing logs in parEvalMap hints at problem with Concurrent instance for WriterT ?

[benhutchison]

I'm seeing FS2's parEvalMap dropping logs emitted by WriterT[IO.., while the sequential cousin evalMap retains them, as does a "desugared" Writer using parEvalMap.

Since parEvalMap is written in terms of Concurrent this hints at a potential problem in the WriterT instance, although I was not able to spot a problem from a visual inspection.

import fs2.*
import cats.*
import cats.syntax.all.*
import cats.data.*
import cats.effect.*
import cats.effect.implicits.*

object ConcurrentWriterTTest extends IOApp.Simple:
  def tell(s: String): WriterT[IO, Chain[String], Unit] = WriterT.tell(Chain(s))

  val run = Stream(1)
    .parEvalMap(Int.MaxValue): _ =>
      tell("loglog")

    .compile.drain.written.flatMap(IO.println) //output: Chain()

object SequentialWriterTTest extends IOApp.Simple:
  def tell(s: String): WriterT[IO, Chain[String], Unit] = WriterT.tell(Chain(s))

  val run = Stream(1)
    .evalMap: _ =>
      tell("loglog")

    .compile.drain.written.flatMap(IO.println) //output: Chain(loglog)

object ConcurrentWriterDesugaredTest extends IOApp.Simple:
  def tell(s: String): IO[(Unit, Chain[String])] = IO.pure(() -> Chain(s))

  val run = Stream(1)
    .parEvalMap(Int.MaxValue): _ =>
      tell("loglog")

    .compile.toList.map(_.map(_._2)).flatMap(IO.println) //output: List(Chain(loglog))

Cats Effect 3.5.1 and FS2 3.7.0

[armanbilge]

See some related discussion in typelevel/fs2#3246 (comment).

tl;dr don't use monad transformer datatypes. You can replicate Writer-like semantics with Ref and you can even hide it behind the Tell abstraction in Cats MTL.

[benhutchison]

I want to respond on two levels..

1. Cats Effect ships with instances for Concurrent for WriterT, and many others.

There's a few options that IMO are OK

• The instances are supported and bugs investigated
• The instances are withdrawn & removed

..and one that is kind of bad:

• the instances remain out of inertia but don't work properly and are unsupported. When people point problems out, they are told "oh don't bother with them, they don't actually work".

Now I can't actually envisage why WriterT couldn't be supported for parEvalMap by monoidally combining the logs from the various concurrent fibers. I'll bet there's a fixable bug somewhere.

There seems a tendency to bucket anything MT-related together, but it's more nuanced than that. There's a class of MTs like EitherT that are short-circuiting and seem to be problematic with the CE error channel, but WriterT isn't of that form.

When I inquired about this issue on TL Discord, Fabio's response actually pertained to a different situation, around error-handling, where WriterT drops logs on error (I actually believe the correct way to handle that is to raise an error of the form (E, L), ie the error E and any logs L accumulated up to the occurrence of the error).

Level two ,"Why are MTs worth bothering with anyway?" is more ideological and will come in a second comment..

[benhutchison]

Level 2: Why are MTs worth bothering with anyway?

I retain an interest in monad transformers not because I love them in themselves - actually they are a confusing pain - but because I retain a naive but as yet unshaken belief in simple functional programming.

By which I mean Runar's definition of functional programming, namely "programming with functions". Pure mappings whose only effect on the world is through their return type.

In the world of pure functions, state dependence manifests via a type signature like (A, S) => (B, S), logging via a signature like (A) => (B, L), errors via (A) => Either[E, B].

These kinds of pure signatures are, to me at least, intuitive, easy to reason about, and easy to test. Ease of testing remains one of the most compelling practical advantages of FP. I want to be able to write the computational pieces of my program using forms like those above. Monad Transformers let me write simple, pure code in the small, and yet lift it into a larger effectful program fairly gracefully.

It does not seem as nice to carry around a mutable Ref side-channel to store state or logs in. Pure functions become effectful. Type signatures start to "lie", in that they now omit some of the effects of a function. Asserting in a test becomes less simple.

[SystemFw]

In the world of pure functions, state dependence manifests via a type signature like (A, S) => (B, S), logging via a signature like (A) => (B, L), errors via (A) => Either[E, B].

I tried to make this point several times in the discussion you linked, but I think I've failed to convey it: in that world, you cannot use real concurrency nor native errors, so it's incompatible with IO. If you're willing to sacrifice that, i.e. having single-threaded simulated concurrency through yet another transformer (FreeT), and make sure that Exceptions are never thrown but always surfaced as an Either, then you might be able to achieve what you want, if not, this expectation is unrealistic.

Now I can't actually envisage why WriterT couldn't be supported for parEvalMap by monoidally combining the logs from the various concurrent fibers
a different situation, around error-handling, where WriterT drops logs on error

I was actually trying to explain that it's a simpler case of the same issue: you need native mutable state, and WriterT gives you simulated state via function passing instead. In the error case, you need native mutable state because you are dealing with native exceptions, in the concurrency case, you need native mutable state to deal with concurrency. Not only that, but because fibers can fail and be interrupted, you also have to deal with the error scenario as well : where do you get the logs in (A, L) if the fiber gets interrupted? again, you need to store them in a Ref.

[benhutchison]

In the world of pure functions, state dependence manifests via a type signature like (A, S) => (B, S), logging via a signature like (A) => (B, L), errors via (A) => Either[E, B].

I tried to make this point several times in the discussion you linked, but I think I've failed to convey it: in that world, you cannot use real concurrency nor native errors, so it's incompatible with IO.

So a design goal I'm seeking for is that the individual pieces of (non-concurrent, non-exception-throwing) domain logic in my program are phrased as pure functions with simple signatures like above. What I think you're pointing out is that, as the program is composed & layered, as it scales up, as concurrency, 3rd party libs, io, and inevitably exceptions get involved, these simple forms aren't sufficient any more.

And parEvalMap is one of those boundaries, introducing concurrency.

At the moment I'm thinking about ways to transition from logging as (A) => (B, L) in the small, to using a Ref in the large.

[benhutchison]

Stepping back a bit though, I do think this issue and others that have appeared in the last year or so are unsettling.

If you read the blurb on CE3 it says something like:

• We have carefully designed typeclasses that model asynchronous and concurrent behavior, in the presence of errors and interruption
• Our typeclasses have laws/invariants
• We have tests for these laws

And have an implementation of GenConcurrent[WriterT[IO, L, *]] that type checks and passes law checks, and yet it deadlocks, and drops logs, even when there are no errors.

That's troubling other people than me, right..?

Where's the gap..

• Is it that the instance is lawful, but not all essential qualities are described by the laws, and hence instances can still deadlock, lose logs and otherwise misbehave to the point of not being practically useful?
• Or the laws are good enough, but this instance isn't truly lawful, but the test suite isn't able to catch the violations?

[djspiewak]

Is it that the instance is lawful, but not all essential qualities are described by the laws, and hence instances can still deadlock, lose logs and otherwise misbehave to the point of not being practically useful?

Very briefly, it's basically this.

In particular, WriterT's behavior exceeds what is specified by Monad (or even Async), and we can't make any particular guarantees about what that means. Put more formally: the laws are consistent, but they are not complete. Behavior exists which is outside what the laws guarantee.

We can see this even with just IO. For example, there are no laws which stipulate how IO.realTime should behave, much less weirder things like IOLocal or IO.trace. Does that mean these functions are bad somehow? Not at all! But we certainly cannot reason about them in the way that we might reason about other things within the calculus.

The laws only guarantee the substitutions present in their formulation, nothing more or less. Where we get ourselves into trouble is when we infer "intuitive" behavior which is unspecified by any laws, we just happen to expect it. In this case, you're expecting WriterT.tell to not lose messages. This is certainly intuitive behavior, but there's no substitution which guarantees this. In a sense, you're expecting more than the system promises you.

Clearly there's a balance here though. We also don't guarantee a bunch of things that you can do with async callbacks, because formalizing that type of thing in substitution calculus is implausibly complicated. Or, even more simply, Ref and Deferred are both outside the laws (something which makes me wildly uncomfortable). It must be valid to use behavior which is not lawfully specified, from which it follows that programs will not be, in general, generated from the laws. And from that it follows that much of program behavior will be unpredicted by the laws, at which point all bets are off on whether it's "right" or "wrong" or "surprising".

[armanbilge]

At the moment I'm thinking about ways to transition from logging as (A) => (B, L) in the small, to using a Ref in the large.

I'm a broken record, but I highly recommend to use the Tell abstraction from Cats MTL.

https://typelevel.org/cats-mtl/mtl-classes/tell.html

It abstracts the main idea of WriterT. To satisfy the the Tell constraint, you can use either WriterT or Ref, but code written in terms of Tell doesn't need to know that.

These kinds of pure signatures are, to me at least, intuitive, easy to reason about, and easy to test. Ease of testing remains one of the most compelling practical advantages of FP.

IMHO a signature written in terms of Tell satisfies these criteria. If you feel that it does not, I would be interested to hear more about why you think that.

---

Is it that the instance is lawful, but not all essential qualities are described by the laws

When we say WriterT has a lawful implementation of GenConcurrent, we mean that it satisfies the laws defined by Cats Effect specifically for the GenConcurrent typeclass.

Notice that these laws have no specific notion of L or the "logging capability" of WriterT, because this is not a feature of the GenConcurrent typeclass. Therefore, the laws for that typeclass cannot make assertions specifically about L, because it does not know about it.

The "gap" is that you know about L and have expectations of its behavior that are not covered in the laws, since they lack specific knowledge of L.

Issues like this crop up in other places as well. For example consider #3079. The original implementation of Resource#memoize was completely lawful, but its behavior was surprising because the GenConcurrent laws cannot make assertions about the behavior of Resource finalizers because they lack specific knowledge of that. In that case, we were able to replace it with another implementation that is lawful but has the expected behavior for finalizers.

---

and yet it deadlocks

Sorry, I missed this. Do you have a reference issue about WriterT deadlocking?

[benhutchison]

and yet it deadlocks

Sorry, I missed this. Do you have a reference issue about WriterT deadlocking?

No, Im sorry.. 😏 Loose talk on my part, I was referring to the EitherT deadlock you're well across..

[armanbilge]

Loose talk on my part, I was referring to the EitherT deadlock you're well across..

No problem, thanks for clarifying :) So I can answer that question now too, and it's essentially the same situation as WriterT.

Specifically, when you have EitherT[F, E, *] you actually have two error channels, one on F[_] and the other on EitherT itself (technically, on Either). However, the GenConcurrent laws do not know about EitherT's error channel. Therefore all lawful behavior is described with respect to errors raised on F[_].

Indeed, if you only ever use GenConcurrent[EitherT[F, E, *]].raiseError(...) I can guarantee you will never encounter those deadlock issues because that's precisely what the laws guarantee. The issue is when you work outside of the typeclass e.g. raising an error with EitherT.leftT[F, E](...). The typeclass does not know about this error channel and thus the laws make no guarantees about its behavior.

The tl;dr here is that laws can only make guarantees about the behavior of implementation of the typeclass, not "special" behaviors of the monad. So when you use the monad's "special" features, their behavior is not specified by the Cats Effect laws. This is why the behavior can be both lawful but "surprising".

[armanbilge]

Btw, I'm a broken record again, once again the solution is to use Cats MTL. Instead of EitherT use Handle.

https://typelevel.org/cats-mtl/mtl-classes/handle.html

The Handle constraint can then be satisfied with EitherT or directly with IO as demonstrated in typelevel/cats-mtl#489. Most importantly, when you use the IO-based implementation you will not experience deadlocks, because the implementation in that PR uses a single error channel whose behavior is fully specified by the laws.

[benhutchison]

Agree, typelevel/cats-mtl#489 looks promising, would like to use.. Um, what is it waiting on?

[armanbilge]

looks promising, would like to use..

Great, feel free to write a comment there! Feedback on naming / API would be helpful.

Um, what is it waiting on?

Nothing in particular. We're all just extremely busy :)

[SystemFw]

What I think you're pointing out is that, as the program is composed & layered, as it scales up, as concurrency, 3rd party libs, io, and inevitably exceptions get involved, these simple forms aren't sufficient any more.

Yes, think about this: the platform provides you with several primitives: 1) calling subroutines, 2) creating data, 3) mutating references, 4) creating system threads, 5) introducing synchronisation. These capabilities cannot be expressed in terms of each other, otherwise they won't be primitives.

In the world of pure functions, state dependence manifests via a type signature like (A, S) => (B, S), logging via a signature like (A) => (B, L), errors via (A) => Either[E, B].

With this, you're stating that basically you only want to use primitives 1) and 2) , and so by definition you won't be able to deal with primitives 3, 4 & 5, and you'd have to necessarily resort to a version of concurrency that uses FreeT without being able to use systems threads (basically no multi-core).

The ingenious trick or, depending on your perspective, the ugly hack that IO does is to let you access the primitives of the underlying platform whilst preserving a key characteristic of pure functions, i.e. referential transparency.

---

These kinds of pure signatures are, to me at least, intuitive, easy to reason about, and easy to test. Ease of testing remains one of the most compelling practical advantages of FP. I want to be able to write the computational pieces of my program using forms like those above. Monad Transformers let me write simple, pure code in the small, and yet lift it into a larger effectful program fairly gracefully. It does not seem as nice to carry around a mutable Ref side-channel to store state or logs in. Pure functions become effectful. Type signatures start to "lie", in that they now omit some of the effects of a function. Asserting in a test becomes less simple.

Now, let me challenge these statements a little bit.

Pure functions become effectful

It's actually hard to find a definition of purity that justifies this statement. Both StateT/ WriterT/ EitherT and IO are referentially transparent in the exact same way. Both force your code to be written in monadic style. The only difference is that StateT/WriterT/EitherT can be completely eliminated at any point in the program, whilst IO can only be eliminated in main, but I would like to understand what specific advantage you are getting out of that

easy to reason about

Well, in a way the very existence of this issue kinda disproves it but let's entertain the statement that all things being equal things like WriterT and EitherT are easier to reason about that IO and Ref. For one, I would say that the fact that WriterT[EitherT[Id, Err, *],L, A] and EitherT[WriterT[Id, L, *], Err, A] differ in the semantics of their state tends to be a point of confusion for a lot of people. But really, the issue is that all things are not equal, since they don't have the same power of IO, and once you replace Id with IO in those transformers, you start hitting the various issues we've been discussing

Type signatures start to "lie", in that they now omit some of the effects of a function

Now, this might be true if you compare:

def myFun(a: Int): WriterT[IO, Bar, String]

with

case class Foo(logs: Ref[IO, Bar]) {
  def myFun(a: Int): IO[String]
}

(although again, the first is fundamentally not as powerful as the second), but I don't think it's true for something like:

trait Logs[F[_]] {
  def log(b: Bar): F[Unit]
 } 
def myFun[F[_]: Logs : Concurrent](a: Int): F[String]

which you can implement over Kleisli[IO, Ref[IO, Bar], *] or by bundling a Ref in Logs[IO] or by using cats-mtl Tell[F, Bar]. This alternative has been proposed a few times and you haven't engaged with it, why don't you like it? (this is genuine curiosity, I personally don't value all this info in type signatures, but if you do, it seems to me you have plenty.)

easy to test. Ease of testing remains one of the most compelling practical advantages of FP. Asserting in a test becomes less simple.

I'll give you that doing:

def myFun(a: Int): WriterT[IO, Logs, String] = ...
myFun(3).run.flatMap { case (logs, value) =>
   assertions(logs) >> assertions(value)
 }

it's slightly easier than:

def myFun[F[_]: Logs : Concurrent](a: Int): F[String] = ...

IO.ref(emptyBar).flatMap { state =>
  implicit val myLogs: Logs[IO] = Logs.fromRef(state)
  myFun(3).mproduct(state.get).flatMap { case (logs, value) =>
      assertions(logs) >> assertions(value)

but is it that much simpler? Especially when you consider that you can extend Logs directly with functions to help you test.

---

..and one that is kind of bad:

the instances remain out of inertia but don't work properly and are unsupported. When people point problems out, they are told "oh don't bother with them, they don't actually work".

I have to agree with you on this, however. @armanbilge has explained well why they can pass laws and still behave unexpectedly, but the end result of user confusion is all the same

[benhutchison]

Both StateT/ WriterT/ EitherT and IO are referentially transparent in the exact same way. Both force your code to be written in monadic style. The only difference is that StateT/WriterT/EitherT can be completely eliminated at any point in the program, whilst IO can only be eliminated in main, but I would like to understand what specific advantage you are getting out of that

I don't like to include IO in a signature if a less powerful effect will suffice. Examples:

• a pure function that does some validation of inputs can signal this with EitherT
• a pure function that also emits a non-fatal warning can signal this with WriterT
• a pure function that uses a RNG seed can signal via StateT

If all these are represented using IO + Refs, then it becomes harder to rule out other things the function should not be doing, eg file IO. The MTL-approach, aka capability-traits, seems to find a nice balance though; express minimum requirements, without precluding an IO + Ref implementation in a real system.

or by bundling a Ref in Logs[IO] or by using cats-mtl Tell[F, Bar]. This alternative has been proposed a few times and you haven't engaged with it, why don't you like it?

For this record, the MTL-style looks like the most promising approach, and I have added several things to my near-term TODO list:

• experimenting with migrating parts of my application from WriterT to Tell
• then implementing Tell via WriterT in unit tests, but IO + Ref in application code.

Only reason I haven't commented further is that I haven't yet had a chance to try it out in practice and see how well it works out...

[armanbilge]

I don't like to include IO in a signature if a less powerful effect will suffice. Examples:

* a pure function that does some validation of inputs can signal this with EitherT

* a pure function that also emits a non-fatal warning can signal this with WriterT

* a pure function that uses a RNG seed can signal via StateT

Ah, but EitherT, WriterT, and StateT are still arbitrarily powerful, depending on the transformed monad F[_].

Now, if you had said:

• a pure function that does some validation of inputs can signal this with Either

• a pure function that also emits a non-fatal warning can signal this with Writer

• a pure function that uses a RNG seed can signal via State

I would 100% agree with you. Indeed, if you only use Either, Writer, and State, I can also guarantee you will not experience weird issues or unexpected behaviors like loss of logs or hanging :)

[benhutchison]

OK then we're in violent agreement. I'd actually adopted @SystemFw's terminology since I was replying him, but that was a white lie, I dont use WriterT etc, I use Writer, Either, State when defining building block functions.

but note Writer, Either, State need to be lifted into a MT over IO, or else embedded somehow into IO (eg submarine error propagation which I use a homemade form of), or otherwise dealt with, when composing into an application

[armanbilge]

need to be lifted into a MT over IO

Right, and this is where the issues start :) don't use the MTs.

However It should be possible to add helpers to Cats MTL for lifting Writer, Either, State into Tell, Handle, and Stateful. If it doesn't already exist, I will open a PR for this.

[armanbilge]

• Add Raise#fromEither and friends cats-mtl#514

[armanbilge]

Yes, just for the record I believe we should deprecate these instances. I expressed a similar opinion in typelevel/fs2#3246 (comment).

[djspiewak]

Fwiw, I'm open to deprecating the instances, or at least making the footgun a bit more obvious. I'll note though that doing so is almost the same as outright deprecating the transformers themselves. Not quite yet sold on that on yet. 🙂

[SystemFw]

The transformers can still be stacked on top of each other and Eval

[CJSmith-0141]

I think that deprecating these would be a good idea. They are an easy to reach for tool that has subtle bugs depending on the effect, and if you don't know about the footgun then you can easily find yourself ( like me! ) with EitherT[F, Throwable, B] all over your codebase.

For me the confusion comes from having not really understood that ApplicativeError[F, Throwable] present in IO means that adding EitherT[F, Throwable, B] is essentially adding a second path for errors of the same type to propagate, which is bad.

[djspiewak]

I converted this from an issue into a discussion because it feels more appropriate for this space.

To be clear, I think this is an absolutely fascinating topic and I'm grateful you brought it up, @benhutchison. I have some long-form thoughts that I don't have time to write at present, but I'll try to get to it. This is a very tricky space, and I generally agree with your dissatisfaction with the current state, but the "where to go from here" is a bit less clear to me.

[benhutchison]

This issue caused typelevel/cats-mtl#516

[benhutchison]

An interesting epilog:

As I mentioned earlier, I implemented my log API in terms of 2 impls of Tell, Ref-backed and a WriterT-backed versions.

For simple logging events, both impls worked well. However, I also have a span operation that emits two "bookend" log entries around a region of the program. It's implemented using MonadCancel#bracketCase.

And when I test this using the WriterT Tell impl, the closing bookend supposed to be added by the bracket finalizer is omitted from the logs. Presumably more WriterT misbehavior.

For me, this is the final nail in coffin for my hopes for Writer logging. I give up. IME it isn't usable in practice, even for unit tests.

[djspiewak]

the closing bookend supposed to be added by the bracket finalizer is omitted from the logs. Presumably more WriterT misbehavior.

That unfortunately doesn't surprise me. And I agree it's pretty damning.

The problem here is that onCancel produces F[Unit], which means that there's no way to do the normal trick of lifting F[(A, B)] into WriterT[F, ...], because the tuple is voided away. Now, we could fake our way around this by just sticking a Ref in there, and maybe that would be less surprising behavior (though it would mean that materializing a MonadCancel for WriterT would require Concurrent[F]), but this is a long game of whack-a-mole.