Improve import experience

[keynmol]

Historically, the weathered scrolls which held the account of what http4s imports one needs have been passed down generation after generation. Elders gathered at the town square and chanted the sacred words:

  import org.http4s.dsl.io.{*, given}
  import org.http4s.implicits.{*, given}
  import org.http4s.ember.server.{EmberServerBuilder}
  import org.http4s.ember.client.{EmberClientBuilder}
  import org.http4s.HttpRoutes
  import org.http4s.headers.*

But centuries have passed, scrolls were replaced by screens, and the chance of remembering which exact imports do what acted as the evolutionary barrier, killing the weakest.

---

For Http4s 1.0.0 at least (ideally much earlier) it would be good to offer the users a simplified org.http4s.all.* import.

Good for scripting, good for getting started docs, hell, good for experienced people who forget where exactly GET and Root are.

Or move all that into the import org.http4s.dsl.io.

Let's discuss!

[bplommer]

This sounds like it would require rethinking the module structure as well, so that the all package would live in some kind of bundle module that depends on the Ember and DSL modules.

[keynmol]

I actually put Ember there accidentally, I believe a good starting point would be to just reduce all the core http4s imports to a single one.

A bundle module with pre-baked stuff (ember server/client + circe) is also possible, but should be a separate artifact and will probably be harder to get agreed on.

[zetashift]

Because an simple up-vote doesn't express how much I would love something like this, I'll express it in words.
The amount of imports might not be a (big) problem if you already know http4s or if you're already well versed in advanced Scala/FP libraries. But for a beginner experience it adds a certain kind of confusion.

This was my experience:

• Browse the docs and stumble upon: https://http4s.org/v1/docs/service.html
  Defining a first service, nice! I guess in total I need the following imports?

import cats.effect._, org.http4s._, org.http4s.dsl.io._
// import cats.effect.unsafe.IORuntime I'll leave this one out because in scala-cli I could use IOApp
import cats.syntax.all._
import com.comcast.ip4s._
import org.http4s.ember.server._
import org.http4s.implicits._
import org.http4s.server.Router
import scala.concurrent.duration._

The problem is the docs don't explain these imports, and my first instinct is what is all these things for? Isn't this all too much? What do they all do? The reader should just "assume" these things, but personally I got confused, because it feels like I need to know what all these namespaces contain, but I just want my server to print Hello World 🥲 !

• Another example from a great resource: https://blog.rockthejvm.com/http4s-tutorial/

import cats._
import cats.effect._
import cats.implicits._
import org.http4s.circe._
import org.http4s._
import io.circe.generic.auto._
import io.circe.syntax._
import org.http4s.dsl._
import org.http4s.dsl.impl._
import org.http4s.headers._
import org.http4s.implicits._
import org.http4s.server._

Oh my...well atleast these are all the exports I need for the tutorial itself, so it isn't bad. But it was another gasp moment.

But even after that initial stumble, it seems that a surprising amount of problems I had were solved by some sort of import lib.syntax.* line, can't this be done with less friction?

Editor tooling help a lot with missing imports, but in certain libraries(like Cask, Express, http in Go), it's really really easy to just get started. For example in Giraffe(think http4s for F#):

open System
open Microsoft.AspNetCore.Builder
open Microsoft.AspNetCore.Hosting
open Microsoft.Extensions.Hosting
open Microsoft.Extensions.Logging
open Microsoft.Extensions.DependencyInjection
open Giraffe

Even though it might not be easier than a Go or JS http server, it still is less intimidating than http4s, for a beginner.

So my problem with imports might itself be more a docs problem than the amount of imports, and I could even make a snippet that just expands to the "standard" imports...but good/nice defaults matter imho and I think http4s can be nicer in that way.

[armanbilge]

I just floated a (crazy?) idea in #7053 (comment) that we should collapse core+server+client+ember into a single http4s artifact.

One compelling motivation for this would be so that you can do something like Client.default[IO] and immediately get a Resource[IO, Client[IO]] that is a default Ember client. This would be a significant acceleration and a win for our toolkit as well.

[rossabaker]

The founding vision of the project was that core would be the Rack / WSGI / WAI / Servlet¹ of Scala: a neutral abstraction with multiple implementations. Of these, only WSGI bundles a concrete implementation², which is not recommended for production. Recommending a default implementation to orient new users is one thing, but to impose it on the classpath of users who make other tradeoffs would be an audacious step.

• Is Ember generally recommended today? Yes.
• Has the recommendation changed multiple times over time? Yes.
• Is there anything inherent in Ember that makes it the only natural implementation evermore? No.

Ember is fine work, but the people who work on it don't even agree on what it should be.

• Pure FP innards vs. every nanosecond is precious
• Observability baked in vs. a minimalist implementation of the spec

The fact that these debates rage on within just one backend is evidence that there's not One Divinely Inspired Http4s Implementation. A toolkit favoring an implementation is reasonable. Toolkits are curated sets of opinions, and if you don't like it, go pick another toolkit. But http4s-core is an abstraction.

Moving client and server in, as long as they don't add dependencies, seems fine.

Footnotes

1. the Servlet analogy is complicated by the fact it's one of our backends, but that's the role it plays in Java. ↩

2. wsgrief, now bundled in the Python standard library. ↩

[armanbilge]

but to impose it on the classpath of users who make other tradeoffs would be an audacious step.

Eh, personally, I don't really buy this 🙂 I would say Ember is pretty lightweight actually. The only dependencies that it adds are KeyPool and HPACK. Besides, we impose stuff on the classpath all the time, like IO, which has not compromised our effect-neutrality 🙂

there's not One Divinely Inspired Http4s Implementation

I say Ember is about as close as it gets ✨ Its "divinity" comes from the fact that it inhabits the space between "implement http4s Server / Client abstractions", "use FS2 for all I/O", and "try not to reach for more than Temporal". Between these constraints, there is not that much design space. I am confident that we all want something maintainable, observable, and performant. The fact that it is based on an FP I/O library already bounds what this looks like.

Furthermore, because it inhabits those constraints, Ember is the only backend that is well-poised to take advantage of the I/O-integrated runtime work in Cats Effect. I think there is "divinity" in how deeply it integrates with the rest of the stack. (Admittedly, the performance benefits of this new runtime remain to be seen :)

The founding vision of the project was that core would be the Rack / WSGI / WAI / Servlet of Scala: a neutral abstraction with multiple implementations

Yes indeed, and we should definitely not lose sight of this vision. I maintain a handful of non-Ember implementations! Meanwhile, we can dramatically enhance the approachability of http4s for the growingly typical usecase without compromising the abstractions at all.

[rossabaker]

There are mixed messages between Cats Effect stratifying modules and FS2 later embracing IO, but as far as I remember, DSL is our only use of IO. It's also possible I'm overlooking some recent changes; I decided it wasn't worth spending my argument tokens on after the Monix folks went quiet.

"Just FS2 IO" and "just Temporal" are not sufficient to determine a reference implementation vs. internal complexity to address production concerns. And, again, it's not just performance, but what features are in and out of scope. I have tried a couple times to establish principles for these tradeoffs and got caught in a crossfire that was more emotion than reason from multiple corners. That's not unexpected when it attracts such smart people. Ember's in capable enough hands that I felt comfortable stepping back, but I also have had to unsubscribe from most of its issues to manage my own burnout. I'm not sharing that to be dour, but to remind everyone that even within Ember's rigid constraints, there's no Pareto-efficient implementation.

It wasn't that long ago people believed fs2-netty was the only appropriate foundation due to all the bugs in NIO. It wasn't that long ago that a canonical implementation only had to be the best fit on the JVM. I'm not begging for more implementations, and am certainly too tired to launch another myself, but backend innovation historically outpaces core. I am in 100% agreement with simplifying imports in the spirit of the original post, but I disagree with this one.

[bplommer]

Would it be too weird to keep the existing modules, add a new http4s module that depends on the other modules and aliases various types such as Client into the org.http4s package, but provide a separate version of the companion object with an extra constructor for a Blaze client? Something like this:

// in http4s-client
package org.http4s.client {
  abstract class Client[F[_]] { /** */ }
  private[http4s] class ClientCompanion { /** */ }
  object Client extends ClientCompanion
}

// in new http4s module (depends on Ember)
package org.http4s {
  type Client[F[_]] = client.Client[F]
  object Client extends client.ClientCompanion {
    def default[F[_]]: Resource[F, Client[F]] = ??? // Ember client
  }
}

Edit: On second thought this might not be ideal - the object org.http4s.Client wouldn't actually be the companion for the type org.http4s.Client, so it wouldn't facilitate bringing any extra implicits into scope.

[armanbilge]

as far as I remember, DSL is our only use of IO

Exactly! Even though our only use of IO is in the DSLs, it's on the classpath of core anyway. I'm proposing the same for Ember: expose it as a Client.default (and maybe a Server.default) implementation to accelerate usability but otherwise leave the core library implementation-neutral. It's a highly useful convenience that doesn't get in the way of anybody that doesn't want it.

Let me refactor my proposal slightly: it's less specific to Ember, it's mostly about having a default client/server so that http4s can work out-of-the-box for the "I just want a client / server, let's go!" usecase with minimal imports.

Ember currently happens to be the best candidate for this. But we can do this in a way where we don't leak the fact that the implementation happens to be Ember. So in the future we can change it if we want.

I appreciate your exposition on Ember development, I'm not sure if I was aware of all of that. I will keep it in mind.

---

Edit: On second thought this might not be ideal - the object org.http4s.Client wouldn't actually be the companion for the type org.http4s.Client, so it wouldn't facilitate bringing any extra implicits into scope.

Thanks, I did think about this as well. I think this sort of encoding gets complicated enough that it starts to trade down the improved usability we're going for.

FWIW Cats Effect is currently modularized into cats.effect.kernel._ that is aliased in cats.effect._ and we've talked about flattening this as well in the next major version. Similar tradeoffs, and in practice a lot of libraries have CE core on the classpath anyway, so all the modularization and aliasing ends up moot.

[Jasper-M]

Don't forget import org.http4s.client.dsl.io._

[bplommer]

Are we just waiting at this point for someone to go ahead and open a strawman PR for this?

Another thing that I was thinking about was to alias relevant ip4s types into the http4s package, as it feels weird and surprising having comcast stuff showing up in the core http4s model - what do people think?

[danicheg]

Another thing that I was thinking about was to alias relevant ip4s types into the http4s package, as it feels weird and surprising having comcast stuff showing up in the core http4s model - what do people think?

💯 if it was at least org.typelevel stuff...

[kubukoz]

Either into http4s or http4s.ember. Either would be better than the comcast import :)