Pass state to the data sources in a type-safe way

[ghost]

Currently, DataSource#fetch looks like this:

trait DataSource[Identity, Result] {
  def fetch(ids: NonEmptyList[Identity]): Query[Map[Identity, Result]]
}

The fetch method of a data source just receives a non empty list of identities. If we want to inyect some state to our data sources (for example an HTTP client to the data sources that make HTTP calls, a connection pool to the data sources that query a database, and so on) we must use the mechanisms that Scala gives us (implicits et al) since it's not directly supported by the library.

It may make sense to support passing state in a type-safe way to the data sources and provide the concrete values when running a Fetch, much like Haxl does. Not sure about how it'd look like yet, but when running a fetch we'd have to provide an additional value with the inyected state. Can the type system make sure that we are providing the state for every data source used inside a fetch? Should this be supported by the libary?

[AlecZorab]

as I discovered today, a moderately satisfying solution to this is to fetch a new datasource, using the resource as an id. As an example:

val config : Config = ???
def getHttpClient(c:Config): Fetch[HttpClient] = Fetch(c)
def getDataSource(httpClient: HttpClient) : Fetch[DataSource[Id, Data]] = Fetch(httpClient)
def getData(id: Id): Fetch[Data] = 
  for {
    h  <- getHttpClient(config)
    ds <- getDataSource(h)
    d  <- Fetch(id)(ds)
  } yield d

It's unfortunate that the new datasource has to be passed in manually, but there's currently no way to mark variables in a for comprehension as implicit

[raulraja]

Would something like this fit the use case? Implicits is a natural way to inject dependencies in implementations.

class HttpDataSource[I, R](implicit H: HttpClient) extends DataSource[I, R] {
   def fetch(ids: NonEmptyList[I]): Eval[Map[I, R]] = H.get(...)
}
implicit def dataSource[I, R](implicit H: HttpClient): DataSource[I, R] = 
  new HttpDataSource[I, R]

Fetch(id) // datasource received implicitly

Other than that @AlecZorab example is very reasonable.
I can think of other ways to have MonadReader|State, etc.. constrains on the final M[_] and user supported algebras reifying effects such as reader, state, etc... but we'd have to deal internally with Coproduct or something similar to compose user defined ones vs Fetch internals which is overkill for this use case.

[AlecZorab]

Sadly, if you generate a new data source for each fetch then multiple queries from the "same" source won't be aggregated. That's why I ended up fetching the source instead

[AlecZorab]

Actually, I say that, but I didn't investigate whether a proper equality definition would fix it.

[AlecZorab]

can confirm that if you have a proper equality definition for the datasource then you can do

def getToken(): Fetch[Token] = ???
implicit def fixingSource(implicit t:Token): DataSource[Id, Data] = ???
def getData(id:Id) : Fetch[Data] = ???
getToken().flatMap(implicit t => getData(id))

Which will, perhaps, get a bit unwieldy in larger blocks, but is bearable for smaller

[gregbeech]

It would be great to have not only the ability to pass config/etc. in, but also to be able to output values such as logs and metrics in a functional way (i.e. effectively ReaderWriterState).

[raulraja]

@gregbeech you can do that today with the Freestyle Fetch integration http://frees.io/docs/integrations/fetch/ using the FetchM algebra since Freestyle also has effects for RWS and you would not need transformers to nest Fetch inside ReaderT or friends.

[peterneyens]

@gregbeech Fetch currently already exposes some of that information through Env, you could access the multiple rounds with the request executed in that round and how long that request took.

You get both the result and the environment by calling .runFetch (instead of the more commonly used .run).

Would you like to see some additional logs/metrics?