Is there a way to await async operations inside of .Select(...), .CrossApply(...), .Do(...)?

[eisbaer66]

Can is use async/await inside of .Select(...), .CrossApply(...), .Do(...)?
Do i have to call .Result/.Wait() myself? Is this safe?

for example calling a REST api and reading the content as string:

stream.Select("get content", async context => await context.httpClient.GetStringAsync(context.url));

[paillave]

I will work on an example asap...
I might even come with couple of amendments to handle this natively.
I make an issue for this, I will come back to you asap

[paillave]

Just for your information, an extension dedicated for REST makes part of my plans.
But to answer your question,????Async().Result should work properly.
You must know that ETL.NET 100% thread safe (that was even the most painful part of its development 😄 !!!)

[eisbaer66]

Good to know ????Async().Result will work, but it's a bit clunky. It would be great if actually awaiting would work, too.

Will the existing file-reading-methods (for example to read csv-files) be extended, so I can use a stream directly (for example from a REST call)?
Or will I have to write it to a file first? skipping that kind of disk IO might increase throughput quite a bit.
(Probably doesn't make a difference for what i'm doing right now, but it would be great in general, I think)

[paillave]

The reason why async methods are not handled too gracefully is because for now, I chose that operators are the only ones who control when what is sent. Internally, many processes are parallelized, but inside these threads elements are handled sequentially. So I would end up making a .Result on returned tasks any way unless I change something in the core plumbing.

For the in-memory file, you must wrap your Stream into a IFileValue (the type that is natively used by ETL.NET to process files) by using the InMemoryFileValue.

Here is how to do it:

[...]
myBlobStream     // let's suppose `myBlobStream` emits instances of objects like { BlobStream: System.IO.Stream, FileName: string}
    .Select("Create File Value", elt => FileValue.Create(elt.BlobStream, elt.FileName, "<give a name that permits to know the source of your file>"))
    .CrossApplyFlatFile([...])
[...]