Research usage of big Payloads

[whyoleg]

On current moment Payload contains 2 properties data and metadata which can store bytes. The problem is, that those properties store ByteReadPacket which works like in-memory buffer. So f.e. to send big file through in one Payload(+fragmentation), it will be needed to firstly read everything from file to ByteReadPacket, so in memory.

Need to research, if and how it's possible to send such big payloads without reading everything to memory

[yschimke]

My mental model has been around an individual payload being available via a stream API, like Channel, InputStream/OutputStream or similar.

[whyoleg]

For inspiration: ktor-io byte channels
Also it has integration with File IO and JVM IO-streams.

[yschimke]

I'd be interested in hooking it into the CLI once it's barely working. Can testing with streaming STDIN/STDOUT pipes etc.

[olme04]

better to postpone this task until ktor/ktor-io will have 2.0.0 version, which will change some Input API.

[olme04]

POC api change: olme04@5946241#diff-b20323f12dc29bebdc7845234a305c8f22b468cb7c98964cb14ad808696ec65f
Main breaking change is that when using Input instead of BRP we will loose possibility to copy payload (or data/metadata).
This will affect user code.

[whyoleg]

Someday I think we will need to migrate to Kotlin/kotlinx-io#131, though, it depends on how well it will interop with ktor and so ktor-io, as this is the main transport now.