Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement callbacks to Akka's FileIO whenever writing successfully adds a chunk to a file #197

Open
1 task
mdedetrich opened this issue Apr 14, 2022 · 0 comments
Labels
file Specifically related to flat file based backend upstream Issues that require changes/co-operation with upstream libraries/projects

Comments

@mdedetrich
Copy link
Collaborator

What is currently missing?

At some point we would like to add file based backups to Guardian.

How could this be improved?

Currently this is not possible because Akka's FileIO doesn't allow you to add a "callback" whenever a chunk of bytes have been successfully written to disk. The callback is required to commit the current Kafka cursor and hence the set of changes required are similar in concept to what was done with Alpakka's S3 client (see akka/alpakka#2770). Here is the location where the current FileIO actually writes the data to disk https://github.com/akka/akka/blob/7abc41cf4e7e8827393b181cd06c5f8ea684e696/akka-stream/src/main/scala/akka/stream/impl/io/FileOutputStage.scala#L63

Is this a feature you would work on yourself?

  • I plan to open a pull request for this feature
@mdedetrich mdedetrich added the upstream Issues that require changes/co-operation with upstream libraries/projects label Apr 14, 2022
@mdedetrich mdedetrich added the file Specifically related to flat file based backend label May 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
file Specifically related to flat file based backend upstream Issues that require changes/co-operation with upstream libraries/projects
Projects
None yet
Development

No branches or pull requests

1 participant