Improved multi-segment file uploads

[rbair23]

We should improve the way we segment a large file into transactions and upload them to the network to be more resilient to failure and to have greater control over when the file is fully uploaded. We had two ideas:

1. We could have a "segment count". When we prepare to upload a file, we can include in the FileUpdateTransactionBody or FileCreateTransactionBody a "1 of X segments" field. This way, the server (and people consuming records in the mirror node) can determine whether all expected file segments have been uploaded.
2. We could have a "file hash" added to those transactions to keep track of the completed hash. Each FileAppendTransactionBody we could have the rolling hash of the file up to that point in time. In this way, you could have a strong guarantee that the file in state is what you expect before applying a segment.

Maybe we do both of the above.

[bugbytesinc]

What is wrong with sequentially applying File Append Transactions and waiting for a receipt before applying the next one?

[rbair23]

It is slow.

[bugbytesinc]

Of course its slower, but it is easy and rock solid. What you're proposing is quite a bit of complexity for saving how many seconds for a client? Not to mention the back end complexity and drag on mirror node development?

[steven-sheehy]

It's not just slower, there's no indication that the file is complete. In theory, an append could be submitted years later to an existing file to complete it. In particular, the mirror node needs the concept of atomic swap so it knows when system files like the address book or the fee schedule are ready to replace the existing files that it serves to users via its APIs.

Though how to accomplish that could be done many ways hence why we're discussing the options here. Besides segment counts, rolling hash, and atomic move transactions we should consider another option. The simplest option could be just to add a boolean last/complete flag to the file create/update/append transactions so downstream clients can react accordingly.

[rbair23]

What if we allowed for out-of-order assembly of a file. Suppose that each chunk had a byte offset associated with it. The server could get chunks in any order, and assemble them on the fly. Since the files have an upper limit (I think it is 1MB, I would need to check), it should be OK.

When a chunk comes in, we read the file data, and we apply the chunk. If the chunk start is < file size, then we just apply the chunk, and overwrite anything that is there. If the chunk start is > file size, then we fill with 0's until we get to the right point, and then insert the chunk.

By allowing chunks to come in out of order we should get maximum performance, because the sender could break the file up and just start submitting transactions to multiple nodes.

I was thinking of yet another solution, where we actually store a file's chunks as unique entries in the merkle tree, and combine them only on queries. But that introduces a lot of complexity if I have two existing chunks and a new one is uploaded that overlaps with both. So probably we don't want to attempt that trick unless we have to.

[netopyr]

Before diving too deep into the specifications, I think we need to agree on the requirements. I made a first draft of the motivation and user stories:

Abstract

The current file upload functionality is slow and error-prone. This HIP aims to improve it while preserving the existing behavior for backward compatibility.

Motivation

Transactions have a maximum size (at the time of this writing, 6KB). If a user wants to upload larger files, they must be split into smaller packages, which are sent with a CreateFile or an UpdateFile transaction, followed by AppendFile transactions. The current approach has several shortcomings:

• It is slow. The packages have to be sent sequentially. After sending a transaction, one has to wait for the receipt.
• The uploaded content is not validated automatically (e.g., with a provided checksum).
• While a file upload is happening, the file content is in an inconsistent state.
• If overriding a file fails, the old content is lost. Once a file upload has started, it cannot be canceled.
• It is not possible to signal the end of a file. This imposes a challenge to the network and other nodes, which may have to trigger functionality after a file is completely uploaded.
• If a single append fails, the upload must be started again from the beginning

This HIP defines additions to the current API that make file uploads faster and less error-prone.

User stories

• As a user, I want to use the existing functionality unchanged so that I am not forced to modify applications that use the current API.
• As a user, I want a simple form of validation that the stored content matches what I sent to be sure it was not corrupted during transmission.
• As a user, I want to upload file splits in parallel and be able to resend failed parts out-of-order so that I can upload large files faster.
• As a user, I always want to receive a complete file when reading its content so that I do not have to deal with corrupt, incomplete files while an upload is in progress.
• As a node operator, I want to be compensated for partially uploaded files to cover additional storage costs.
• As a user, I want to be able to cancel an already-started upload so that I am not charged recurring costs for partially uploaded files.

Did I miss anything?

[rbair23]

A consequence of "slow" is that it is also unreliable. Each "chunk" you upload has to be aimed at a specific 3 minute window. Otherwise, you have to try again from the beginning, or get all expired chunks re-signed and re-sent. For cases like council uploads where getting chunks resigned is not automated, this is extremely painful.

[david-bakin-sl]

I'd like to see an API that took the file offset and total file size with the current chunk size + data for uploading a chunk. Then first one through consensus could allocate the file and all could write at the proper offset into the file and you could submit all chunks simultaneously and wait for all to complete - which could be in the very same block.

We could also fix the chunk size, not let it be arbitrary (except for the last chunk). This would help with the "rolling hash" issue if the hash was "segmented" by chunk.

When people are uploading a file they know everything about the file in advance - it's not like they're generating it on the fly one sequential chunk at a time. So file size is known in advance.

[david-bakin-sl]

Also @netopyr - I'd add somehow to the user stories. that the user knows the entire file in advance before beginning the upload. Specifying that enables more possibilities - considering a "streaming" source file means you can't have solutions that upload chunks out of order.