feat: Chunk Operator #1583
feat: Chunk Operator #1583
Conversation
JasonPaulGithub
added
Enhancement
|
| file_size: u64, | ||
| data_map_chunk: Chunk, | ||
| chunk_vec: Vec<(XorName, PathBuf)>, |
There was a problem hiding this comment.
Why do we store the file_size, data_map_chunk and chunk_vec here? During new() they get default values and they're directly copied into ChunkOperator after chunking.
| pub struct ChunkOperator { | ||
| pub head_chunk_address: ChunkAddress, | ||
| pub data_map: Chunk, | ||
| pub file_size: u64, | ||
| pub chunk_vec: Vec<(XorName, PathBuf)>, | ||
| } |
There was a problem hiding this comment.
Is this the result of the BuildChunkOperator? If so, can we state that in the docs? I think the docs is misleading. Maybe ChunkOperatorResult would work here? We use the convention in our codebase.
|
|
||
| output_file | ||
| .write_all(&self.data_map_chunk.value) | ||
| .expect("error writing all"); |
There was a problem hiding this comment.
We might have to return proper errors here. And cannot use expect which can panic.
| .chunk_dir | ||
| .join(hex::encode(*self.data_map_chunk.name())); | ||
| let mut output_file = | ||
| File::create(data_map_path.clone()).expect("error creating output file"); |
There was a problem hiding this comment.
We cannot use expect here as it can panic. We should reutrn proper error types.
| chunks_paths | ||
| } | ||
|
|
||
| pub fn build(self, file_path: &Path, chunk_dir: &Path) -> ChunkOperator { |
There was a problem hiding this comment.
Looking at the code here, I think the rust builder pattern is not used here. We do not configure a lot of things and instead the 3 required args are passed via the build fn and with the use of two build_* fns. Why should we go with builder here instead of having something like ChunkOperator with execute(file_path, chunk_dir) / execute_with_data_map_in_chunks(file_path, chunk_dir)? I
Fix for Issue : #1487
I have created a Chunk Operator, aka Chop
Chop will then encrypt data based on the file size. As you will have three separate outcomes requiring the same generic operation, this is handled using basic polymorphism.
Chop will remove the need for the boolean flag 'data_map_in_chunks', and instead simply use either: build() or build_with_data_map_in_chunks()
Chop code is encapsulated inside its own .rs file and has been removed from the FilesApi code.
Chop makes use of the builder design pattern.