command/sync: add --include flag
#600
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ahmethakanbesel
force-pushed
the
cp-rm-sync-include-flag
branch
from
c21ee76
to
ab97b40
Compare
ahmethakanbesel
requested review from
igungor and
ilkinulas
and removed request for
a team
|
@ahmethakanbesel This feature deserves its own section in README with a few realistic examples. Could you write one?
|
igungor
reviewed
Jul 26, 2023
ilkinulas
reviewed
Jul 27, 2023
command/cp.go
Outdated
| @@ -777,6 +798,23 @@ func (c Copy) shouldOverride(ctx context.Context, srcurl *url.URL, dsturl *url.U | |||
| return stickyErr | |||
| } | |||
|
|
|||
| // shouldCopyObject checks is object should be copied. | |||
There was a problem hiding this comment.
Suggested change
| // shouldCopyObject checks is object should be copied. | |
| // shouldCopyObject determines whether or not the object should be copied. |
command/rm.go
Outdated
| @@ -195,6 +219,23 @@ func (d Delete) Run(ctx context.Context) error { | |||
| return multierror.Append(merrorResult, merrorObjects).ErrorOrNil() | |||
| } | |||
|
|
|||
| // shouldDeleteObject checks is object should be deleted. | |||
| func (d Delete) shouldDeleteObject(object *storage.Object, verbose bool, prefix string) bool { | |||
There was a problem hiding this comment.
This logic is duplicated in shouldCopyObject function. Can we extract this logic to a function and re-use it in both cp and rm commands?
There was a problem hiding this comment.
Yes, it's possible. f008722
This PR adds a new io.WriterAt adapter for non-seekable writers. It uses an internal linked list to order the incoming chunks. The implementation is independent from the download manager of aws-sdk-go, and because of that currently it can not bound the memory usage. In order to limit the memory usage, we would have had to write a custom manager other than the aws-sdk-go's implementation, which seemed unfeasible. The new implementation is about %25 percent faster than the older implementation for a 9.4 GB file with partSize=50MB and concurrency=20 parameters, with significantly higher memory usage, on average it uses 0.9 GB of memory and at most 2.1 GB is observed. Obviously, the memory usage and performance is dependent on the partSize-concurrency configuration and the link. Resolves peak#245 Co-authored-by: İbrahim Güngör <igungor@gmail.com>
|
can --include filter be added to ls command? |
@tooptoop4 could you please open an issue for this request? |
ilkinulas
reviewed
Aug 4, 2023
Co-authored-by: İlkin Balkanay <ilkinulas@gmail.com>
ilkinulas
previously approved these changes
Aug 4, 2023
sonmezonur
approved these changes
Aug 4, 2023
ilkinulas
approved these changes
Aug 7, 2023
|
Hi, curious when the next release is planned to go out? We'd like to start using the |
|
@markkdev Hey. Next release will be cut after all the issues are fixed in https://github.com/peak/s5cmd/milestone/9. |
|
@igungor Great, thanks for the response. Looking forward to it. |
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #516
Changes are made:
--includeflag to thecpcommand--includeflag to thermcommandAs
syncusescpandrmcommands in the background,--includecan also be used withsync.--excludehas precedence over the--includeflag.