New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Minio fails to mirror file after changes if file's size matches #3331
Comments
@xoxys can you share your |
@harshavardhana sure:
|
Hi, I also confirm I have the exact same issue. Using I also removed the file locally, create a new one with the same name and put some random content with the exact same size but the file wasn't uploaded. I'm running the latest versions of minIO ( As a "workaround", the only way is
|
Is there any updates on this please? This is actually incredibly easy to reproduce. mc mb minio/test
mkdir tmp
cd tmp
echo "this is some random content" > content.txt
mc mirror --json --overwrite --remove --preserve --md5 ./ minio/test/
# {
# "status": "success",
# "source": "/root/tmp/content.txt",
# "target": "minio/test/content.txt",
# "size": 28,
# "totalCount": 1,
# "totalSize": 28
# }
# {
# "status": "success",
# "total": 0,
# "transferred": 56,
# "speed": 2485.540810183332
# }
echo "this is SOME rand0m c0ntent" > content.txt
# Note the file size is the same but the content is different.
mc mirror --json --overwrite --remove --preserve --md5 ./ minio/test/
# {
# "status": "success",
# "total": 0,
# "transferred": 0,
# "speed": 0
# }
rm content.txt
mc mirror --json --overwrite --remove --preserve --md5 minio/test/ ./
# {
# "status": "success",
# "source": "minio/test/content.txt",
# "target": "content.txt",
# "size": 28,
# "totalCount": 1,
# "totalSize": 28
# }
# {
# "status": "success",
# "total": 0,
# "transferred": 56,
# "speed": 9440.131109935215
# }
cat content.txt # the file content is the original content from the first 'echo command'
# this is some random content |
Hi @aureq , |
Tested with the reproducer from @aureq:
|
Great! |
@ebozduman I understand there is logic that checks, if the "same file" already exists in the destination and then skips the file to speed up the entire process. In our usecase, it would be perfectly fine to always mirror all files without such optimization, as we rarely have 100% identical files. That is, I'd like to err on the safe side: be slightly less efficient, but no chance to "forget" a file. |
@jnweiger in this case you will wait a lot longer between syncs... Personally I dont want a global ignore flag or whatever... I would like to have a working AND efficient sync |
Another way may be compare last modified timestamps on the file. |
@ebozduman correct, single server setup here |
@ebozduman Same here, it's a single site setup as well. So single minio server running. |
@xoxys, @aureq, @jnweiger, @i0x71, @dpgarrick, As @harshavardhana explained in his comment in PR#3353, to answer my question: I asked:
@harshavardhana answered:
So, here is an example how you can use
I hope |
@ebozduman Thanks for the details and the example. What exactly is |
@ebozduman Thanks for the information. From the help on From a quick test But my use case is I want to (manually) periodically back up a directory onto minio by running the So is my manual backup use case not supported or is it only possible by continuously running Some more configurable options to force |
Since Another reason we've stopped checking You are welcome to open up an issue for missing information/documentation about |
Thank you for your input.
Actually, when Unfortunately manual backup/mirroring is not supported as far as mtime changes go. Yes, you can always daemonize a process in shell. |
That didn't happen when I tested it as follows:
Wait 5 seconds and then interrupt the
Wait 5 seconds and then interrupt the My versions
In that case, what mc commands should be used to reliably back up data to minio via an automated nightly cron job for example? I run both manual and automated backups but all are using mc mirror in that fashion and for various reasons I do not want to run |
@ebozduman From all the discussion here and there, I understand why the default behavior is what it is but would be nice if this issue could be documented in the help for Seems the only workaround at this stage is to use |
You are right. The only workaround at this point is to use There is a MinIO document on how to use |
I'm still unable to get it to work... That's what I've tried now: I've started the mirror with:
What I need is a way to periodically or continuously mirror from a central upload.example.com minio server to multiple webserver local directory... |
Maybe that's because I try to mirror from a remote minio server to local fs? Is the active-active/watch mirror only working from local fs to remote minio? In this case that's not a solution for me as well and I would also need to checkout rclone. |
Expected behavior
mc mirror should always mirror files with changes
Actual
As described in the original issue I'm still able to reproduce this behavior. Files with the same size will not be synced even if the content has changed. #2187
mc --version
mc version RELEASE.2020-07-11T05-18-52Z
System information
Ubuntu 18.08
The text was updated successfully, but these errors were encountered: