Add flag to skip sync/copy 0-byte files (e.g. --min-size)

[mweibel]

Which version of the AzCopy was used?

10.22.1

Which platform are you using? (ex: Windows, Mac, Linux)

Linux

What command did you run?

azcopy sync / azcopy copy

What problem was encountered?

Application writing the files writes first 0 byte files first before writing the actual file. The application is written by a third party and not under our control.

How can we reproduce the problem in the simplest way?

create a 0 byte file and sync/copy it using azcopy.

Have you found a mitigation/solution?

It would be great if azcopy supported a way to skip 0 byte files using e.g. a new flag --min-size or similar. Both for copy and sync.

[siminsavani-msft]

Hi @mweibel ! Thank you for sending a feature request, we will keep you posted if we choose to pick this up. In the meantime, I would like to suggest a few workarounds that may be helpful for your scenario. Since you want to exclude specific files, you may be able to leverage the --exclude-* flags we have, seen in this copy documentation and sync documentation.

In particular, I believe these flags will be the most useful:
--exclude-path (string) Exclude these paths when copying. This option doesn't support wildcard characters (*). Checks relative path prefix(For example: myFolder;myFolder/subDirName/file.pdf). When used in combination with account traversal, paths don't include the container name.

--exclude-pattern (string) Exclude these files when copying. This option supports wildcard characters (*)

--exclude-regex (string) Exclude all the relative path of the files that align with regular expressions. Separate regular expressions with ';'.

Let me know if you have any other questions!

[mweibel]

hi @siminsavani-msft. Thank you for your reply.

To clarify a bit more our use case:

• the files are generated by a third party software we don't control
• what files are generated we don't know beforehand (i.e. we don't know names etc)
• the 0 byte files get removed after a while and then at some point when the task is done, the real file gets written

Therefore the exclude path/pattern/regex flags don't work in this case.