New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Filtering attributes/tags/access tier when transfering blobs between storage accounts #2621
Comments
Hi @catalin-micu, filtering blobs using tags or access tier is currently not supported. You can use one of the below ways for filtering blobs during copy.
|
All my data is |
Do you mean changing the blob type from Block Blob to Append Blob or Page Blob? If yes, then there is no direct way to do that. |
Yes, I meant changing the blob type, I understand it's not possible. Is there anything else worth trying? I was leaning towards filtering based on blob tags or blob attributes, but it does not seem possible |
What blob attribute do you want to use for filtering the wrong data (other than tags or access tier)? |
I don't have any in mind, basically anything that I can set to a specific value for all the wrong data, then pass said value to azcopy to filter |
Alright, I see a |
Blob Inventory (https://learn.microsoft.com/azure/storage/blobs/blob-inventory) captures metadata/attributes on objects like Access Tier. You could use a Blob Inventory report as a input to AzCopy with the |
Interesting solution, but sadly, it won't work because of the performance issues. The resulting list of files would have millions of entries, every time, for multiple transer jobs I will do (200+) |
AzCopy 10.23
Linux OS
azcopy copy "source_storage_account_container" "destination_storage_account_container" --recursive
Problem: Copying entire storage containers and using azcopy to filter some blobs
There is an unpredictible amount of data, scattered throughout the container, that we want to filter out. We are talking about petabytes worth of data in total. We can identifiy all the data that needs to be filtered. Due to internal policies, we
cannot
alter the data (cannot rename/add prefix or anything of the sort , thereforecannot
use--exclude-pattern
or--exclude-regex
), nor can we archive it. These two options are out of the question.What I want to do is filter data in a
storage account to storage account transfer
,through azcopy copy
, based on either a tag, or access tier (everything is currently hot tier, but unwanted data can be moved to cool or cold) or any other blob attribute that can be assigned to the data, without changing names, directory structure or archiving.Can this be done?
The text was updated successfully, but these errors were encountered: