New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Fix] Full update of data_dir in DatasetInfo with update_data_dir() #5297
base: master
Are you sure you want to change the base?
Conversation
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Thanks @youliangtan for your interest in TFDS! Do you have an example when the update is incomplete causing issues? We use this function only once in: datasets/tensorflow_datasets/core/dataset_builder.py Lines 702 to 703 in 47f7a2e
data_dir in splits, but the identity should already have the right data_dir after dataset builder was initialized: datasets/tensorflow_datasets/core/dataset_builder.py Lines 281 to 284 in 47f7a2e
|
Thanks for the reply, I'm using this method to update a new data_dir, after The code example is here. This script is mainly to merge or reshard the rlds data, maybe there's better way of doing this too. 🤷♂️ |
I think For your code I suggest to consider these options :
|
Thanks for the suggestion. I think I will keep the current code as light as it is for now without depending on external lib. Also, could the |
I agree, that would be a nice change. |
Have you looked into builder_from_directories? (see
|
Description
The previous
DatasetInfo.update_data_dir()
only updates the datadir in each split, but not its DatasetIdentity, making the update incomplete. The datadir property doesn't have a setter too, thus, this adds a single liner to update the identity when the update_data_dir() is called.