Filter nested properties based on metadata #94
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Update filter_data_by_metadata function to allow filtering of nested fields - e.g. if property `address` has selected set to True, but property `address.street` has selected set to False, only the street would be excluded. Processes data recursively.
make formatting a little clearer
Fix array type breadcrumb name
breadcrumb path documentation
change based on tests - must remove field from data object, not just set value to None.
line lenght :)
|
There are tests in for the Transform code. Do you think you could write a few to show this works as expected? https://github.com/singer-io/singer-python/blob/master/tests/test_transform.py#L263 |
|
Yes - will try to get to this next week. |
|
Hi Chris, would this also work to filter the properties of nested arrays of objects? |
|
@dmosorast @cmerrick @KAllan357 If I write some tests for this could this be merged? I need this functionality for |
|
@dmosorast @cmerrick @KAllan357 |
|
Worth closing this now that #130 is merged? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Not sure if this will be too big a change, but I wanted to be able to filter out nested properties based on metadata rather than just top level properties.
So if I have a field
billing_addressof typeobjectwith the propertiesstreet,state,zip, andcountry, I could haveselected: Trueforbilling_address, butbilling_address.streetcould beselected: False.The result would be the three other fields (
state,zipandcountry) would be synced.All tests pass - it shouldn't break any existing use cases because it will only filter out nested fields that are explicitly unselected or unsupported.