Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mixin method to restrict table by keys of upstream tables #930

Closed
wants to merge 1 commit into from

Conversation

samuelbray32
Copy link
Collaborator

Description

Resolves #923

  • add method restrict_from_upstream to mixin class
  • prevents need for user to know all joins necessary to link a key to a deep table entry

Example use:
spikesortingv1 pipline does several key reductions with uuid's. This operation let's you effectively search up through table layers to apply restriction.

from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput
table = sgs.CurationV1()
key = {"nwb_file_name":"BS2820231107_.nwb","sort_group_id":28}
result = table.restrict_from_upstream(key,max_recursion=5)
result

To Do before review

  • Address case of aliasing between parent and child table (e.g. merge_id -> pos_merge_id)
  • Make better stopping condition by tracking previously found restriction items up through recursion
  • Remove development print statements
  • Add examples in tutorial notebooks

Checklist:

  • This PR should be accompanied by a release: (yes/no/unsure)
  • (If release) I have updated the CITATION.cff
  • I have updated the CHANGELOG.md
  • I have added/edited docs/notebooks to reflect the changes

@samuelbray32 samuelbray32 self-assigned this Apr 13, 2024
@samuelbray32
Copy link
Collaborator Author

@CBroz1 whenever you have time could you look at this and give thoughts on how to track aliasing between tables? Or point me to where you solved it before in delete cases. No rush and thanks!

Copy link
Member

@CBroz1 CBroz1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple first pass comments. I can try running to play around with it later this week

return table


def check_complete_restrict(table, key):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dj has a builtin for this called 'assert_join_compatibility' - if adopted, it would need to be wrapped in a try/except and call it using the key as a QueryExpression, but I'm inclined to let them handle this kind of thing rather than add more for us to maintain.



# Utility Function
def safe_join(table_1, table_2):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the permissive join operator could help you here a @ b - they talk about it in the docstring here

return restrict_from_upstream(self, key, **kwargs)


def restrict_from_upstream(table, key, max_recursion=3):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking I could fold this into either dj_chains or dj_graph to make use of how restr are cascaded there. it might cutdown on the footprint for this feature

@edeno edeno added the infrastructure Unix, MySQL, etc. settings/issues impacting users label Apr 19, 2024
@edeno
Copy link
Collaborator

edeno commented May 6, 2024

@CBroz1 it is my understanding that this should be closed in favor of #949 ? @samuelbray32 are you good with that?

@samuelbray32
Copy link
Collaborator Author

@edeno, agreed. I think that one is more robust. Let's not delete the branch for a little as we merge in and test though

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure Unix, MySQL, etc. settings/issues impacting users
Projects
None yet
Development

Successfully merging this pull request may close these issues.

General-purporse key lookup tool(s) for buried primary keys
3 participants