Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

corto_select should return objects from store when MOUNT_QUERY is not specified #628

Open
hendrenja opened this issue Nov 3, 2017 · 0 comments

Comments

@hendrenja
Copy link
Member

hendrenja commented Nov 3, 2017

When a mount specifies OWNER_LOCAL ownership, objects in the object store that are owned by that mount are ignored by corto_select. To ensure that corto_select does not return duplicate results and provides the full dataset, it will query the mount and use the resulting data instead.

This ensures that corto_select does not have to check for duplicate values- which would increase space complexity to O(n) (instead of the current O(1)) and algorithmic complexity to O(log n) (instead of the current O(1)).

However, when a mount does not implement onQuery, corto_select will take objects from the object store, as in this case the mount must be storing objects directly in the store. This behavior should be extended with also falling back on the object store when CORTO_MOUNT_QUERY is not provided in the mount mask (indicating that the mount onQuery method should be deactivated).

Currently, when not specifying CORTO_MOUNT_QUERY in combination with a CORTO_OWNER_LOCAL ownership policy, corto_select will not invoke onQuery and also ignore data from the object store, which results in returning an empty set.

This issue can be reproduced by configuring a mount (with onQuery or onHistoricalQuery implementations) to a mount point such as /data/weather. Assume there is data in the object store along the data/weather/x/y/z path. If the mount is configured without MOUNT_QUERY or MOUNT_HISTORICAL_QUERY masks, then Corto will not return data. Instead, the data from the object store should be returned.

@SanderMertens SanderMertens changed the title corto_select return object store data when mount query mask not specified corto_select should return objects from store when MOUNT_QUERY is not specified Nov 4, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants