Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Indexer to Get all Documents via Athena #3

Open
rlingineni opened this issue Sep 30, 2018 · 0 comments
Open

Update Indexer to Get all Documents via Athena #3

rlingineni opened this issue Sep 30, 2018 · 0 comments
Labels
help wanted Extra attention is needed optimization Doing this might make search faster

Comments

@rlingineni
Copy link
Owner

AWS Athena can quickly return S3 details.

The main problem is that in order to query, it requires creating a db-table for a schema. It's still cheap nonetheless.

To implement, we would need to dynamically update the Athena Table upon each new index config added or update and set up a config in the Cloudformation template accordingly.

It would be interesting to see how much of an improvement this might make. A good starting point before implementation is to do a study on a manual bucket with a bunch of documents, and replace the listObjects call with Athena.

@rlingineni rlingineni added help wanted Extra attention is needed optimization Doing this might make search faster labels Sep 30, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed optimization Doing this might make search faster
Projects
None yet
Development

No branches or pull requests

1 participant