Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve Global Search Implementation #343

Open
victorskl opened this issue Oct 26, 2021 · 2 comments · Fixed by #349
Open

Improve Global Search Implementation #343

victorskl opened this issue Oct 26, 2021 · 2 comments · Fixed by #349
Assignees
@victorskl
Labels
epic Epic feature! triage & trello Triage and migrate into respective Trello project

Comments

@victorskl
Copy link
Member

victorskl commented Oct 26, 2021 •
edited

Context:

  • Current impl is early exploration (before fully utilising DRF), hence, a bit unnecessary complex in both endpoint controller layer and model

Impact:

  • This should potentially deprecate the following modules:
    • search_query.py
    • s3_object_search.py
    • views.py
  • Drop & deprecate S3Object and LIMSRow association table
    • S3LIMS
    • Association is nothing but string matching problem (hence, to utilise MySQL regex query) at runtime which is fast and more optimal; instead of maintaining large association table (high cardinality, high maintenance, eventual slow update, etc)
    • Performance: S3 objects metadata ingest #143

Look & Feel:

  • Adapt "full-text search" technique as first class mechanism
    • i.e. always return some results by relevancy
    • use MySQL regex query string matching on S3 key or GDS file path; with metadata lookup as augmented decision factor
  • Adapt "advanced search" as secondary mechanism
    • i.e. deterministic database search; facilitate advanced UI and allow user to provide specific field(s) they would like to filter

Actions:

  • Trace code (both backend & frontend), analyse and understand how current "global search" work
  • Improve impl by making use of DRF (Django REST Framework) built-in search filter & MySQL regex query
@victorskl victorskl added the epic Epic feature! label Oct 26, 2021
@brainstorm brainstorm mentioned this issue Nov 3, 2021
Merged
@victorskl
Copy link
Member Author

PR #349 is just additional use case for Global Search. Keeping this opening.

@victorskl victorskl reopened this Nov 9, 2021
victorskl added a commit that referenced this issue Apr 21, 2022
@victorskl
Updated to address S3 event processing performance issue (#143)
a17678f 
* Deactivated association links between S3 and LIMSRow
  i.e. graceful first-pass for #343 refactor
* Further reduced Report data size ingest limit to 1MB
* Commented out X-Ray recorder so that it does not even
  ping to AWS instrumentation endpoint for init boot
  i.e. "radio silent" mode that'd save few (sub) milliseconds
@victorskl victorskl mentioned this issue Apr 21, 2022
Merged
victorskl added a commit that referenced this issue Apr 22, 2022
@victorskl
Reinstated S3 delete event processing (#143)
2f87dcf 
* Association table S3LIMS has FK constraint on older
  records. Hence, it can not be skipped related FK link
  until we carry out proper table drop with refactor #343
* Added Lambda concurrency limit set to 20 to tame db conn
This was referenced Apr 22, 2022
Merged
Closed
@victorskl victorskl added this to the Release 2.0 milestone Oct 10, 2022
@victorskl victorskl self-assigned this Oct 10, 2022
@victorskl victorskl mentioned this issue Oct 18, 2022
Open
@victorskl victorskl removed this from the Release 2.0 milestone Oct 31, 2022
@victorskl
Copy link
Member Author

Descope from release 2.0 due to other priority + this needs a bit more tinkering.

@victorskl victorskl added the triage & trello Triage and migrate into respective Trello project label May 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@victorskl victorskl

Labels
epic Epic feature! triage & trello Triage and migrate into respective Trello project
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Start adding source/type filters to look for FFPE/FF umccr/data-portal-apis
1 participant
@victorskl