Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AAE-22075 Fix "detaching an uninitialized collection" logs and improve task search query performance #1441

Conversation

tom-dal
Copy link
Contributor

@tom-dal tom-dal commented May 14, 2024

Issue link: AAE-22075

The cause for the task search being slow when process variable have to be fetched is that Hibernate is performing N+1 queries: one query to search the tasks and N queries to fetch process variables for each task. (and also N queries to fetch candidate users and candidate groups).

A first attempt to solve the problem has been made just applying EntityGraph. Using an EntityGraph we are able to specify which field to retrieve eagerly, but the drawback is that, being the table join a cartesian product, Hibernate is not able anymore to handle pagination and sorting in the query. This way, pagination and sorting are performed in-memory which cause OOM error when the task list is huge. Additionally, even for smaller task lists that still exceed the page size, overall performance is not improved because of the large amount of data to retrieve from database.
https://stackoverflow.com/questions/75637639/spring-data-3-fetch-join-with-paging-and-specifications-is-not-working-properly

The proposed solution follows a similar approach to the one already leveraged in this other repository method

It consists of performing 3 queries:

  1. A first query to count all the tasks (needed for the paginated response)
  2. A second query to retrieve only the Id list of the filtered tasks, with offset and limit according to paging parameters (and sorting if present)
  3. A third query which retrieves the tasks by the Id list joining process variables, candidate users and candidate groups.

Additionally, the user Spring @transactional annotation ,instead of Jarkarta one, with readOnly=true attribute, should improve performance by skipping dirty-checking on entities, since DB flush is not needed.
https://medium.com/@jkha7371/is-transactional-readonly-true-a-silver-bullet-1dbf130c97f8
https://vladmihalcea.com/spring-read-only-transaction-hibernate-optimization/

Outcome of simple performance tests performed via Postman:

Screenshot 2024-05-15 at 14 44 44

Jmeter tests are underway.

AkhileshPamidimarthi and others added 24 commits April 29, 2024 17:11
…azy_load_no_trans=true", if set to false, exception is raised)
* AAE-21097 add permission getter on Jwt adapter, implement conversion to granted authority and include permissions in auth flow

* AAE-21097 add AuthorizationConfigurerIT, fix omitted methods logic

* AAE-21097 fix tests

* AAE-21097 add license header

* AAE-21097 remove unused import

* AAE-21097 CustomAuthorizationManager implemented with different authority types for role and permission

* AAE-21097 move CustomAuthorizationManager

* AAE-21097 license header
* AAE-22123 added key check in KeycloakResourceJwtAdapter

* AAE-22123 change test
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.23.1 to 3.25.3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@0b21cf2...d39d31e)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* AAE-22123 fix omitted methods management

* AAE-22123 fix omitted methods management and tests
…alized_collection_with_enabled_filters_from_a_session
@tom-dal tom-dal marked this pull request as ready for review May 14, 2024 14:23
@tom-dal tom-dal requested a review from jesty May 14, 2024 14:26
@tom-dal tom-dal requested review from gicappa and igdianov May 15, 2024 09:21
@tom-dal tom-dal marked this pull request as draft May 15, 2024 11:32
@tom-dal tom-dal marked this pull request as draft May 15, 2024 11:32
Copy link

sonarcloud bot commented May 15, 2024

Quality Gate Passed Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

@tom-dal tom-dal marked this pull request as ready for review May 15, 2024 12:52
Copy link
Contributor

@igdianov igdianov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work!

@tom-dal tom-dal merged commit 70b36ec into develop May 16, 2024
17 checks passed
@tom-dal tom-dal deleted the fix/AAE-22075-fixing_Detaching_an_uninitialized_collection_with_enabled_filters_from_a_session branch May 16, 2024 07:34
jesty added a commit that referenced this pull request May 17, 2024
…e task search query performance (#1441)

* AAE-22075 Wip initial commit

* WIP

* AAE-22075 Wip fixed failing test

* AAE-22075 reproduce error. ("spring.jpa.properties.hibernate.enable_lazy_load_no_trans=true", if set to false, exception is raised)

* AAE-22075 reproduce exact situation of issue

* AAE-22075 tentative with DSL filters

* Revert "AAE-22075 tentative with DSL filters"

This reverts commit 6a148ac.

* AAE-22075 test with entity graph

* AAE-18428 Fix Swagger API suggest to use the HAL-JSON format (#1306)

* AAE-21097 Implement permission-based authorization (#1427)

* AAE-21097 add permission getter on Jwt adapter, implement conversion to granted authority and include permissions in auth flow

* AAE-21097 add AuthorizationConfigurerIT, fix omitted methods logic

* AAE-21097 fix tests

* AAE-21097 add license header

* AAE-21097 remove unused import

* AAE-21097 CustomAuthorizationManager implemented with different authority types for role and permission

* AAE-21097 move CustomAuthorizationManager

* AAE-21097 license header

* AAE-22123 Fix NPE when getting permissions (#1431)

* AAE-22123 added key check in KeycloakResourceJwtAdapter

* AAE-22123 change test

* chore(deps): upgrade Activiti/Activiti to version 8.5.0-alpha.2 (#1432)

from: https://github.com/Activiti/Activiti

Co-authored-by: alfresco-build <>

* build(deps): bump github/codeql-action from 3.23.1 to 3.25.3 (#1429)

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.23.1 to 3.25.3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@0b21cf2...d39d31e)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* AAE-22123 Fix omitted methods (#1433)

* AAE-22123 fix omitted methods management

* AAE-22123 fix omitted methods management and tests

* AAE-22075 perform single queries on join table after select on task table

* Revert "AAE-22075 perform single queries on join table after select on task table"

This reverts commit 3e35763.

* AAE-22075 use jpaquery and fluentyquery

* AAE-22075 reduced number of queries, applied sorting parameters

* AAE-22075 fix sorting

* AAE-22075 fix sorting

* AAE-22075 fix

* AAE-22075 remove unused imports

* AAE-22075 remove sql logs from test

* AAE-22075 fix import

* AAE-22075 added tests

* AAE-22075 refactoring

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Akhilesh Pamidimarthi <akhilesh.pamidimarthi@hyland.com>
Co-authored-by: jesty <davidecerbo@gmail.com>
Co-authored-by: RahulKumar1606 <126051097+RahulKumar1606@users.noreply.github.com>
Co-authored-by: Alfresco Build <8039454+alfresco-build@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
erdemedeiros pushed a commit that referenced this pull request May 17, 2024
…e task search query performance (#1441) (#1443)

Please refer to #1441
RCorsaletti pushed a commit that referenced this pull request May 22, 2024
…e task search query performance (#1441)

* AAE-22075 Wip initial commit

* WIP

* AAE-22075 Wip fixed failing test

* AAE-22075 reproduce error. ("spring.jpa.properties.hibernate.enable_lazy_load_no_trans=true", if set to false, exception is raised)

* AAE-22075 reproduce exact situation of issue

* AAE-22075 tentative with DSL filters

* Revert "AAE-22075 tentative with DSL filters"

This reverts commit 6a148ac.

* AAE-22075 test with entity graph

* AAE-18428 Fix Swagger API suggest to use the HAL-JSON format (#1306)

* AAE-21097 Implement permission-based authorization (#1427)

* AAE-21097 add permission getter on Jwt adapter, implement conversion to granted authority and include permissions in auth flow

* AAE-21097 add AuthorizationConfigurerIT, fix omitted methods logic

* AAE-21097 fix tests

* AAE-21097 add license header

* AAE-21097 remove unused import

* AAE-21097 CustomAuthorizationManager implemented with different authority types for role and permission

* AAE-21097 move CustomAuthorizationManager

* AAE-21097 license header

* AAE-22123 Fix NPE when getting permissions (#1431)

* AAE-22123 added key check in KeycloakResourceJwtAdapter

* AAE-22123 change test

* chore(deps): upgrade Activiti/Activiti to version 8.5.0-alpha.2 (#1432)

from: https://github.com/Activiti/Activiti

Co-authored-by: alfresco-build <>

* build(deps): bump github/codeql-action from 3.23.1 to 3.25.3 (#1429)

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.23.1 to 3.25.3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@0b21cf2...d39d31e)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* AAE-22123 Fix omitted methods (#1433)

* AAE-22123 fix omitted methods management

* AAE-22123 fix omitted methods management and tests

* AAE-22075 perform single queries on join table after select on task table

* Revert "AAE-22075 perform single queries on join table after select on task table"

This reverts commit 3e35763.

* AAE-22075 use jpaquery and fluentyquery

* AAE-22075 reduced number of queries, applied sorting parameters

* AAE-22075 fix sorting

* AAE-22075 fix sorting

* AAE-22075 fix

* AAE-22075 remove unused imports

* AAE-22075 remove sql logs from test

* AAE-22075 fix import

* AAE-22075 added tests

* AAE-22075 refactoring

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Akhilesh Pamidimarthi <akhilesh.pamidimarthi@hyland.com>
Co-authored-by: jesty <davidecerbo@gmail.com>
Co-authored-by: RahulKumar1606 <126051097+RahulKumar1606@users.noreply.github.com>
Co-authored-by: Alfresco Build <8039454+alfresco-build@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
6 participants