[Overview] Improving performance

[rubensworks]

This issue contains an overview of all (known) performance issues in Comunica, and steps to resolve them.

Sub-optimal query plan

• Joins
  • How to make the “Ruben's topics” query fast? #548
  • Slow property path query for disjunctive form #725
  • Add (symmetric) hash join algorithm that can handle left joins #772
  • suboptimal triple-pattern query plan when using FILTER #834
  • Implement different join algorithms #552
  • Abstract BGP actor #427
  • Take into account system and network load during query planning #879
  • Improve performance of federation over heterogeneous sources #882
  • Dynamic inner joins can create excessive number of requests #1196

The following issues already have concrete solutions, and just need implementation:

• Implement sort-merge join #877
• Implement algebraic query optimization techniques #875
  • Push down query filters when possible #876
• Possibilities for tweaking join algorithms #884
• Implement existing query planners as multi-join algorithms #874
• Implement different join selectivity estimators #878
• Multiple FROM and FROM NAMED in virtual source #128
• Optimizing infinite path queries #709
• Mediate over minus actors, and add hash-based minus operator #719
• Implement OPT+ operator #825
• Improve default join actor configuration #538

The following are just ideas:

• Call optimize bus right before each Bind Join subquery? This could optimize join plans.
• Always use NLJ if no overlapping vars.
• Prefer SHJ over HJ
• Check contentlength in response headers to know how many indexes to create in rdf-store

Federation:

• In the directors query (see examples in http://query.linkeddatafragments.org/ ), we have 2 filters, but they are combined into 1 filter during query parsing. This makes filter pushdown more difficult. Should we decouple them so our filter pushdown optimizer can handle it?
• Look at FedShop RSA queries to see which optimizations can still be done.
• In new bind-based join actors: don't push down if there are no common variables. As seen in https://arxiv.org/pdf/2102.03269.pdf (3 conditions on page 8)

Memory issues over files

• comunica-sparql-file: slow queries and javascript heap out of memory #676
• Out-of-memory for filter not exists #835

Low-level optimizations

• Perform joins based on dictionary-encoded triples #873
• Investigate applicablility of WASM #756
• Improve performance of empty-patterns in federated actor #410
• Use Object.preventExtensions #499
• Federation with one source is slower than just one source #562
• Use graphy.js parsers #582
• Ensure backpressure is maintained in streams #759

Other

• Improve query explainer #880

[github-actions]

Thanks for reporting!