Dynamic inner joins can create excessive number of requests

[jeswr]

Issue type:

As shown here performing dynamic inner joins can result in orders of magnitude more requests to a TPF endpoint than required to select the entire dataset. This can cause major performance degradations as network latency is the main bottleneck in such situations.

We need to find a way to make sure that cases such as those in the example repo perform the join by just doing an inner join on the patterns ?s ex:worksFor ?o1 and ?o1 ex:name ?o; rather than doing a dynamic inner join.

• 🐌 Performance issue

---

Description:

---

Environment:

[github-actions]

Thanks for reporting!

[rubensworks]

Looks similar to #548.

Note that this is mainly a research problem (for which some solutions exist already), not really an implementation problem.

Adding it to the list in #846

[jeswr]

Note that this is mainly a research problem (for which some solutions exist already), not really an implementation problem.

Yup - my 2c on what seems to be a key missing heuristic is the estimated time required to retrieve all the data for a quad pattern (which would be Math.ceil(cardinality / pagesize)); and for dynamic joins estimating the time required to do the join in terms of number of request required (which would be (cardinality of first stream) * Math.ceil( (approx cardinality of each stream requested as part of the join) / (pageSize) )).

Is there a way of achieving this as part of the addition of Dataset cardinality work in #1194?