MapDB3 based collection factory should provide a queue and set implementation

[JervenBolleman]

Problem description

qEndpoint has an issue where a path query triggers an out of memory exception. See the makeQuey and makeSet calls in PathIteration.

For the MapDb3 implementation we should be able to add a queue that spills to disk.

Preferred solution

Implement a java.util.Queue wrapper for a mapdb3 QueuLong that should be able to work here.

Are you interested in contributing a solution yourself?

Yes

Alternatives you've considered

No response

Anything else?

Spilling to disk might make existing queries much slower. Perhaps take a much higher switch point from in memory to mapdb then the MemoryTillSizeXSet default.

[kenwenzel]

Do you think it makes sense to consider an LMDB-based persistent data structure?
I've added a PersistentSet to the LmdbStore module.

[JervenBolleman]

An lmdb based collection factory is still in my would like to do list.

…

On Tue, Feb 6, 2024, 17:38 kenwenzel ***@***.***> wrote: Do you think it makes sense to consider an LMDB-based persistent data structure? I've added a PersistentSet to the LmdbStore module. — Reply to this email directly, view it on GitHub <#4899 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHQYFNIXIMXUDSWRAXCXNDYSJMCHAVCNFSM6AAAAABC4GNCWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZQGI4TCNJSGY> . You are receiving this because you were assigned.Message ID: ***@***.***>

[JervenBolleman]

MapDb3 QueueLong is undocumented and does not work like I assumed. Have a simple wrapper around a map where head and tail are maintained as keys.