Queuing of queries (basic question)

[markdanese]

We are looking for a way to shard our data and process queries in parallel across shards. Our SQL is unpredictable and difficult to optimize (it comes from a query generator). ActorDB seems like an interesting option. I can't tell from the documentation whether or how it handles the situation where we have the data sharded into many more shards than there are processors/threads on the server. Is there a queuing mechanism? For example, with 16 cores, is there a way to iterate over all of the shards in groups of 16 at at time? Apologies if I am not understanding how it works and this should be obvious. Thanks in advance.

[SergejJurecko]

It would work if you do your own sharding manually across actors. However it sounds like you want something that would act entirely transparently on SQLs written for MySQL. That is not supported.

[markdanese]

We do all the sharding as part of our ETL process, so that is not an issue. Does that mean ActorDB should handle the queuing?

As for specific SQL syntax, we typically generate SQLite or PostGres SQL, but we could generate MySQL if we have to. Can you clarify?

I am not sure I follow what you mean by "act entirely transparently" -- if we do the sharding is there another issue you are thinking about?

[SergejJurecko]

No it is SQLite syntax.