Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow sharding of key space on a single machine #26

Open
adamfowleruk opened this issue Oct 11, 2020 · 0 comments
Open

Allow sharding of key space on a single machine #26

adamfowleruk opened this issue Oct 11, 2020 · 0 comments
Labels
enhancement New feature or request

Comments

@adamfowleruk
Copy link
Owner

Please start with a use case description for a USER of this work
[Who] As a database admin
[What] I need to use all of the processing power of my DB server when storing/retrieving keys and performing queries
[Value] In order to achieve very high throughput on sets, gets, queries

Describe the solution you'd like
Wrapper ShardingKeyValueStore(KeyValueStore[] shards) that only passes on sets and gets for keys to the correct sharded KV store. It should also map/reduce the queries passed to it

Describe alternatives you've considered
Consider letting the Database class handle this, but it's more of a storage detail, and so belongs in a KeyValueStore implementation. The suggested approach also makes it easier to go from an unsharded store to a sharded one, and doesn't impose knowledge on sharding on underlying KV store implementations

Additional context
Requirements:-

  • The deduction step to go from std::size_t hash -> KV index must be blindingly fast.
  • Use C++ threads with incoming/outgoing work queues if possible (will make doing this later over the network easier)
  • Fixed number of shards for now
  • No support for resharding for now

Not requirements:-

  • Not multi-machine for now - still single server/embedded
@adamfowleruk adamfowleruk added the enhancement New feature or request label Oct 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant