Project Roadmap: 2023-24

[ohsayan]

The roadmap

Note: This comment is frequently updated with progress from time to time

When we started out with Skytable — we didn't start with a small goal. We started out ambitious — to reduce the problems that developers face while modeling data. Over the last one year, we have learned many things and received a lot of feedback from the community, and now it is time that we use all those learnings and put them to paper (or to code xD):

What we put together

• Stable key/value store
• Simple configuration (environment variables, configuration file or command-line arguments)
• Stable protocol
• TCP/TLS connections (with multi-socket support)
• Snapshots for seamless backups
• Multiple keyspaces
• Multiple tables
• Hardcoded implementation of data models
• Storage engine for multiple keyspace/tables
• Volatile tables for caching
• Protocol level support for data types
  • Binary data type
  • UTF-8 string data type
• Key/value store with native support for multiple data types
• Query level support for table properties
• DDL queries
  • INSPECT to inspect containers
  • CREATE/DROP for creating keyspaces/tables

What's next

• Migration tool (tracked in Tracking issue: Migration tool for versions > 0.6 #200)
• Dev/prod mode
• Complete rewrite of the memory engine core with support for expiring keys and newer models (tracked in Internal: Rewrite internal core structures #247)
• Real-time features
  • PUB/SUB
  • Advanced RT features
• Authentication
  • Token based
  • Admin port (admin actions can only be run on admin ports)
• Group querying:
  • Pipelines
  • Batches
• Simple and minimal query language — BlueQL (tracked in Implement, document and stabilize BlueQL #263)
• Data modeling features
• More data types
• LRU engine (going a few bits off memory)
• Replication (the specifics are to be shared later)
• Clustering (the specifics are to be shared later)
  • Architecture "whitepaper"
  • Clustering protocol
• Advanced query types (ACID transactions, ...)

Leave comments if you have any specific feedback for the ideas listed here. If you have any other ideas, it's a good idea to open a feature request or discuss it on our discord server.

Let's make it happen!

[MarkSwanson]

Nice to see the roadmap. Especially interested to see replication specifics!

[r3wt]

Hi, currently working on building a client for skytable in node.js. I'm curious if you had given any thought to mTLS as an authentication scheme?

[ohsayan]

Hi, currently working on building a client for skytable in node.js. I'm curious if you had given any thought to mTLS as an authentication scheme?

+1. Zero Trust authentication (or security) definitely sounds like one of the things we'd want to have for authentication, and simplicity too. I'm definitely keeping this noted (the auth feature's ETA is 0.7.2+).

[ohsayan]

Token-based authn has landed! One more box ticked :)

[firedupmike]

Really glad to see the roadmap! Have been following the project for a while now, and it's getting better everyday. Just to add here, are there any plans for adding PUB/SUB? That would help in the move away from Redis

[ohsayan]

Really glad to see the roadmap! Have been following the project for a while now, and it's getting better everyday. Just to add here, are there any plans for adding PUB/SUB? That would help in the move away from Redis

We've been asked for PUB/SUB several times, and I think it's time — yes, PUB/SUB is on the roadmap now.

[takkuumi]

Are there a skytable connection pool?

[ohsayan]

Are there a skytable connection pool?

Since pools are implemented at the library level, it depends on the client you are using. The official Rust client driver has connection pooling in the preview release track.

[ohsayan]

@NateLing Just to keep you updated, the Skytable Rust driver now supports connection pooling (out of preview).

[takkuumi]

@ohsayan It's great, i will try it

[forrestli74]

What's the reason to keep some of the roadmap secret? If it's because it's not mature enough, you could just say "tentative" or something.

[ohsayan]

What's the reason to keep some of the roadmap secret? If it's because it's not mature enough, you could just say "tentative" or something.

I've updated it. It was mostly kept as a surprise for the community, which is why.

[emchristiansen]

For replication, I'd love to see an option for simple peer discovery and synchronization via gossip with eventual consistency and some simple heuristics for conflict resolution.

AFAICT there's a market need for a dead-simple replicated, scalable, key-value store (etcd comes closest but it's surprisingly painful to dynamically add nodes or deal with node failures). The user should be able to add a new node to the network by just giving it an authentication token and pointing it at an existing node. If a node goes offline the other nodes should just update their peer lists and continue on their merry ways.

I've looked at MongoDB, Redis, etcd, Cassandra, ScyllaDB, TiKV, and several others, and none of them seem to fit the mold.

[ohsayan]

@emchristiansen Thank you for your note here. Different database implementations for C&R have been influenced by a multitude of decisions, and I'd avoid commenting on the same.

While we're utilizing existing "proven" schemes for logical partitioning and FT — hear me out: I do not necessarily find the ideas behind the implementations of some of these systems to resonate with mine. That is to say, while we're using existing research and ideas from current systems — we're taking our own approaches to many design decisions.

Another important influence is wrt the consistency model chosen and the tradeoffs offsetting from it. Skytable intends to chose AP over C, but we're also doing something here.

TL;DR — We're working on our own innovations in this space and will try to make the best tradeoffs in our design choices.

Please hang on with us — we're on it!

[insanpraja]

Hi, would a TiKV backend be possible for Skytable? that would made it instant clustering.

Great database, btw!

[zamazan4ik]

@ohsayan Could you please clarify the status of the project? As far as I see, Skytable didn't have commits for a long time. So if you decided to stop working on Skytable - could you please add a note about it somewhere to the README file?

[ohsayan]

@ohsayan Could you please clarify the status of the project? As far as I see, Skytable didn't have commits for a long time. So if you decided to stop working on Skytable - could you please add a note about it somewhere to the README file?

Hi @zamazan4ik! Skytable is 100% alive and is in active development. Skytable 0.8 is set to be released soon with several of the features in the roadmap. Development is being done in a private repo and will be merged into this repo once everything is ready :)

[ohsayan]

Opened #317 for the release. See you at the release blog post :))

[ohsayan]

Folks, Skytable Octave has been shipped this week. If you missed the announcement, here's a link to it: https://skytable.io/blog/2023/12/skytable-octave-is-here

I think it's time that we open a new roadmap issue! Next up clustering and replication 🚀

[ohsayan]

For everyone following this issue, Skytable 0.8 was released with a lot of these features including a new query language and a new storage engine. The work continues!