Roadmap for OrientDB v 4.0

[luigidellaquila]

Hi all,

As you know, OrientDB v 3.2 was released a few weeks ago as the final release of SAP Enterprise OrientDB as a commercial product.

Now it's time to take decisions about the evolution of OrientDB Community Edition, the Open Source project, and the Community will have a relevant impact on these decisions.

I'm opening this thread to discuss with you the scope of next release(s); I flagged it as v 4.0, but it could reasonably be a 3.3.

Please add your comments below with what you would like to see in next OrientDB roadmap, I'll start with my ideas.

Thanks!

Luigi

[edit]

Adding here a summary of all the ideas in the thread

• Support new JDK versions (17?)
• Indexes on embedded fields
• Remove legacy SQL layer and redirect the old (deprecated) APIs to the new one
• More stable Distributed, simultaneous startup of all nodes
• Atomic/transactional DDL operations
• Storage improvements/PhLogging
• Calculable properties
• Automatic inverse properties
• Better support for drivers (Python, PHP, .NET)
• In-place binary upgrade
• Dependency upgrades (eg. Lucene)
• Decouple client from server components (eg. exceptions propagation, need to have Lucene and Distrbuted jars on the client)

[luigidellaquila]

And here is my wishlist:

• go for a 4.0, so that we can do some proper cleanup of the codebase, also removing some deprecated features
• completely remove the old SQL executor, redirect the existing APIs to use the new SQL layer, so that backward compatibility is guaranteed and also unsupported drivers can benefit of its improved performance and stability
• indexes on embedded fields!!! I'm sure you all agree on this, it's something everybody is asking since the beginning. It's not there because it needs some important refactoring in the database schema
• make DDL operations (eg. schema changes) transactional
• keep improving the storage and the distributed layer, for stability and performance

This is just my take and it's very high level, it will need to be discussed and expanded of course.

[PhantomYdn]

@luigidellaquila , I'm fully agree with your list! Few other points:

• Support or jump to more modern JDK version. JDK 11+
• Support simultanius cluster startup and assembling. Right now it requires to have 1 node be up before others. And it's a little inconvinient.
• Consider Orienteer for more close integration. We are working on the same goal: make OrientDB more successful. For example it will be good to migrate such features as "calculable properties" and "automatic inverse properties" directly to OrientDB.

[luigidellaquila]

@PhantomYdn

Oh yes, +1000 for JDK upgrade (I completely forgot to write it, but it's one of my priorities as well). IMHO we could also target JDK 17, as it should be next LTS and it will be released in September, so most likely before our next release. I'd like to hear the opinion of the Community on this, as this could be a blocker for for someone who still has constraints with old JDKs.

Simultaneous cluster startup should be one of the good consequences of a refactoring of the distributed layer, so definitely a good point.

I also like calculable properties are something we can easily integrate. Inverse properties will need some more attention to avoid performance problems, but we can definitely consider them

Thanks!

Luigi

[PhantomYdn]

Almost forgot to mention: will be goot to have different approach for DDL at all. Right now every DDL invokes RAM flash and as a result: on machines with high amount of RAM - DDL on OrientDB executes incredibly slow. I see that you mention making DDL transactional - so hope this one can be covered as well.

[imanabu]

I have had significant set of struggles with drivers from Apache Gremlin and Python Native drivers in 3.x releases. I would like to see stable and "runs out of the box" drivers all tested and maintained and ready and be also updated where they are normally pulled, like pip, from for the dev ecosystems It is essential for a wide adoption of OrientDB. It has a lot going.

[luigidellaquila]

Hi @imanabu

Thank you very much for pointing it out, this is another point where I see possible changes in positive.
The Core team members are all Java developers, some of us know JS and Rust very well, we all had a chance to write some Python or PHP but that's definitely not our sweet spot. The drivers for these languages (together with .NET) were contributed by Community members, but lately they are not having much attention. Probably this is a good moment to have more involvement from new contributors and refresh the drivers with support to latest features.

Thanks!

Luigi

[timw]

For us, it's probably supportability, reliability and maintainability that are top of our wishlist:

• Replicated cluster startup and failure survivability
• Database backup and restore reliability - there seem to be lots of ongoing issues in this area, and we've had problems doing this reliably in our testing environments.
• Database engine upgrades (in-place or via backup/restore) - again seem to be a pain point
• Upgrading Lucene to something more recent
• Incremental backup - really nice to have. It's in the Enterprise product, but what is the product/IP status of that in an OrientDB 4.0 world?
• JDK17 support (seems like a logical LTS to land on, maybe keep 11 but drop 8 for a 4.0)
• Clean up module structure and dependencies so clients don't need to depend on components only used by server code (e.g. use of Lucene indexes today).

I think OrientDB is in a bit of a tenuous position in terms of use with the transition (it appears) to a full community driven model.
We're currently in a pre-release state for a commercial product, and whether we remain on OrientDB will depend far more on quality attributes, how stable the product is and how responsive the community is to fixing issues, than it will on adding new features.

On that note, less product things to add to the wishlist:

• A clear statement of the remaining (if any) commercial backing for OrientDB, and what the status of the Enterprise edition features and IP is. i.e. will any of the enterprise features make it into community now, and if not would pull requests adding any of those functions be accepted?
• Cleaning up GitHub issues (there are lots of issues that are just left open indefinitely) and maybe adding some lightweight process/bots to help keep these clean for users and contributors alike.
• Sorting out discussion forums, website and documentation, killing off the defunct ones and fixing the go-forward ones - at the moment it's really worrying how all of these are out of sync and unmaintained to various degrees, as it sends the wrong messages to prospective users about the health of the project.

[luigidellaquila]

Hi @timw

Thank you very much, a lot of important points here.

I'm adding some of the items you listed from the roadmap to the short list. I'll consider the others (eg. backup stability) as implicit requirements, ie. bug fixes.

Let me also comment on the "business" items

I cannot share specific data about the names and the number of customers SAP is currently supporting (reserved information, I guess you understand), but there is a statement saying there will be "continued support of existing SAP Enterprise OrientDB customers" without specifying any deadline.

About the PRs, I see no problems in accepting them, as long as they don't contain proprietary code. The overlap of the features IMHO is an issue we can address.
Will EE features make it into CE? That's a good question, but unfortunately I don't have a final answer. What I can say is that it's something I'd really like a lot and it's something I already asked for internally. This kind of decisions (ie. open-sourcing code) involve many aspects like legal and marketing, that a big company like SAP has to manage very carefully, so I can expect that, if it happens at all, it will take some time and some effort on formal procedures. If something changes and any decisions are taken on this, you'll know it immediately of course.

GitHub issues cleanup: +1000, we definitely need it. Any advice on additional tools we can use is very welcome

Web assets: yes, there is a lot of confusion now, the .com website is practically down, the .org website is not very solid, the forums are too sparse (github issues, github discussions, discourse, the Google group - yes, it's still there). IMHO it would be much better to move everything on a single platform; since the code is on GitHub, probably having everything here (Discussions for the community, GitHub Pages for a new, minimal and well organised website) is the best approach. Again, any advice/suggestion/contribution is welcome!

Thanks

Luigi

[mbtools]

Re: Web assets

As someone who occasionally stops by wondering whether to learn more about OrientDB and use it for some projects, the current state of websites and forums is very off-putting. If you want a flourishing and growing community, updating the .org website and consolidating as much as possible to GitHub should be prio1. And redirect .com to .org if SAP doesn't use it, please!

Put a notice up on all the other forums and shut them down end of 2021. Just do it.

There also seem to be several "support" channels: email, various chats, etc. I think GH will do (and maybe one chat option like Slack).

Re: Github

I know it's hard to keep up, but 1000+ open issues, 100+ branches, dozens of PRs (some from 2013?), do not encourage anyone to join the community. Rather than cleaning it up with great effort, I think it's better to start a new "orientdb4" repo.

Start with clean code. Use GH to the max. Sure, some with complain but if there's anything essential you can always link to the old repo or submit a PR to the new repo.

SAP ending EE is an opportunity. Use it for a fresh start.

Best wishes,
Marc

[mmacfadden]

One thing that I would really like to see is the Auto Sharding feature in distributed mode. There have been several previous discussions:

#6256
#8015
#8234

To me this is the final piece of making the distributed mode completely useful and in parity with other database engines out there that do this.

[luigidellaquila]

Hi @mmacfadden

Yes, I think (Auto)Sharding is the dream of every OrientDB user and it would be really cool to have it in next release, but I'd like to keep the expectations realistic.
We are not too far from that conceptually, but practically this thing involves so many aspects of the system that in my opinion (and from my experience with previous releases) it's unlikely to have it in next release. What we can do is probably to prepare the system, refactoring all the components one by one to prepare them for this feature, and then plan it for next release.

Thanks

Luigi

[laa]

I am closing this one to have a space to form an updated roadmap. Hopefully soon.