Please move away from Berkeley DB

[mgorny]

Berkeley DB is deprecated since Fedora 33 and will eventually be removed. Gentoo is also following suit. Please consider switching to another database backend.

[Blinker73]

Please do. I would hate to loose native installation support of my favorite player inside my favorite distro

[virtuald]

Which database would you propose that we move to?

[Blinker73]

I googled a bit and without knowing anything about how berkeley DB is used, i found this site that proposes alternatives
https://alternativeto.net/software/berkeley-db/
I found that leveldb and qdbm are both present in Gentoo's package manager

[virtuald]

That's not a very compelling argument.

• qdbm looks like it hasn't been updated since 2013? Why not just stay with berkeley db then?
• To use leveldb there needs to be a python interface. Looks like https://plyvel.readthedocs.io/en/latest/ is staying current, but most other interfaces are really old

[mgorny]

Which database would you propose that we move to?

I think the closest option to Berkeley DB is GDBM. It's supported by Python out-of-the-box via dbm.gnu module (except on Windows).

The more modern and portable option would be sqlite3, also supported by Python out-of-the-box. You could also use SQLite3 via sqlalchemy, if you prefer an ORM API.

(Note that by "out of the box", I mean in default Python build. Technically you could build Python without them but I think that's rather rare and breaks other packages. On some distros the Python package may be split and the relevant extensions be installed separately.)

[sjohannes]

I've looked into some of these simple key-value databases before.

• Berkeley DB. What we use now. Works everywhere. Most OSes build the bindings against version 5.3 (before the switch to AGPL), which AFAIK doesn't have any breaking issues. Arch doesn't do this but they build against 6.2, which was already several years outdated at the time—​I don't know if there was a particular reason.

• gdbm & ndbm. Bindings maintained by the Python project. Doesn't work on Windows. MSYS2 used to include it (not anymore) and during Exaile 4.0 beta testing we found out that it was broken there as well. This was the point where we decided to exclusively use bsddb/berkeleydb.

• Python's dumbdb. Works everywhere. "Intended as a last resort fallback ... not written for speed ... not nearly as heavily used ..." basically not very encouraging words from the developers themselves.

• LMDB. Bindings used to be an issue but now seems widely available (checked on Debian, Fedora, Arch, and MSYS2). If I remember correctly, LMDB requires manual garbage collection (by copying the data to a new database) because it doesn't reuse space from removed items.

• LevelDB. Bindings packaged on Debian but not Fedora or MSYS2, as far as I can find.

• SQLite. Well, it's not very simple and not a k-v database, but we could create a simple wrapper that just stores everything in one table with two columns. Bindings maintained by Python project and available everywhere.

For the record, the latest Berkeley DB version (18.1) is released under GNU AGPL 3.0, which is compatible with the license of Exaile's codebase. As far as I know, it's also compatible with the licenses of our dependencies (Mutagen was GPL 2.0-only but they switched to GPL 2.0-or-later). It would make the whole Exaile+dependencies distribution AGPL 3.0 but I'm not aware of any legal issues.

I think ideally someone with clout in Linux distro circles should fork Berkeley DB 5.3 and declare the project complete. Then projects can simply link to that new library and be done with all this bikeshedding.

Otherwise, if Fedora does decide to remove Berkeley DB without an alternative (have they decided on this, or is it still just a proposal?), I think realistically the best way forward for us is to use SQLite with that stupid k-v wrapper I mentioned.

[mgorny]

I think ideally someone with clout in Linux distro circles should fork Berkeley DB 5.3 and declare the project complete.

It's not "complete", it's dead. It already requires patching to build at all and code rot will increase.

(have they decided on this, or is it still just a proposal?)

From what I understand, it's been decided but they didn't decide when it's going to happen.

[virtuald]

I think realistically the best way forward for us is to use SQLite with that stupid k-v wrapper I mentioned

I agree.