Add Env Var to disable WAL journaling

[gnattu]

Changes

SQLite WAL journaling is not compatible with all filesystems/VFS extensions, causing the database being locked up in certain conditions.

It's worth noting that database lockup is just one potential issue that can arise from compatibility issues with certain filesystems. In some cases, the use of WAL can even lead to the database being saved in a non-consistent state, resulting in database corruption.

Add an env var to let the user to choose the Journal mode for better compatibility.

For example, when the default WAL is causing problems:

export JELLYFIN_SQLITE__JOURNALMODE=TRUNCATE

Issues

[Bond-009]

Why not make the option a string so users can set any journalmode they want?

[gnattu]

Why not make the option a string so users can set any journalmode they want?

Because of too many potential user errors?

[joshuaboniface]

I agree with making it an option. On supported systems this can improve DB performance right? So it can be beneficial in those cases.

[joshuaboniface]

Also do we know exactly which filesysts have the issue? We can adjust packaging defaults if we know that.

[gnattu]

Also do we know exactly which filesysts have the issue? We can adjust packaging defaults if we know that.

You cannot predict this, the user can literally use any filesystem they want.

[cvium]

Will enabling pooling fix it?

[gnattu]

Will enabling pooling fix it?

I'm not very sure because I don't have the exact environment of the users. Probably this needs to be an extra option to let the user choose.

[memehammad]

Any easy way to try this out?

[gnattu]

Any easy way to try this out?

I compiled a tarball for the docker image: https://github.com/gnattu/jellyfin/releases/tag/env-disable-wal

Sorry but this is probably the easiest way I can prepare for you.

[jmqm]

Wouldn't it be better if TRUNCATE is the default value if WAL can be problematic on some filesystems?

[gitdeath]

Any easy way to try this out?

I compiled a tarball for the docker image: https://github.com/gnattu/jellyfin/releases/tag/env-disable-wal

Sorry but this is probably the easiest way I can prepare for you.

Attempted to test this with a known Database is Locked file system use case, but the below compose seems to create a database still using WAL - jellyfin.db-wal file is created in the config/data directory and PRAGMA journal_mode; returns wal

version: '3.8'
services:

  jellyfin-dev:
    image: jellyfin-disable-wal
    user: 0:100
    environment:
     - TZ=America/Chicago
     - JELLYFIN_SQLITE__JOURNALMODE=TRUNCATE

    networks:
     - default
    ports:
     - 8097:8096
     - 8927:8920
     - 1907:1900

    volumes:
     - jellyfin_dev_config:/config
     - jellyfin_dev_cache:/cache
     - media:/media:ro

host:~$ docker load < jellyfin-test-disable-wal.tar.xz
73242b419cec: Loading layer [==================================================>]  454.6MB/454.6MB
2d40ebc031c5: Loading layer [==================================================>]  295.6MB/295.6MB
5f70bf18a086: Loading layer [==================================================>]  1.024kB/1.024kB
87d78ce7befa: Loading layer [==================================================>]  2.048kB/2.048kB
f99821c77d4a: Loading layer [==================================================>]  155.1MB/155.1MB
0aa5721c5e74: Loading layer [==================================================>]  57.34MB/57.34MB
Loaded image ID: sha256:b7f31f02c71da06562117ab162a95cfa4ae28288a18f9ca2b4bb3644528f8d2d
host:~$ docker tag b7f31f02c71d jellyfin-disable-wal
host:~$ docker image ls
REPOSITORY                                     TAG       IMAGE ID       CREATED        SIZE
jellyfin-disable-wal                           latest    b7f31f02c71d   11 days ago    1.03GB

[memehammad]

That'd explain why this didn't fix my issue lol

[gitdeath]

That'd explain why this didn't fix my issue lol

I'm very interested to see if this does in fact improve the situation. Because from what I can tell, for use cases that work in 10.8, but not 10.9, the problem is the switch to MS.SQL from SQL.pretty - it seems MS has a performance hit compared to SQL.pretty on some FS setups that results in the database is locked issues (example NFSv3 storage with nolocks parameter - #11387.)

Now the MS.SQL change needs to happen, if the plan is to open up other database backends - but that isn't available yet, so I don't know if this was required to do before that.

If this WAL to TRUNCATE doesn't pan out, then I'll fork Jelly and reverse the changes from the below PR for a test.

#10138

[crobibero]

After testing I can confirm that Jellyfin does set the journal_mode correctly, but it is only available during the connection- not from the cli. I verified with the following patch:

Index: Emby.Server.Implementations/Data/BaseSqliteRepository.cs
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/Emby.Server.Implementations/Data/BaseSqliteRepository.cs b/Emby.Server.Implementations/Data/BaseSqliteRepository.cs
--- a/Emby.Server.Implementations/Data/BaseSqliteRepository.cs	(revision 8d8fcada18674becfcde6875d11e78a32d06c04d)
+++ b/Emby.Server.Implementations/Data/BaseSqliteRepository.cs	(date 1715531783505)
@@ -121,6 +121,12 @@
             if (!string.IsNullOrWhiteSpace(JournalMode))
             {
                 connection.Execute("PRAGMA journal_mode=" + JournalMode);
+
+                using var cmd = connection.CreateCommand();
+                cmd.CommandText = "PRAGMA journal_mode;";
+                using var reader = cmd.ExecuteReader();
+                reader.Read();
+                var currentJournalMode = reader.GetString(0);
             }
 
             if (JournalSizeLimit.HasValue)

[gitdeath]

After testing I can confirm that Jellyfin does set the journal_mode correctly, but it is only available during the connection- not from the cli. I verified with the following patch:

Index: Emby.Server.Implementations/Data/BaseSqliteRepository.cs
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/Emby.Server.Implementations/Data/BaseSqliteRepository.cs b/Emby.Server.Implementations/Data/BaseSqliteRepository.cs
--- a/Emby.Server.Implementations/Data/BaseSqliteRepository.cs	(revision 8d8fcada18674becfcde6875d11e78a32d06c04d)
+++ b/Emby.Server.Implementations/Data/BaseSqliteRepository.cs	(date 1715531783505)
@@ -121,6 +121,12 @@
             if (!string.IsNullOrWhiteSpace(JournalMode))
             {
                 connection.Execute("PRAGMA journal_mode=" + JournalMode);
+
+                using var cmd = connection.CreateCommand();
+                cmd.CommandText = "PRAGMA journal_mode;";
+                using var reader = cmd.ExecuteReader();
+                reader.Read();
+                var currentJournalMode = reader.GetString(0);
             }
 
             if (JournalSizeLimit.HasValue)

I've was unable to produce a state with this package (fresh install) that doesn't end with the creation of the WAL file. Is it possible Jellyfin is making multiple connections to the database? It would seem that if any connection sets to WAL then all will see WAL unless they specifically change for their connection.

https://www.sqlite.org/wal.html - Section 3.3

3.3. Persistence of WAL mode
Unlike the other journaling modes, [PRAGMA journal_mode=WAL](https://www.sqlite.org/pragma.html#pragma_journal_mode) is persistent. If a process sets WAL mode, then closes and reopens the database, the database will come back in WAL mode. In contrast, if a process sets (for example) PRAGMA journal_mode=TRUNCATE and then closes and reopens the database will come back up in the default rollback mode of DELETE rather than the previous TRUNCATE setting.

The persistence of WAL mode means that applications can be converted to using SQLite in WAL mode without making any changes to the application itself. One has merely to run "PRAGMA journal_mode=WAL;" on the database file(s) using the [command-line shell](https://www.sqlite.org/cli.html) or other utility, then restart the application.

The WAL journal mode will be set on all connections to the same database file if it is set on any one connection.

[gnattu]

After testing I can confirm that Jellyfin does set the journal_mode correctly, but it is only available during the connection- not from the cli. I verified with the following patch:

Index: Emby.Server.Implementations/Data/BaseSqliteRepository.cs
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/Emby.Server.Implementations/Data/BaseSqliteRepository.cs b/Emby.Server.Implementations/Data/BaseSqliteRepository.cs
--- a/Emby.Server.Implementations/Data/BaseSqliteRepository.cs	(revision 8d8fcada18674becfcde6875d11e78a32d06c04d)
+++ b/Emby.Server.Implementations/Data/BaseSqliteRepository.cs	(date 1715531783505)
@@ -121,6 +121,12 @@
             if (!string.IsNullOrWhiteSpace(JournalMode))
             {
                 connection.Execute("PRAGMA journal_mode=" + JournalMode);
+
+                using var cmd = connection.CreateCommand();
+                cmd.CommandText = "PRAGMA journal_mode;";
+                using var reader = cmd.ExecuteReader();
+                reader.Read();
+                var currentJournalMode = reader.GetString(0);
             }
 
             if (JournalSizeLimit.HasValue)

I've was unable to produce a state with this package (fresh install) that doesn't end with the creation of the WAL file. Is it possible Jellyfin is making multiple connections to the database? It would seem that if any connection sets to WAL then all will see WAL unless they specifically change for their connection.

https://www.sqlite.org/wal.html - Section 3.3

3.3. Persistence of WAL mode
Unlike the other journaling modes, [PRAGMA journal_mode=WAL](https://www.sqlite.org/pragma.html#pragma_journal_mode) is persistent. If a process sets WAL mode, then closes and reopens the database, the database will come back in WAL mode. In contrast, if a process sets (for example) PRAGMA journal_mode=TRUNCATE and then closes and reopens the database will come back up in the default rollback mode of DELETE rather than the previous TRUNCATE setting.

The persistence of WAL mode means that applications can be converted to using SQLite in WAL mode without making any changes to the application itself. One has merely to run "PRAGMA journal_mode=WAL;" on the database file(s) using the [command-line shell](https://www.sqlite.org/cli.html) or other utility, then restart the application.

The WAL journal mode will be set on all connections to the same database file if it is set on any one connection.

If I understand it right the persistency is stored into the database itself and the connection will be created in WAL on default.

It should not affect the rest of the connection after marking it with TRUNCATE though.

[jellyfin-bot]

This pull request has merge conflicts. Please resolve the conflicts so the PR can be successfully reviewed and merged.