3.1.0 - "<path>" is not empty. Only new or empty existing directories are accepted for --datadir

[Osiris-Team]

Worked fine in versions before 3.1.0 @vorburger , maybe due to mariadb update?

    public static void initIntegratedMariaDB() throws ManagedProcessException {
        DBConfigurationBuilder configBuilder = DBConfigurationBuilder.newBuilder();
        configBuilder.setPort(0); // OR, default: setPort(0); => autom. detect free port
        configBuilder.setDataDir(new File(System.getProperty("user.dir") + "/db").getAbsolutePath());
        db = DB.newEmbeddedDB(configBuilder.build());
        db.start();
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            try {
                System.out.println("Stopping DB...");
                db.stop();
                System.out.println("Stopped DB successfully.");
            } catch (ManagedProcessException e) {
                throw new RuntimeException(e);
            }
        }));
    }

2024-04-04 09:20:10.521 [main] INFO ch.vorburger.mariadb4j.Util - Created directory: C:\Users\arman\AppData\Local\Temp\MariaDB4j\base
2024-04-04 09:20:10.526 [main] INFO ch.vorburger.mariadb4j.Util - Created directory: C:\Users\arman\AppData\Local\Temp\MariaDB4j\base\libs
2024-04-04 09:20:10.527 [main] INFO ch.vorburger.mariadb4j.Util - Created directory: C:\Users\arman\AppData\Local\Temp\MariaDB4j\tmp\51248
2024-04-04 09:20:12.052 [main] INFO ch.vorburger.mariadb4j.Util - Unpacked 80 files from classpath*:ch/vorburger/mariadb4j/mariadb-10.11.5-fix1/winx64/** to C:\Users\arman\AppData\Local\Temp\MariaDB4j\base
2024-04-04 09:20:12.052 [main] INFO ch.vorburger.mariadb4j.DB - Installing a new embedded database to: C:\Users\arman\AppData\Local\Temp\MariaDB4j\base
2024-04-04 09:20:12.097 [main] INFO ch.vorburger.exec.ManagedProcess - Starting Program [C:\Users\arman\AppData\Local\Temp\MariaDB4j\base\bin\mariadb-install-db.exe, --datadir=C:\Users\arman\IdeaProjects\daniel_manager\db] (in working directory C:\Users\arman\AppData\Local\Temp\MariaDB4j\base)
2024-04-04 09:20:12.216 [main] INFO ch.vorburger.exec.ManagedProcess - Thread is now going to wait for this process to terminate itself: Program [C:\Users\arman\AppData\Local\Temp\MariaDB4j\base\bin\mariadb-install-db.exe, --datadir=C:\Users\arman\IdeaProjects\daniel_manager\db] (in working directory C:\Users\arman\AppData\Local\Temp\MariaDB4j\base)
2024-04-04 09:20:12.260 [CommonsExecStreamPumper-pool-4-thread-1] INFO ch.vorburger.exec.ManagedProcess - mariadb-install-db.exe: Running bootstrap
2024-04-04 09:20:12.261 [CommonsExecStreamPumper-pool-4-thread-2] ERROR ch.vorburger.exec.ManagedProcess - mariadb-install-db.exe: ERROR : Data directory C:\Users\arman\IdeaProjects\daniel_manager\db is not empty. Only new or empty existing directories are accepted for --datadir
2024-04-04 09:20:12.281 [main] ERROR ch.vorburger.exec.ManagedProcess - Program [C:\Users\arman\AppData\Local\Temp\MariaDB4j\base\bin\mariadb-install-db.exe, --datadir=C:\Users\arman\IdeaProjects\daniel_manager\db] (in working directory C:\Users\arman\AppData\Local\Temp\MariaDB4j\base) failed
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
	at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:355)
	at org.apache.commons.exec.DefaultExecutor.lambda$execute$0(DefaultExecutor.java:269)
	at java.base/java.lang.Thread.run(Thread.java:1583)
2024-04-04 09:20:12.283 [CommonsExecDefaultExecutorpool-1-thread-1] ERROR ch.vorburger.exec.LoggingExecuteResultHandler - Program [C:\Users\arman\AppData\Local\Temp\MariaDB4j\base\bin\mariadb-install-db.exe, --datadir=C:\Users\arman\IdeaProjects\daniel_manager\db] (in working directory C:\Users\arman\AppData\Local\Temp\MariaDB4j\base) failed unexpectedly
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
	at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:355)
	at org.apache.commons.exec.DefaultExecutor.lambda$execute$0(DefaultExecutor.java:269)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Exception in thread "main" ch.vorburger.exec.ManagedProcessException: An error occurred while installing the database
	at ch.vorburger.mariadb4j.DB.install(DB.java:139)
	at ch.vorburger.mariadb4j.DB.newEmbeddedDB(DB.java:89)
	at com.osiris.danielmanager.Application.initIntegratedMariaDB(Application.java:73)
	at com.osiris.danielmanager.Application.main(Application.java:45)
Caused by: ch.vorburger.exec.ManagedProcessException: Program [C:\Users\arman\AppData\Local\Temp\MariaDB4j\base\bin\mariadb-install-db.exe, --datadir=C:\Users\arman\IdeaProjects\daniel_manager\db] (in working directory C:\Users\arman\AppData\Local\Temp\MariaDB4j\base) failed with Exception: , last 100 lines of console:
Running bootstrap
ERROR : Data directory C:\Users\arman\IdeaProjects\daniel_manager\db is not empty. Only new or empty existing directories are accepted for --datadir
	at ch.vorburger.exec.ManagedProcess.checkResult(ManagedProcess.java:324)
	at ch.vorburger.exec.ManagedProcess.waitForExitMaxMsWithoutLog(ManagedProcess.java:471)
	at ch.vorburger.exec.ManagedProcess.waitForExit(ManagedProcess.java:433)
	at ch.vorburger.mariadb4j.DB.install(DB.java:137)
	... 3 more
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
	at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:355)
	at org.apache.commons.exec.DefaultExecutor.lambda$execute$0(DefaultExecutor.java:269)
	at java.base/java.lang.Thread.run(Thread.java:1583)

[vorburger]

Worked fine in versions before 3.1.0 @vorburger , maybe due to mariadb update?

Yeah, looks like the more recent MariaDB version has become "stricter" than the previous one...

ERROR : Data directory C:\Users\arman\IdeaProjects\daniel_manager\db is not empty. Only new or empty existing directories are accepted for --datadir

But you could easily address this in your own code by first deleting that directory, no? (Or have I misunderstood the problem?)

I likely won't myself be making any fix for this corner case.

But if you would like to contribute a Pull Request with a "fix" for this (with test coverage), I would be happy to review it.

[Osiris-Team]

But you could easily address this in your own code by first deleting that directory, no? (Or have I misunderstood the problem?)

@vorburger I am using the db persistently for the backend of a project so that would delete all the data if I understood it correctly, so no can't do that :/

[vorburger]

@vorburger I am using the db persistently for the backend of a project so that would delete all the data if I understood it correctly, so no can't do that :/

Oh, right; I see.

In that case, perhaps you would like to further debug what exactly is causing this?

It seems a bit unlikely that this is a MariaDB bug, no? (I have not had a closer look.)

Is MariaDB4j doing something wrong and e.g. calling mariadb-install-db.exe at a point in time when it should not? (That's just a hunch; I haven't looked at it closely.)

Pull Request to change the start-up behavior to be correct for existing DB, if that's indeed the real problem here, would certainly be welcome.

[Osiris-Team]

@vorburger you should be able to reproduce this by simply executing the program twice with the above code.
However it could also be that its maybe due to the old java process still being active somehow in background, I'm not sure though if this was really the case when I saw that issue.

Sorry I can't be much of a help, I directly reverted back the version since I need to work on this project at the moment and time is not really something I got much of currently :/

I may have time next week to look into this in more detail.

[TheKnowles]

I can recreate in a unit test on Windows. Linux does not exhibit the same behavior.

Looking at comments here from 2019: https://mariadb.com/kb/en/mysql_install_dbexe/

It looks like the behavior at one point was to blow the entire directory away? I am wondering if a check got put in at some point to avoid that behavior. I have not had a chance to confirm in code yet.

Investigating.

[TheKnowles]

Found it.

So this got changed in June 2020.

https://jira.mariadb.org/browse/MDEV-23052

Not a huge fan of how they merge branches into each other, it makes tracing the origin of this change a little difficult.

If the installation failed for whatever reason, the data dir could be destroyed, so it looks like they decided to just avoid that issue altogether and force a clean directory on install.

Looking at ways we could work around this on windows.

[Osiris-Team]

Great, maybe even never versions of mariadb fix this issue?

[TheKnowles]

Great, maybe even never versions of mariadb fix this issue?

Unfortunately no, even on the latest 11.5, this is the behavior.

[Osiris-Team]

This is a pretty weird issue especially since it's only on windows. What if the data dir is simply not set, where does the data get saved then?

[TheKnowles]

This is a pretty weird issue especially since it's only on windows. What if the data dir is simply not set, where does the data get saved then?

It will look in config files and then default to /var/lib/mysql/data or Windows equivalent.

[Osiris-Team]

@TheKnowles Okay as long as that isnt a temp directory I am fine with that, since I don't see another workaround for this.

[TheKnowles]

The same issue would apply. The datadir is not empty (coming from a parameter or default) and it will kill the install. I need to investigate the upgrade code for windows a little more closely. MariaDB4j could check if the datadir is not empty and run upgrade instead of install. A cursory look at the upgrade code last week looked like it was very service/msi esque heavy.

[Osiris-Team]

@TheKnowles that sounds like it could work!

[Osiris-Team]

@TheKnowles any news?