Cannot open readOnly and readWrite connection on the same database directory

[andyfengHKU]

Adding question from Discord for context:

Is there a way to share the same Kuzu database among multiple processes. At the moment this seems to be impossible: if I connect to a Kuzu DB from the CLI and then run a Python script that connects to the same DB I get an error RuntimeError: Could not set lock on file : kuzu/.lock

I currently have the Kuzu CLI open on the database (so I guess that will have write permission), and I just changed my python code to kuzu_db = kuzu.Database(database_path=kuzu_db_path, buffer_pool_size=1024 ** 3, read_only=True) I still get the error RuntimeError: Could not set lock on file : kuzu/.lock

[hououou]

We usefcntl to ensure the access mode to databases.
fcntl can not add any other locks while already holding an exclusive write lock.

An exclusive or write lock gives a process exclusive access for writing to the specified part of the file. While a write lock is in place, no other process can lock that part of the file.

[ray6080]

This should be expected. The way to share the same Kuzu database among multiple processes is to open the database under READ ONLY mode.

We should prevent concurrent read and write access to the same kuzu database from different processes, as there is no guarantee we can provide to make sure that read and write transactions from different processes are consistent. So I would suggest we don't change this behaviour. but we should document this more clearly in our docs for sure.

[sapalli2989]

We should prevent concurrent read and write access to the same kuzu database from different processes, as there is no guarantee we can provide to make sure that read and write transactions from different processes are consistent.

To clarify and make sure I got this correctly: It is expected behavior that access from multiple (ro+rw) Database objects/instances to one database directory on filesystem within the same process is not possible either (see #2934). Here is a shorter version:

import tempfile
import kuzu

with tempfile.TemporaryDirectory() as d:
    conn_rw = kuzu.Connection(kuzu.Database(d))
    conn_ro = kuzu.Connection(kuzu.Database(d, read_only=True))

    conn_rw.execute("CREATE NODE TABLE V(id STRING,PRIMARY KEY(id));")
    conn_rw.execute("CREATE (v:V {id: 'foo'}) RETURN v;")

    print((conn_rw.execute("MATCH (v:V {id: 'foo'}) RETURN v;").get_next()[0]))
    # {'_id': {'offset': 0, 'table': 0}, '_label': 'V', 'id': 'foo'}

    print((conn_ro.execute("MATCH (v:V {id: 'foo'}) RETURN v;").get_next()[0]))
    # RuntimeError: Binder exception: Table V does not exist.

I think it makes sense to provide an appropriate error here as well.

[ray6080]

hi @sapalli2989 , yes. you're right. and we should error on this too. also related to #3292.