BLOCK storage format between AuraDB and Community editions

[jjbindr]

We have several AuraDB paid instances. Our workflow requires developers to run our full stack locally which means they need to pull snapshots from Aura frequently and run them in a local Neo4j Docker instance for development purposes.

This has been working for 2 years until this month.

It looks like your new BLOCK storage format has been rolled out to Aura but apparently has been made "Enterprise Only", which makes dump files completely non portable for us now.

When attempting to load a dump from Aura we are met with

Loading dump file: 'neo4j-05-08-24.dump'...
Done: 218 files, 6.386MiB processed.
Failed to load database 'neo4j': Block format detected for database neo4j but unavailable in this edition.Load failed for databases: 'neo4j'
Load failed for databases: 'neo4j'

Please tell me this is an oversight

[jjbindr]

Adding additional information...

loading a BLOCK storage dump file actually does change all my local files (see below). In the browser the "neo4j" database becomes "unavailable"

From:

/var/lib/neo4j/data/databases/neo4j

-rw-r--r-- 1 neo4j neo4j    73728 Apr 10 15:26 neostore.counts.db
-rw-r--r-- 1 neo4j neo4j   499712 Apr 10 15:26 neostore.indexstats.db
-rw-r--r-- 1 neo4j neo4j     8192 Apr  9 03:00 neostore.labeltokenstore.db
-rw-r--r-- 1 neo4j neo4j    40960 Apr 10 15:26 neostore.labeltokenstore.db.id
-rw-r--r-- 1 neo4j neo4j     8192 Apr  9 03:00 neostore.labeltokenstore.db.names
-rw-r--r-- 1 neo4j neo4j    40960 Apr 10 15:26 neostore.labeltokenstore.db.names.id

[...]

To:

-rw-r--r-- 1 neo4j neo4j 589824 Apr 10 15:47 block.big_values.db
-rw-r--r-- 1 neo4j neo4j  49152 Apr 10 15:47 block.big_values.db.id
-rw-r--r-- 1 neo4j neo4j  49152 Apr 10 15:47 block.counts.db
-rw-r--r-- 1 neo4j neo4j  40960 Apr 10 15:47 block.huge.db
-rw-r--r-- 1 neo4j neo4j 344064 Apr 10 15:47 block.indexstats.db
-rw-r--r-- 1 neo4j neo4j  49152 Apr 10 15:47 block.metadata.db
-rw-r--r-- 1 neo4j neo4j 491520 Apr 10 15:47 block.node.xd.db

[..]

I was hoping I could then migrate it back to ALIGNED but that fails saying it doesn't see a "neo4j" database. If this is a bug it would be great if it could be fixed. This is a huge problem for my engineering team.

[root@ip-10-10-71-30 import]# neo4j-admin database migrate --to-format=aligned neo4j
2024-04-10 15:57:31.330+0000 INFO  [o.n.c.d.MigrateStoreCommand] Starting migration for database 'neo4j'
2024-04-10 15:57:31.734+0000 ERROR [o.n.c.d.MigrateStoreCommand] Failed to migrate database 'neo4j': Database 'neo4j' does not exist
2024-04-10 15:57:31.735+0000 ERROR [o.n.c.d.MigrateStoreCommand] Migration failed for databases: 'neo4j'
Migration failed for databases: 'neo4j'
Run with '--verbose' for a more detailed error message.

[mauvo]

Hey Josh - sorry your workflow has been disrupted.

Currently we're only rolling out Block format to new databases in Aura. Existing databases aren't being automatically converted. Do you recreate your Aura databases frequently? (I'd love to understand why.)

The dump files can be converted to other Neo4j formats using Neo4j EE's neo4j-admin copy command and the --to-format option. neo4j-admin copy is free to use.

Neo4j-admin documentation

Can you use the developer version of Neo4j EE that comes as part of Neo4j Desktop? It has some limitations on the license - one of which is that it should only run on the developer's laptop. Perhaps it's worth speaking to your account manager to figure out if it's possible to use Neo4jEE.

[jjbindr]

Hi David, thanks for the response.

Unfortunately we cannot afford an enterprise license yet so we make do with AuraDB.

We don't regularly recreate databases. This change seemed to happen sometime between March 28 and first week in April. I know we didn't recreate it during this time. Does it switch to the new format on resizing? We do that semi regularly for costs reasons.

Thanks for the suggestion on Neo4j Desktop. While I can load a BLOCK dump file ok, I can't see any way to dump it again in a different format using the desktop interface. Is there a neo4j-admin tool that comes with Desktop? I can't seem to find one. We're on mac btw.

Shouldn't neo4j-admin migrate allow me to "downgrade" the format to ALIGNED? that seems like a bug.

[mauvo]

neo4j-admin is packaged with Neo4j EE and CE - but you need the EE version since CE doesn't have the ability to read Block format yet. It's in the /bin folder alongside the main neo4j application.

I wondered if Neo4j Desktop would meet your developer's needs instead of Docker. And, less straight-forwardly, if it would be a good place to start a conversation with your account manager. It's better for everyone if you can use EE rather than CE. That said, any license cost would depend on the details of how you're currently using CE.

Aura shouldn't switch format on resizing... but maybe something unexpected is happening. We'll think through those steps and see how that might come about. You can force Aura back to the old format by uploading your dump (once you've converted it back to Aligned format). Although it's not a long-term fix, and that behavior will probably change at some point.

[jonny4j]

Hey @jjbindr. I'm from the Aura Product team. Sorry to hear that you've been impacted by this change, hopefully we can find a way forward.

Can I ask if you have logged a Support ticket in relation to this?

[jjbindr]

Thanks for the response @jonny4j

If you want me to raise a support issue about using Neo4j Desktop to solve this issue, I figured that out.

It took me awhile to convert the database back to aligned format using Desktop but I finally got it.
What used to take a few seconds to import an AuraDB will now take substantially longer. I wrote up a wiki page for my team on it and attached it below for anyone else who runs into this. You can see the number of steps and is somewhat painful.

Using Neo4j Desktop as part of our stack isn't realistic for reasons too long to go into. If I could use only paid Aura instance I would, but the Community edition has to be part of our overall workflow for cost reasons.

I appreciate your involvement seeing how you guys are from the Product Mgmt team directly.
I can only ask that Neo4j reconsiders this decision and keep dump files portable between editions.

If you still want me to raise a support ticket in general about this, I'm happy to do that.

Thanks for listening.

wiki-dump file from BLOCK format to ALIGNED format-110424-215957.pdf

[vigneshsrinivasan9]

Yes. I get the same error and I use community edition 5.19.0. How do I migrate the block format to aligned format?

Failed to load database 'neo4j': Block format detected for database neo4j but unavailable in this edition.
Load failed for databases: 'neo4j'
org.neo4j.cli.CommandFailedException: Load failed for databases: 'neo4j'
at org.neo4j.commandline.dbms.LoadCommand.checkFailure(LoadCommand.java:235)
at org.neo4j.commandline.dbms.LoadCommand.loadDump(LoadCommand.java:223)
at org.neo4j.commandline.dbms.LoadCommand.execute(LoadCommand.java:160)

[jonny4j]

Hey @jjbindr @vigneshsrinivasan9. Apologies for the delayed response, I've been out of the office recently.

Thanks for the additional context here, this is useful. Let me take this away, speak with some colleagues and come back to you shortly.