File with name config.cd does not exist in storage

[jamieb22]

OrientDB Version: 3.2.25

Java Version: 21

OS: Windows

Expected behavior

Have an Orient Db database created by 3.1.7. Upgrade Orient DB to 3.2.25. On DB startup, receive message that the database wasn't closed, and need to run older Orient DB version first. Run 3.1.7 again. Shutdown db. Move to 3.2.25. On startup an error is outputted.

Actual behavior

com.orientechnologies.orient.core.exception.ODatabaseException: Cannot open database 'archiva'
	at com.orientechnologies.orient.core.db.OrientDBEmbedded.open(OrientDBEmbedded.java:555)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:899)
	at com.orientechnologies.orient.core.db.OPartitionedDatabasePool$DatabaseDocumentTxPooled.internalOpen(OPartitionedDatabasePool.java:437)
	at com.orientechnologies.orient.core.db.OPartitionedDatabasePool.openDatabase(OPartitionedDatabasePool.java:304)
	at com.orientechnologies.orient.core.db.OPartitionedDatabasePool.acquire(OPartitionedDatabasePool.java:259)
	at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.<init>(OrientBaseGraph.java:178)
	at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.<init>(OrientTransactionalGraph.java:82)
	at com.tinkerpop.blueprints.impls.orient.OrientGraph.<init>(OrientGraph.java:125)
	at com.tinkerpop.blueprints.impls.orient.OrientGraphFactory$1.getGraph(OrientGraphFactory.java:92)
	at com.tinkerpop.blueprints.impls.orient.OrientGraphFactory.getTx(OrientGraphFactory.java:243)
..
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Exception during execution of atomic operation inside of storage archiva
	at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.executeInsideAtomicOperation(OAtomicOperationsManager.java:146)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:543)
	at com.orientechnologies.orient.core.db.OrientDBEmbedded.getAndOpenStorage(OrientDBEmbedded.java:592)
	at com.orientechnologies.orient.core.db.OrientDBEmbedded.open(OrientDBEmbedded.java:544)
	... 19 common frames omitted
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Exception during execution of component operation inside component config.cd in storage archiva
	at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.executeInsideComponentOperation(OAtomicOperationsManager.java:172)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.executeInsideComponentOperation(OAtomicOperationsManager.java:157)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.executeInsideComponentOperation(ODurableComponent.java:102)
	at com.orientechnologies.orient.core.storage.cluster.v2.OPaginatedClusterV2.open(OPaginatedClusterV2.java:196)
	at com.orientechnologies.orient.core.storage.config.OClusterBasedStorageConfiguration.load(OClusterBasedStorageConfiguration.java:250)
	at com.orientechnologies.orient.core.storage.disk.OLocalPaginatedStorage.initConfiguration(OLocalPaginatedStorage.java:536)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.lambda$open$1(OAbstractPaginatedStorage.java:554)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.executeInsideAtomicOperation(OAtomicOperationsManager.java:140)
	... 22 common frames omitted
Caused by: com.orientechnologies.orient.core.exception.OStorageException: File with name config.cd does not exist in storage archiva
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.loadFile(OWOWCache.java:655)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationBinaryTracking.loadFile(OAtomicOperationBinaryTracking.java:345)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.openFile(ODurableComponent.java:180)
	at com.orientechnologies.orient.core.storage.cluster.v2.OPaginatedClusterV2.lambda$open$1(OPaginatedClusterV2.java:201)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.executeInsideComponentOperation(OAtomicOperationsManager.java:165)
	... 29 common frames omitted

Steps to reproduce

See above
archiva.zip

<An SQL script to reproduce the problem or a JUnit test case will increase A LOT the chance to have a quick fix>

[laa]

Hi @jamieb22

My best guess is that it was not flushed to the disk.
Let me inspect the code to ensure it will never happen to you.

[laa]

Hi @jamieb22
May I ask you to share any logs you have with us?

[jamieb22]

Andrey... additional logs shown below.

• Running V2.1.20 Orient DB server (not embedded)
• Shut the server down CLEANLY
• Installed Orient DB 3.2.26
• Start Orient DB 3.2.26 up .. logs said something to the effect that cant restore from Orient DB 3.1.20, first need to restore from Orient DB 3.1.20. Its not clear why it was doing a restore in any case, since I definitely performed a clean shutdown of Orient DB, and the process wasn't hung or killed.
• Shutdown Orient DB 3.2.26. CLEANLY
• Install Orient DB 3.1.20 server (not embedded)
• Start Orient DB 3.1.20.
• Get error File with name config.cd does not exist in storage OSystem ( see logs below)

As far as I know, there is no way to recover from this situation. The above upgrade issue is reproducible. I've seen nearly every time I attempted to an upgrade from 3.1. to 3.2.

As an aside: are there any test cases to test the upgrading of Orient DB? As you know, I've highlighted upgrade issues in the past. My experience remains that Orient DB frequently get corrupted when upgrading. Furthermore, once the database gets corrupted, its difficult (and often impossible) for an administrator to recover. It would be nice if Orient DB could repair itself somehow. A missing config.cd file. Can it be regenerated? I still have a server running 2.0.32 because I am too scared to upgrade it.

2024-02-06 13:10:09:624 INFO  Limit of open files for disk cache will be set to 60000. [OEngineLocalPaginated]
2024-02-06 13:10:09:663 INFO  Loading configuration from: /opt/orientdb/config/orientdb-server-config.xml... [OServerConfigurationLoaderXml]
2024-02-06 13:10:10:018 INFO  OrientDB Server v3.1.20 - Veloce (build a9065d2198411f8b5eab3bb5240c4aad67b3dbc9, branch UNKNOWN) is starting up... [OServer]
2024-02-06 13:10:11:062 INFO  System is started under an effective user : `orientdb` [OEngineLocalPaginated]
2024-02-06 13:10:12:250 INFO  WAL maximum segment size is set to 23,866 MB [OrientDBDistributed]
2024-02-06 13:10:12:273 INFO  Databases directory: /mnt/volume_ams3_02/databases/orient [OServer]
2024-02-06 13:10:12:458 INFO  Page size for WAL located in /mnt/volume_ams3_02/databases/orient/OSystem is set to 4096 bytes. [CASDiskWriteAheadLog]
2024-02-06 13:10:12:514 INFO  DWL:OSystem: block size = 4096 bytes, maximum segment size = 23866 MB [DoubleWriteLogGL]
2024-02-06 13:10:12:596 SEVER Exception `02583EC5` in storage `plocal:/mnt/volume_ams3_02/databases/orient/OSystem`: 3.1.20 - Veloce (build a9065d2198411f8b5eab3bb5240c4aad67b3dbc9, branch UNKNOWN) [OLocalPaginatedStorage]
com.orientechnologies.orient.core.exception.OStorageException: Exception during execution of atomic operation inside of storage OSystem
        at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.executeInsideAtomicOperation(OAtomicOperationsManager.java:162)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:359)
        at com.orientechnologies.orient.core.db.OrientDBEmbedded.getAndOpenStorage(OrientDBEmbedded.java:500)
        at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthorization(OrientDBEmbedded.java:455)
        at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthorization(OrientDBEmbedded.java:78)
        at com.orientechnologies.orient.server.OSystemDatabase.openSystemDatabase(OSystemDatabase.java:93)
        at com.orientechnologies.orient.server.OSystemDatabase.checkServerId(OSystemDatabase.java:169)
        at com.orientechnologies.orient.server.OSystemDatabase.init(OSystemDatabase.java:157)
        at com.orientechnologies.orient.server.OSystemDatabase.<init>(OSystemDatabase.java:50)
        at com.orientechnologies.orient.server.OServer.initSystemDatabase(OServer.java:1282)
        at com.orientechnologies.orient.server.OServer.activate(OServer.java:445)
        at com.orientechnologies.orient.server.OServerMain$1.run(OServerMain.java:49)
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Exception during execution of component operation inside component config.cd in storage OSystem
        at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.executeInsideComponentOperation(OAtomicOperationsManager.java:188)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.executeInsideComponentOperation(OAtomicOperationsManager.java:173)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.executeInsideComponentOperation(ODurableComponent.java:102)
        at com.orientechnologies.orient.core.storage.cluster.v2.OPaginatedClusterV2.open(OPaginatedClusterV2.java:200)
        at com.orientechnologies.orient.core.storage.config.OClusterBasedStorageConfiguration.load(OClusterBasedStorageConfiguration.java:250)
        at com.orientechnologies.orient.core.storage.disk.OLocalPaginatedStorage.initConfiguration(OLocalPaginatedStorage.java:549)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.lambda$open$1(OAbstractPaginatedStorage.java:370)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.executeInsideAtomicOperation(OAtomicOperationsManager.java:156)
        ... 11 more
Caused by: com.orientechnologies.orient.core.exception.OStorageException: File with name config.cd does not exist in storage OSystem
        at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.loadFile(OWOWCache.java:632)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationBinaryTracking.loadFile(OAtomicOperationBinaryTracking.java:364)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.openFile(ODurableComponent.java:188)
        at com.orientechnologies.orient.core.storage.cluster.v2.OPaginatedClusterV2.lambda$open$1(OPaginatedClusterV2.java:205)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.executeInsideComponentOperation(OAtomicOperationsManager.java:181)
        ... 18 more