when machine is shutdown accidentally, microstream storage is corrupted

[yqbjtu]

Environment Details

• MicroStream Version:08.01.00-MS-GA
• JDK version: 17.0.4
• OS:Linux CentOS Linux release 7.7.1908 (Core)
• Spring 2.6.8

Describe the bug

I used the LazyArrayList to store my entity, the virtual machine is shutdown accidentally（not shutdown by man or plan, it is caused by some air condition issue ） , then I start my program by ‘systemctl start xxx’, it reports the following error

2023-06-15 19:32:28,040 ERROR o.m.s.types.StorageChannelTask.addProblem [MicroStream-StorageChannel-0] Error occurred in storage channel#0
one.microstream.storage.exceptions.StorageException: null
        at one.microstream.storage.types.StorageFileManager$Default.readTransactionsFile(StorageFileManager.java:722)
        at one.microstream.storage.types.StorageFileManager$Default.readStorage(StorageFileManager.java:688)
        at one.microstream.storage.types.StorageChannel$Default.readStorage(StorageChannel.java:728)
        at one.microstream.storage.types.StorageChannelTaskInitialize$Default.internalProcessBy(StorageChannelTaskInitialize.java:193)
        at one.microstream.storage.types.StorageChannelTaskInitialize$Default.internalProcessBy(StorageChannelTaskInitialize.java:36)
        at one.microstream.storage.types.StorageChannelTask$Abstract.processBy(StorageChannelTask.java:252)
        at one.microstream.storage.types.StorageChannel$Default.work(StorageChannel.java:409)
        at one.microstream.storage.types.StorageChannel$Default.run(StorageChannel.java:492)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: one.microstream.storage.exceptions.StorageException: Zero length transactions entry.
        at one.microstream.storage.types.StorageTransactionsAnalysis$Logic.processBufferedEntities(StorageTransactionsAnalysis.java:377)
        at one.microstream.storage.types.StorageTransactionsAnalysis$Logic.processInputFile(StorageTransactionsAnalysis.java:343)
        at one.microstream.storage.types.StorageTransactionsAnalysis$Logic.processInputFile(StorageTransactionsAnalysis.java:301)
        at one.microstream.storage.types.StorageLiveTransactionsFile$Default.processBy(StorageLiveTransactionsFile.java:93)
        at one.microstream.storage.types.StorageFileManager$Default.readTransactionsFile(StorageFileManager.java:716)
        ... 8 common frames omitted
2023-06-15 19:32:28,047 WARN o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext.refresh [main] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'xxx': Unsatisfied dependency expressed through field 'connDB'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cellConnDB' defined in ai.momenta.osmdb.cellconn.CellConnConfig: Unsatisfied dependency expressed through method 'cellConnDB' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storageService' defined in ai.momenta.osmdb.cellconn.CellConnConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [xxx.StorageService]: Factory method 'storageService' threw exception; nested exception is one.microstream.storage.exceptions.StorageException: Problem in channel #0

To Reproduce

Expected behavior

when the machine is shutdown accidentally, microstream storage is not corrupted

Screenshots

Additional context

my storage is opened by the following way.

    StorageEntityCache.Default.setGarbageCollectionEnabled(enableGC);//todo change from default to each instance

    var builder = EmbeddedStorageConfiguration.Builder();
    var foundation = builder
      .setStorageDirectory(graphDbPath)
      .setChannelCount(1)
      .setDataFileMaximumSize(ByteSize.New(2, ByteUnit.GB))
      .createEmbeddedStorageFoundation();

    foundation.onConnectionFoundation(BinaryHandlersJDK8::registerJDK8TypeHandlers);

    return foundation.createEmbeddedStorageManager().start();

[hg-ms]

The exception indicates that the transaction file has an invalid entry written. The “zero length” relates to the transaction entry header and contains the header length that is a constant value (see comment in one.microstream.storage.types.StorageTransactionsAnalysis.java).
As the error occurred after the VM has been shut down accidentally I assume it is caused by an incomplete io operation while writing the transaction entry to the transaction log.

Nevertheless we must ensure that such erroneous transaction entries don’t prevent the storage from starting up if possible.

@yqbjtu:
The starting code you provided is ok, that is most likely not causing the problem.

Did you try to delete the transaction log before restarting the storage?
(The transaction logs are the “.sft” files in the channel directories) if no transaction log is found the storage will try to start without checking the transactions.

Do you still have the corrupted transaction log files? Or did you check that files if they only contain zeros?

[yqbjtu]

The exception indicates that the transaction file has an invalid entry written. The “zero length” relates to the transaction entry header and contains the header length that is a constant value (see comment in one.microstream.storage.types.StorageTransactionsAnalysis.java). As the error occurred after the VM has been shut down accidentally I assume it is caused by an incomplete io operation while writing the transaction entry to the transaction log.

Nevertheless we must ensure that such erroneous transaction entries don’t prevent the storage from starting up if possible.

@yqbjtu: The starting code you provided is ok, that is most likely not causing the problem.

Did you try to delete the transaction log before restarting the storage? (The transaction logs are the “.sft” files in the channel directories) if no transaction log is found the storage will try to start without checking the transactions.

Do you still have the corrupted transaction log files? Or did you check that files if they only contain zeros?

the corrupted microstream files info :
ericyang@ericyangs-MacBook-Pro channel_0 % ls -al
total 3285120
drwxr-xr-x 4 ericyang staff 128 6 15 01:11 .
drwxr-xr-x 4 ericyang staff 128 6 3 21:53 ..
-rw-r--r-- 1 ericyang staff 1508540416 6 15 18:25 channel_0_16.dat
-rw-r--r-- 1 ericyang staff 163581952 6 15 18:25 transactions_0.sft
ericyang@ericyangs-MacBook-Pro channel_0 % du -sh channel_0_16.dat
1.4G channel_0_16.dat
ericyang@ericyangs-MacBook-Pro channel_0 % du -sh transactions_0.sft
156M transactions_0.sft

the transactions_0.sft size is 156M, When I delete transactions_0.sft, program fails to start

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00000001180217bc, pid=10532, tid=33835

JRE version: OpenJDK Runtime Environment Temurin-17.0.3+7 (17.0.3+7) (build 17.0.3+7)

Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.3+7 (17.0.3+7, mixed mode, emulated-client, tiered, compressed class ptrs, z gc, bsd-aarch64)

Problematic frame:

v ~StubRoutines::jbyte_disjoint_arraycopy

No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

[useryq8]

do we have any plan to fix this issue?