You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hey, I'm currently faced with a problem of database crash with very high CPU load and abnormal disk I/O multiple times, during which process observed Detected VM stop-the-world pause.
Configurations
Neo4j version: community 5.14.0
Operating system: rocky linux 8 (4vCPU, 4GiB memory, 100GiB storage) on GCP
API: Python driver v5.1.3 and Cypher
Example bug report
2024-01-06 04:15:13.816+0000 INFO [o.n.k.i.t.l.p.LogPruningImpl] [neo4j/344d7f25] No log version pruned. The strategy used was '2 days 2147483648 size'.
2024-01-06 05:30:41.731+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=859, gcTime=0, gcCount=0}
2024-01-08 02:18:21.729+0000 INFO [o.n.g.f.m.GlobalModule] Logging config in use: File '/etc/neo4j/server-logs.xml'
2024-01-08 02:18:21.925+0000 WARN [o.n.k.i.JvmChecker] The max heap memory has not been configured. It is recommended that it is always explicitly configured, to ensure the system has a balanced configuration. Until then, a JVM computed heuristic of 958398464 bytes is used instead. If you are running neo4j server, you need to configure server.memory.heap.max_size in neo4j.conf. If you are running neo4j embedded, you have to launch the JVM with -Xmx set to a value. You can run neo4j-admin server memory-recommendation for memory configuration suggestions.
2024-01-08 02:18:21.926+0000 WARN [o.n.k.i.JvmChecker] The initial heap memory has not been configured. It is recommended that it is always explicitly configured, to ensure the system has a balanced configuration. Until then, a JVM computed heuristic of 60817408 bytes is used instead. If you are running neo4j server, you need to configure server.memory.heap.initial_size in neo4j.conf. If you are running neo4j embedded, you have to launch the JVM with -Xms set to a value. You can run neo4j-admin server memory-recommendation for memory configuration suggestions.
2024-01-08 02:18:22.039+0000 WARN [o.n.i.p.PageCache] The server.memory.pagecache.size setting has not been configured. It is recommended that this setting is always explicitly configured, to ensure the system has a balanced configuration. Until then, a computed heuristic value of 1434413056 bytes will be used instead. Run `neo4j-admin memory-recommendation` for memory configuration suggestions.
2024-01-08 02:18:22.192+0000 INFO [o.n.i.d.DiagnosticsManager]
********************************************************************************
[ System diagnostics ]
********************************************************************************
--------------------------------------------------------------------------------
[ System memory information ]
--------------------------------------------------------------------------------
Total Physical memory: 3.564GiB
Free Physical memory: 1.571GiB
Committed virtual memory: 4.138GiB
Total swap space: 0B
Free swap space: 0B
--------------------------------------------------------------------------------
[ JVM memory information ]
--------------------------------------------------------------------------------
Free memory: 76.19MiB
Total memory: 107.0MiB
Max memory: 914.0MiB
Garbage Collector: G1 Young Generation: [G1 Eden Space, G1 Survivor Space, G1 Old Gen]
Garbage Collector: G1 Old Generation: [G1 Eden Space, G1 Survivor Space, G1 Old Gen]
Memory Pool: CodeHeap 'non-nmethods' (Non-heap memory): committed=2.438MiB, used=1.234MiB, max=5.566MiB, threshold=0B
Memory Pool: Metaspace (Non-heap memory): committed=21.81MiB, used=21.55MiB, max=-1B, threshold=0B
Memory Pool: CodeHeap 'profiled nmethods' (Non-heap memory): committed=5.500MiB, used=5.445MiB, max=117.2MiB, threshold=0B
Memory Pool: Compressed Class Space (Non-heap memory): committed=2.438MiB, used=2.327MiB, max=1.000GiB, threshold=0B
Memory Pool: G1 Eden Space (Heap memory): committed=75.00MiB, used=3.000MiB, max=-1B, threshold=?
Memory Pool: G1 Old Gen (Heap memory): committed=29.00MiB, used=25.09MiB, max=914.0MiB, threshold=0B
Memory Pool: G1 Survivor Space (Heap memory): committed=3.000MiB, used=2.208MiB, max=-1B, threshold=?
Memory Pool: CodeHeap 'non-profiled nmethods' (Non-heap memory): committed=2.438MiB, used=1.488MiB, max=117.2MiB, threshold=0B
--------------------------------------------------------------------------------
[ Operating system information ]
--------------------------------------------------------------------------------
Operating System: Linux; version: 4.18.0-513.5.1.el8_9.x86_64; arch: amd64; cpus: 4
Max number of file descriptors: 65535
Number of open file descriptors: 192
Process id: 13274
Byte order: LITTLE_ENDIAN
Local timezone: UTC
Memory page size: 4096
Unaligned memory access allowed: true
--------------------------------------------------------------------------------
[ JVM information ]
--------------------------------------------------------------------------------
VM Name: OpenJDK 64-Bit Server VM
VM Vendor: Red Hat, Inc.
VM Version: 17.0.9+9-LTS
JIT compiler: HotSpot 64-Bit Tiered Compilers
VM Arguments: [-XX:+UseG1GC, -XX:-OmitStackTraceInFastThrow, -XX:+AlwaysPreTouch, -XX:+UnlockExperimentalVMOptions, -XX:+TrustFinalNonStaticFields, -XX:+DisableExplicitGC, -Djdk.nio.maxCachedBufferSize=1024, -Dio.netty.tryReflectionSetAccessible=true, -Djdk.tls.ephemeralDHKeySize=2048, -Djdk.tls.rejectClientInitiatedRenegotiation=true, -XX:FlightRecorderOptions=stackdepth=256, -XX:+UnlockDiagnosticVMOptions, -XX:+DebugNonSafepoints, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, -Dlog4j2.disable.jmx=true, -Dfile.encoding=UTF-8]
--------------------------------------------------------------------------------
[ Java classpath ]
--------------------------------------------------------------------------------
[classpath] /usr/share/neo4j/lib/jProcesses-1.6.5.jar
[classpath] /usr/share/neo4j/lib/annotations-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-dsl-2023.6.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-javacc-parser-5.14.0.jar
[classpath] /usr/share/neo4j/lib/lucene-analysis-common-9.8.0.jar
[classpath] /usr/share/neo4j/lib/stax-ex-1.8.1.jar
[classpath] /usr/share/neo4j/lib/neo4j-dbms-5.14.0.jar
[classpath] /usr/share/neo4j/lib/eclipse-collections-api-11.1.0.jar
[classpath] /usr/share/neo4j/lib/jaxb-runtime-2.3.2.jar
[classpath] /usr/share/neo4j/lib/neo4j-ssl-5.14.0.jar
[classpath] /usr/share/neo4j/lib/istack-commons-runtime-3.0.8.jar
[classpath] /usr/share/neo4j/lib/neo4j-values-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-5.14.0.jar
[classpath] /usr/share/neo4j/lib/jackson-core-2.15.2.jar
[classpath] /usr/share/neo4j/lib/neo4j-command-line-5.14.0.jar
[classpath] /usr/share/neo4j/lib/WMI4Java-1.6.3.jar
[classpath] /usr/share/neo4j/lib/neo4j-import-util-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-kernel-api-5.14.0.jar
[classpath] /usr/share/neo4j/lib/netty-tcnative-classes-2.0.61.Final.jar
[classpath] /usr/share/neo4j/lib/jakarta.inject-2.6.1.jar
[classpath] /usr/share/neo4j/lib/jetty-xml-10.0.17.jar
[classpath] /usr/share/neo4j/lib/jackson-annotations-2.15.2.jar
[classpath] /usr/share/neo4j/lib/netty-common-4.1.100.Final.jar
[classpath] /usr/share/neo4j/lib/server-api-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-wal-5.14.0.jar
[classpath] /usr/share/neo4j/lib/jetty-servlet-10.0.17.jar
[classpath] /usr/share/neo4j/lib/neo4j-token-api-5.14.0.jar
[classpath] /usr/share/neo4j/lib/ipaddress-5.4.0.jar
[classpath] /usr/share/neo4j/lib/zstd-proxy-5.14.0.jar
[classpath] /usr/share/neo4j/lib/jackson-databind-2.15.2.jar
[classpath] /usr/share/neo4j/lib/slf4j-api-2.0.9.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-macros-5.14.0.jar
[classpath] /usr/share/neo4j/lib/netty-resolver-4.1.100.Final.jar
[classpath] /usr/share/neo4j/lib/jetty-io-10.0.17.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-config-5.14.0.jar
[classpath] /usr/share/neo4j/lib/hk2-utils-2.6.1.jar
[classpath] /usr/share/neo4j/lib/commons-text-1.10.0.jar
[classpath] /usr/share/neo4j/lib/lucene-queryparser-9.8.0.jar
[classpath] /usr/share/neo4j/lib/commons-compress-1.24.0.jar
[classpath] /usr/share/neo4j/lib/shiro-crypto-cipher-1.12.0.jar
[classpath] /usr/share/neo4j/lib/jetty-webapp-10.0.17.jar
[classpath] /usr/share/neo4j/lib/caffeine-3.1.8.jar
[classpath] /usr/share/neo4j/lib/netty-transport-4.1.100.Final.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-physical-planning-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-native-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-slf4j-provider-5.14.0.jar
[classpath] /usr/share/neo4j/lib/shiro-config-ogdl-1.12.0.jar
[classpath] /usr/share/neo4j/lib/jline-terminal-jansi-3.21.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-fulltext-index-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-layout-5.14.0.jar
[classpath] /usr/share/neo4j/lib/netty-transport-classes-kqueue-4.1.100.Final.jar
[classpath] /usr/share/neo4j/lib/neo4j-consistency-check-5.14.0.jar
[classpath] /usr/share/neo4j/lib/jetty-http-10.0.17.jar
[classpath] /usr/share/neo4j/lib/hk2-locator-2.6.1.jar
[classpath] /usr/share/neo4j/lib/log4j-core-2.20.0.jar
[classpath] /usr/share/neo4j/lib/commons-io-2.14.0.jar
[classpath] /usr/share/neo4j/lib/kiama_2.13-2.5.1.jar
[classpath] /usr/share/neo4j/lib/netty-buffer-4.1.100.Final.jar
[classpath] /usr/share/neo4j/lib/neo4j-kernel-5.14.0.jar
[classpath] /usr/share/neo4j/lib/commons-lang3-3.13.0.jar
[classpath] /usr/share/neo4j/lib/jna-5.13.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-procedure-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-unsafe-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-id-generator-5.14.0.jar
[classpath] /usr/share/neo4j/lib/commons-beanutils-1.9.4.jar
[classpath] /usr/share/neo4j/lib/shiro-crypto-hash-1.12.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-runtime-util-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-java-driver-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-lock-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-exceptions-5.14.0.jar
[classpath] /usr/share/neo4j/lib/shiro-cache-1.12.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-spatial-index-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-monitoring-5.14.0.jar
[classpath] /usr/share/neo4j/lib/shiro-core-1.12.0.jar
[classpath] /usr/share/neo4j/lib/jersey-common-2.34.jar
[classpath] /usr/share/neo4j/lib/shiro-crypto-core-1.12.0.jar
[classpath] /usr/share/neo4j/lib/mercator_2.13-0.2.1.jar
[classpath] /usr/share/neo4j/lib/jline-terminal-3.21.0.jar
[classpath] /usr/share/neo4j/lib/commons-logging-1.2.jar
[classpath] /usr/share/neo4j/lib/neo4j-diagnostics-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-rewriting-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-front-end-5.14.0.jar
[classpath] /usr/share/neo4j/lib/netty-transport-native-kqueue-4.1.100.Final-osx-x86_64.jar
[classpath] /usr/share/neo4j/lib/netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar
[classpath] /usr/share/neo4j/lib/netty-transport-native-epoll-4.1.100.Final-linux-aarch_64.jar
[classpath] /usr/share/neo4j/lib/jetty-servlet-api-4.0.6.jar
[classpath] /usr/share/neo4j/lib/neo4j-logging-5.14.0.jar
[classpath] /usr/share/neo4j/lib/hk2-api-2.6.1.jar
[classpath] /usr/share/neo4j/lib/magnolia_2.13-0.17.0.jar
[classpath] /usr/share/neo4j/lib/FastInfoset-1.2.16.jar
[classpath] /usr/share/neo4j/lib/neo4j-storage-engine-util-5.14.0.jar
[classpath] /usr/share/neo4j/lib/jersey-hk2-2.34.jar
[classpath] /usr/share/neo4j/lib/commons-collections-3.2.2.jar
[classpath] /usr/share/neo4j/lib/neo4j-bootcheck-5.14.0.jar
[classpath] /usr/share/neo4j/lib/jakarta.annotation-api-1.3.5.jar
[classpath] /usr/share/neo4j/lib/jersey-server-2.34.jar
[classpath] /usr/share/neo4j/lib/zstd-jni-1.5.5-6.jar
[classpath] /usr/share/neo4j/lib/neo4j-push-to-cloud-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-resource-5.14.0.jar
[classpath] /usr/share/neo4j/lib/commons-configuration2-2.9.0.jar
[classpath] /usr/share/neo4j/lib/jetty-util-10.0.17.jar
[classpath] /usr/share/neo4j/lib/jersey-container-servlet-2.34.jar
[classpath] /usr/share/neo4j/lib/lucene-backward-codecs-9.8.0.jar
[classpath] /usr/share/neo4j/lib/jctools-core-4.0.1.jar
[classpath] /usr/share/neo4j/lib/neo4j-configuration-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-data-collector-5.14.0.jar
[classpath] /usr/share/neo4j/lib/lucene-core-9.8.0.jar
[classpath] /usr/share/neo4j/lib/jline-reader-3.21.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-procedure-api-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-graphdb-api-5.14.0.jar
[classpath] /usr/share/neo4j/lib/netty-transport-native-unix-common-4.1.100.Final.jar
[classpath] /usr/share/neo4j/lib/neo4j-server-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-query-router-5.14.0.jar
[classpath] /usr/share/neo4j/lib/netty-handler-4.1.100.Final.jar
[classpath] /usr/share/neo4j/lib/neo4j-capabilities-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-fabric-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-security-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-cache-5.14.0.jar
[classpath] /usr/share/neo4j/lib/jansi-2.4.0.jar
[classpath] /usr/share/neo4j/lib/jersey-client-2.34.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-ir-5.14.0.jar
[classpath] /usr/share/neo4j/lib/argparse4j-0.9.0.jar
[classpath] /usr/share/neo4j/lib/reactive-streams-1.0.4.jar
[classpath] /usr/share/neo4j/lib/shiro-lang-1.12.0.jar
[classpath] /usr/share/neo4j/lib/reactor-core-3.5.10.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-planner-spi-5.14.0.jar
[classpath] /usr/share/neo4j/lib/cypher-literal-interpreter-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-ast-factory-5.14.0.jar
[classpath] /usr/share/neo4j/lib/log4j-api-2.20.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-record-storage-engine-5.14.0.jar
[classpath] /usr/share/neo4j/lib/picocli-4.7.5.jar
[classpath] /usr/share/neo4j/lib/jetty-security-10.0.17.jar
[classpath] /usr/share/neo4j/lib/neo4j-index-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-ast-5.14.0.jar
[classpath] /usr/share/neo4j/lib/jakarta.validation-api-2.0.2.jar
[classpath] /usr/share/neo4j/lib/jakarta.activation-api-1.2.2.jar
[classpath] /usr/share/neo4j/lib/jettison-1.5.4.jar
[classpath] /var/lib/neo4j/plugins/apoc-5.14.0-core.jar
[classpath] /usr/share/neo4j/lib/log4j-layout-template-json-2.20.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-concurrent-5.14.0.jar
[classpath] /usr/share/neo4j/lib/cypher-ast-factory-5.14.0.jar
[classpath] /usr/share/neo4j/lib/jackson-jaxrs-json-provider-2.15.2.jar
[classpath] /usr/share/neo4j/lib/scala-library-2.13.11.jar
[classpath] /usr/share/neo4j/lib/jakarta.ws.rs-api-2.1.6.jar
[classpath] /usr/share/neo4j/lib/neo4j-collections-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-graph-algo-5.14.0.jar
[classpath] /usr/share/neo4j/lib/shiro-event-1.12.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-import-tool-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-expressions-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-lucene-index-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-planner-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-codegen-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-schema-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-io-5.14.0.jar
[classpath] /usr/share/neo4j/lib/javassist-3.25.0-GA.jar
[classpath] /usr/share/neo4j/lib/netty-codec-http-4.1.100.Final.jar
[classpath] /usr/share/neo4j/lib/neo4j-csv-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-common-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-browser-5.12.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-util-5.14.0.jar
[classpath] /usr/share/neo4j/lib/jackson-jaxrs-base-2.15.2.jar
[classpath] /usr/share/neo4j/lib/jPowerShell-3.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-slotted-runtime-5.14.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-bolt-5.14.0.jar
[classpath] /usr/share/neo4j/lib/netty-transport-native-kqueue-4.1.100.Final-osx-aarch_64.jar
[classpath] /usr/share/neo4j/lib/netty-transport-classes-epoll-4.1.100.Final.jar
[classpath] /usr/share/neo4j/lib/netty-codec-4.1.100.Final.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-interpreted-runtime-5.14.0.jar
[classpath] /usr/share/neo4j/lib/jackson-module-jaxb-annotations-2.15.2.jar
[classpath] /usr/share/neo4j/lib/jakarta.xml.bind-api-2.3.2.jar
[classpath] /usr/share/neo4j/lib/jetty-server-10.0.17.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-expression-evaluator-5.14.0.jar
[classpath] /usr/share/neo4j/lib/jersey-container-servlet-core-2.34.jar
[classpath] /etc/neo4j/*
[classpath] /usr/share/neo4j/lib/txw2-2.3.2.jar
[classpath] /usr/share/neo4j/lib/shiro-config-core-1.12.0.jar
[classpath] /usr/share/neo4j/lib/eclipse-collections-11.1.0.jar
[classpath] /usr/share/neo4j/lib/neo4j-cypher-logical-plans-5.14.0.jar
[classpath] /usr/share/neo4j/lib/scala-collection-contrib_2.13-0.3.0.jar
[classpath] /usr/share/neo4j/lib/scala-reflect-2.13.11.jar
--------------------------------------------------------------------------------
[ Library path ]
--------------------------------------------------------------------------------
/usr/java/packages/lib
/usr/lib64
/usr/lib64
/usr/lib
/usr/lib
--------------------------------------------------------------------------------
[ System properties ]
--------------------------------------------------------------------------------
jdk.tls.rejectClientInitiatedRenegotiation = true
sun.jnu.encoding = UTF-8
sun.arch.data.model = 64
user.timezone = UTC
sun.java.launcher = SUN_STANDARD
user.country = US
sun.boot.library.path = /usr/lib/jvm/java-17-openjdk-17.0.9.0.9-2.el8_8.x86_64/lib
sun.java.command = org.neo4j.server.CommunityEntryPoint --home-dir=/var/lib/neo4j --config-dir=/etc/neo4j
jdk.debug = release
io.netty.tryReflectionSetAccessible = true
sun.cpu.endian = little
user.home = /root
user.language = en
file.separator = /
sun.stdout.encoding = UTF-8
jdk.tls.ephemeralDHKeySize = 2048
sun.management.compiler = HotSpot 64-Bit Tiered Compilers
user.name = root
jdk.nio.maxCachedBufferSize = 1024
path.separator = :
file.encoding = UTF-8
jnidispatch.path = /root/.cache/JNA/temp/jna12570155067950948984.tmp
jna.loaded = true
user.dir = /root
native.encoding = UTF-8
sun.io.unicode.encoding = UnicodeLittle
log4j2.disable.jmx = true
--------------------------------------------------------------------------------
[ (IANA) TimeZone database version ]
--------------------------------------------------------------------------------
TimeZone version: 2023c (available for 603 zone identifiers)
--------------------------------------------------------------------------------
[ Network information ]
--------------------------------------------------------------------------------
Interface eth0:
address: fe80:0:0:0:8470:1747:d191:ff41%eth0
address: 10.60.20.46
Interface lo:
address: 0:0:0:0:0:0:0:1%lo
address: 127.0.0.1
--------------------------------------------------------------------------------
[ Native access information ]
--------------------------------------------------------------------------------
Native access details: Linux native access is available.
--------------------------------------------------------------------------------
[ DBMS config ]
--------------------------------------------------------------------------------
DBMS provided settings:
db.tx_log.rotation.retention_policy=2 days 2G
server.bolt.advertised_address=10.60.20.46:7687
server.bolt.enabled=true
server.bolt.listen_address=0.0.0.0:7687
server.bolt.tls_level=DISABLED
server.directories.data=/data/neo4j/data
server.directories.import=/var/lib/neo4j/import
server.directories.lib=/usr/share/neo4j/lib
server.directories.logs=/var/log/neo4j
server.directories.neo4j_home=/var/lib/neo4j
server.directories.plugins=/var/lib/neo4j/plugins
server.http.advertised_address=10.60.20.46:7474
server.http.enabled=true
server.http.listen_address=0.0.0.0:7474
server.https.enabled=false
server.jvm.additional=-XX:+UseG1GC
-XX:-OmitStackTraceInFastThrow
-XX:+AlwaysPreTouch
-XX:+UnlockExperimentalVMOptions
-XX:+TrustFinalNonStaticFields
-XX:+DisableExplicitGC
-Djdk.nio.maxCachedBufferSize=1024
-Dio.netty.tryReflectionSetAccessible=true
-Djdk.tls.ephemeralDHKeySize=2048
-Djdk.tls.rejectClientInitiatedRenegotiation=true
-XX:FlightRecorderOptions=stackdepth=256
-XX:+UnlockDiagnosticVMOptions
-XX:+DebugNonSafepoints
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
-Dlog4j2.disable.jmx=true
server.logs.config=/etc/neo4j/server-logs.xml
server.logs.user.config=/etc/neo4j/user-logs.xml
server.windows_service_name=neo4j
Directories in use:
server.directories.data=/data/neo4j/data
server.directories.dumps.root=/data/neo4j/data/dumps
server.directories.import=/var/lib/neo4j/import
server.directories.lib=/usr/share/neo4j/lib
server.directories.licenses=/var/lib/neo4j/licenses
server.directories.logs=/var/log/neo4j
server.directories.neo4j_home=/var/lib/neo4j
server.directories.plugins=/var/lib/neo4j/plugins
server.directories.run=/var/lib/neo4j/run
server.directories.script.root=/data/neo4j/data/scripts
server.directories.transaction.logs.root=/data/neo4j/data/transactions
--------------------------------------------------------------------------------
[ Packaging ]
--------------------------------------------------------------------------------
Could not read packaging info: java.nio.file.NoSuchFileException: /var/lib/neo4j/packaging_info
2024-01-08 02:18:22.240+0000 INFO [o.n.d.i.DefaultIdentityModule] Found ServerId on disk: ServerId{74a11739} (74a11739-2d17-42b4-b8c8-b98429075eb0)
2024-01-08 02:18:22.242+0000 INFO [o.n.d.i.DefaultIdentityModule] This instance is ServerId{74a11739} (74a11739-2d17-42b4-b8c8-b98429075eb0)
2024-01-08 02:18:23.493+0000 INFO [o.n.s.CommunityNeoWebServer] ======== Neo4j 5.14.0 ========
2024-01-08 02:18:23.519+0000 INFO [o.n.d.d.DatabaseLifecycles] Creating 'DatabaseId{00000000[system]}'.
2024-01-08 02:18:23.605+0000 INFO [a.ApocConfig] successfully registered ApocConfig for @Context
2024-01-08 02:18:23.609+0000 INFO [a.Pools] successfully registered Pools for @Context
Reproduce steps
Just installing the Neo4j database on the machine, loaded data via python driver and performed queries via driver or cypher.
Expected behavior: no database crash
Actual behavior
The problem has recurred 3 times during the past month, with interval of approximately 5-7 days. The downtime lasted for various of time, from hours to days with below observations:
continuous close to 100% CPU utilization
continuous disk throughput of steady 240MiB/s on pure reading (currently disk data is around 7GiB only)
during the down time, occasional logs of Detected VM stop-the-world pause: {pauseTime=245207, gcTime=0, gcCount=0} in debug.log
During the period when CPU peeks to 100%, I can't event ssh to the instance. The only way to recover was to restart the host machine. In the latest crash, though CPU recovered after 11 hours, database is still unavailable though neo4j status shows running status.
I referred to the closed issue on Detected VM stop-the-world pause due to limited JVM memory. But this does resolve my problem on the continuous massive I/O operation and high CPU utilization. Any hints on what caused the situation without external heavy load of write/read operations?
The text was updated successfully, but these errors were encountered:
024-01-08 02:18:21.925+0000 WARN [o.n.k.i.JvmChecker] The max heap memory has not been configured. It is recommended that it is always explicitly configured, to ensure the system has a balanced configuration. Until then, a JVM computed heuristic of 958398464 bytes is used instead. If you are running neo4j server, you need to configure server.memory.heap.max_size in neo4j.conf. If you are running neo4j embedded, you have to launch the JVM with -Xmx set to a value. You can run neo4j-admin server memory-recommendation for memory configuration suggestions.
2024-01-08 02:18:21.926+0000 WARN [o.n.k.i.JvmChecker] The initial heap memory has not been configured. It is recommended that it is always explicitly configured, to ensure the system has a balanced configuration. Until then, a JVM computed heuristic of 60817408 bytes is used instead. If you are running neo4j server, you need to configure server.memory.heap.initial_size in neo4j.conf. If you are running neo4j embedded, you have to launch the JVM with -Xms set to a value. You can run neo4j-admin server memory-recommendation for memory configuration suggestions.
2024-01-08 02:18:22.039+0000 WARN [o.n.i.p.PageCache] The server.memory.pagecache.size setting has not been configured. It is recommended that this setting is always explicitly configured, to ensure the system has a balanced configuration. Until then, a computed heuristic value of 1434413056 bytes will be used instead. Run `neo4j-admin memory-recommendation` for memory configuration suggestions.
Please follow the Neo4j documentation about memory configuration. The lack of heap space is likely causing the GC pauses
Hey, I'm currently faced with a problem of database crash with very high CPU load and abnormal disk I/O multiple times, during which process observed
Detected VM stop-the-world pause
.Configurations
Example bug report
Reproduce steps
Just installing the Neo4j database on the machine, loaded data via python driver and performed queries via driver or cypher.
Expected behavior: no database crash
Actual behavior
The problem has recurred 3 times during the past month, with interval of approximately 5-7 days. The downtime lasted for various of time, from hours to days with below observations:
Detected VM stop-the-world pause: {pauseTime=245207, gcTime=0, gcCount=0}
in debug.logDuring the period when CPU peeks to 100%, I can't event ssh to the instance. The only way to recover was to restart the host machine. In the latest crash, though CPU recovered after 11 hours, database is still unavailable though
neo4j status
shows running status.I referred to the closed issue on
Detected VM stop-the-world pause
due to limited JVM memory. But this does resolve my problem on the continuous massive I/O operation and high CPU utilization. Any hints on what caused the situation without external heavy load of write/read operations?The text was updated successfully, but these errors were encountered: