Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Observed high CPU and disk I/O read with database unavailable #13374

Open
amanda-tian-awx opened this issue Jan 8, 2024 · 2 comments
Open

Observed high CPU and disk I/O read with database unavailable #13374

amanda-tian-awx opened this issue Jan 8, 2024 · 2 comments
Labels

Comments

@amanda-tian-awx
Copy link

Hey, I'm currently faced with a problem of database crash with very high CPU load and abnormal disk I/O multiple times, during which process observed Detected VM stop-the-world pause.

Configurations

  • Neo4j version: community 5.14.0
  • Operating system: rocky linux 8 (4vCPU, 4GiB memory, 100GiB storage) on GCP
  • API: Python driver v5.1.3 and Cypher

Example bug report

2024-01-06 04:15:13.816+0000 INFO  [o.n.k.i.t.l.p.LogPruningImpl] [neo4j/344d7f25] No log version pruned. The strategy used was '2 days 2147483648 size'. 
2024-01-06 05:30:41.731+0000 WARN  [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=859, gcTime=0, gcCount=0}
2024-01-08 02:18:21.729+0000 INFO  [o.n.g.f.m.GlobalModule] Logging config in use: File '/etc/neo4j/server-logs.xml'
2024-01-08 02:18:21.925+0000 WARN  [o.n.k.i.JvmChecker] The max heap memory has not been configured. It is recommended that it is always explicitly configured, to ensure the system has a balanced configuration. Until then, a JVM computed heuristic of 958398464 bytes is used instead. If you are running neo4j server, you need to configure server.memory.heap.max_size in neo4j.conf. If you are running neo4j embedded, you have to launch the JVM with -Xmx set to a value. You can run neo4j-admin server memory-recommendation for memory configuration suggestions.
2024-01-08 02:18:21.926+0000 WARN  [o.n.k.i.JvmChecker] The initial heap memory has not been configured. It is recommended that it is always explicitly configured, to ensure the system has a balanced configuration. Until then, a JVM computed heuristic of 60817408 bytes is used instead. If you are running neo4j server, you need to configure server.memory.heap.initial_size in neo4j.conf. If you are running neo4j embedded, you have to launch the JVM with -Xms set to a value. You can run neo4j-admin server memory-recommendation for memory configuration suggestions.
2024-01-08 02:18:22.039+0000 WARN  [o.n.i.p.PageCache] The server.memory.pagecache.size setting has not been configured. It is recommended that this setting is always explicitly configured, to ensure the system has a balanced configuration. Until then, a computed heuristic value of 1434413056 bytes will be used instead. Run `neo4j-admin memory-recommendation` for memory configuration suggestions.
2024-01-08 02:18:22.192+0000 INFO  [o.n.i.d.DiagnosticsManager] 
                                                                ********************************************************************************
                                                                                             [ System diagnostics ]                             
                                                                ********************************************************************************
                                                                --------------------------------------------------------------------------------
                                                                                         [ System memory information ]                          
                                                                --------------------------------------------------------------------------------
                                                                Total Physical memory: 3.564GiB
                                                                Free Physical memory: 1.571GiB
                                                                Committed virtual memory: 4.138GiB
                                                                Total swap space: 0B
                                                                Free swap space: 0B
                                                                
                                                                --------------------------------------------------------------------------------
                                                                                           [ JVM memory information ]                           
                                                                --------------------------------------------------------------------------------
                                                                Free  memory: 76.19MiB
                                                                Total memory: 107.0MiB
                                                                Max   memory: 914.0MiB
                                                                Garbage Collector: G1 Young Generation: [G1 Eden Space, G1 Survivor Space, G1 Old Gen]
                                                                Garbage Collector: G1 Old Generation: [G1 Eden Space, G1 Survivor Space, G1 Old Gen]
                                                                Memory Pool: CodeHeap 'non-nmethods' (Non-heap memory): committed=2.438MiB, used=1.234MiB, max=5.566MiB, threshold=0B
                                                                Memory Pool: Metaspace (Non-heap memory): committed=21.81MiB, used=21.55MiB, max=-1B, threshold=0B
                                                                Memory Pool: CodeHeap 'profiled nmethods' (Non-heap memory): committed=5.500MiB, used=5.445MiB, max=117.2MiB, threshold=0B
                                                                Memory Pool: Compressed Class Space (Non-heap memory): committed=2.438MiB, used=2.327MiB, max=1.000GiB, threshold=0B
                                                                Memory Pool: G1 Eden Space (Heap memory): committed=75.00MiB, used=3.000MiB, max=-1B, threshold=?
                                                                Memory Pool: G1 Old Gen (Heap memory): committed=29.00MiB, used=25.09MiB, max=914.0MiB, threshold=0B
                                                                Memory Pool: G1 Survivor Space (Heap memory): committed=3.000MiB, used=2.208MiB, max=-1B, threshold=?
                                                                Memory Pool: CodeHeap 'non-profiled nmethods' (Non-heap memory): committed=2.438MiB, used=1.488MiB, max=117.2MiB, threshold=0B
                                                                
                                                                --------------------------------------------------------------------------------
                                                                                        [ Operating system information ]                        
                                                                --------------------------------------------------------------------------------
                                                                Operating System: Linux; version: 4.18.0-513.5.1.el8_9.x86_64; arch: amd64; cpus: 4
                                                                Max number of file descriptors: 65535
                                                                Number of open file descriptors: 192
                                                                Process id: 13274
                                                                Byte order: LITTLE_ENDIAN
                                                                Local timezone: UTC
                                                                Memory page size: 4096
                                                                Unaligned memory access allowed: true
                                                                
                                                                --------------------------------------------------------------------------------
                                                                                              [ JVM information ]                               
                                                                --------------------------------------------------------------------------------
                                                                VM Name: OpenJDK 64-Bit Server VM
                                                                VM Vendor: Red Hat, Inc.
                                                                VM Version: 17.0.9+9-LTS
                                                                JIT compiler: HotSpot 64-Bit Tiered Compilers
                                                                VM Arguments: [-XX:+UseG1GC, -XX:-OmitStackTraceInFastThrow, -XX:+AlwaysPreTouch, -XX:+UnlockExperimentalVMOptions, -XX:+TrustFinalNonStaticFields, -XX:+DisableExplicitGC, -Djdk.nio.maxCachedBufferSize=1024, -Dio.netty.tryReflectionSetAccessible=true, -Djdk.tls.ephemeralDHKeySize=2048, -Djdk.tls.rejectClientInitiatedRenegotiation=true, -XX:FlightRecorderOptions=stackdepth=256, -XX:+UnlockDiagnosticVMOptions, -XX:+DebugNonSafepoints, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, -Dlog4j2.disable.jmx=true, -Dfile.encoding=UTF-8]
                                                                
                                                                --------------------------------------------------------------------------------
                                                                                               [ Java classpath ]                               
                                                                --------------------------------------------------------------------------------
                                                                 [classpath] /usr/share/neo4j/lib/jProcesses-1.6.5.jar
                                                                 [classpath] /usr/share/neo4j/lib/annotations-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-dsl-2023.6.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-javacc-parser-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/lucene-analysis-common-9.8.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/stax-ex-1.8.1.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-dbms-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/eclipse-collections-api-11.1.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jaxb-runtime-2.3.2.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-ssl-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/istack-commons-runtime-3.0.8.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-values-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jackson-core-2.15.2.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-command-line-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/WMI4Java-1.6.3.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-import-util-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-kernel-api-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-tcnative-classes-2.0.61.Final.jar
                                                                 [classpath] /usr/share/neo4j/lib/jakarta.inject-2.6.1.jar
                                                                 [classpath] /usr/share/neo4j/lib/jetty-xml-10.0.17.jar
                                                                 [classpath] /usr/share/neo4j/lib/jackson-annotations-2.15.2.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-common-4.1.100.Final.jar
                                                                 [classpath] /usr/share/neo4j/lib/server-api-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-wal-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jetty-servlet-10.0.17.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-token-api-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/ipaddress-5.4.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/zstd-proxy-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jackson-databind-2.15.2.jar
                                                                 [classpath] /usr/share/neo4j/lib/slf4j-api-2.0.9.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-macros-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-resolver-4.1.100.Final.jar
                                                                 [classpath] /usr/share/neo4j/lib/jetty-io-10.0.17.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-config-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/hk2-utils-2.6.1.jar
                                                                 [classpath] /usr/share/neo4j/lib/commons-text-1.10.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/lucene-queryparser-9.8.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/commons-compress-1.24.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/shiro-crypto-cipher-1.12.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jetty-webapp-10.0.17.jar
                                                                 [classpath] /usr/share/neo4j/lib/caffeine-3.1.8.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-transport-4.1.100.Final.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-physical-planning-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-native-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-slf4j-provider-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/shiro-config-ogdl-1.12.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jline-terminal-jansi-3.21.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-fulltext-index-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-layout-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-transport-classes-kqueue-4.1.100.Final.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-consistency-check-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jetty-http-10.0.17.jar
                                                                 [classpath] /usr/share/neo4j/lib/hk2-locator-2.6.1.jar
                                                                 [classpath] /usr/share/neo4j/lib/log4j-core-2.20.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/commons-io-2.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/kiama_2.13-2.5.1.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-buffer-4.1.100.Final.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-kernel-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/commons-lang3-3.13.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jna-5.13.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-procedure-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-unsafe-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-id-generator-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/commons-beanutils-1.9.4.jar
                                                                 [classpath] /usr/share/neo4j/lib/shiro-crypto-hash-1.12.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-runtime-util-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-java-driver-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-lock-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-exceptions-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/shiro-cache-1.12.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-spatial-index-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-monitoring-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/shiro-core-1.12.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jersey-common-2.34.jar
                                                                 [classpath] /usr/share/neo4j/lib/shiro-crypto-core-1.12.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/mercator_2.13-0.2.1.jar
                                                                 [classpath] /usr/share/neo4j/lib/jline-terminal-3.21.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/commons-logging-1.2.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-diagnostics-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-rewriting-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-front-end-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-transport-native-kqueue-4.1.100.Final-osx-x86_64.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-transport-native-epoll-4.1.100.Final-linux-aarch_64.jar
                                                                 [classpath] /usr/share/neo4j/lib/jetty-servlet-api-4.0.6.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-logging-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/hk2-api-2.6.1.jar
                                                                 [classpath] /usr/share/neo4j/lib/magnolia_2.13-0.17.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/FastInfoset-1.2.16.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-storage-engine-util-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jersey-hk2-2.34.jar
                                                                 [classpath] /usr/share/neo4j/lib/commons-collections-3.2.2.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-bootcheck-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jakarta.annotation-api-1.3.5.jar
                                                                 [classpath] /usr/share/neo4j/lib/jersey-server-2.34.jar
                                                                 [classpath] /usr/share/neo4j/lib/zstd-jni-1.5.5-6.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-push-to-cloud-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-resource-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/commons-configuration2-2.9.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jetty-util-10.0.17.jar
                                                                 [classpath] /usr/share/neo4j/lib/jersey-container-servlet-2.34.jar
                                                                 [classpath] /usr/share/neo4j/lib/lucene-backward-codecs-9.8.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jctools-core-4.0.1.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-configuration-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-data-collector-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/lucene-core-9.8.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jline-reader-3.21.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-procedure-api-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-graphdb-api-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-transport-native-unix-common-4.1.100.Final.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-server-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-query-router-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-handler-4.1.100.Final.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-capabilities-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-fabric-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-security-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-cache-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jansi-2.4.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jersey-client-2.34.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-ir-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/argparse4j-0.9.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/reactive-streams-1.0.4.jar
                                                                 [classpath] /usr/share/neo4j/lib/shiro-lang-1.12.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/reactor-core-3.5.10.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-planner-spi-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/cypher-literal-interpreter-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-ast-factory-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/log4j-api-2.20.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-record-storage-engine-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/picocli-4.7.5.jar
                                                                 [classpath] /usr/share/neo4j/lib/jetty-security-10.0.17.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-index-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-ast-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jakarta.validation-api-2.0.2.jar
                                                                 [classpath] /usr/share/neo4j/lib/jakarta.activation-api-1.2.2.jar
                                                                 [classpath] /usr/share/neo4j/lib/jettison-1.5.4.jar
                                                                 [classpath] /var/lib/neo4j/plugins/apoc-5.14.0-core.jar
                                                                 [classpath] /usr/share/neo4j/lib/log4j-layout-template-json-2.20.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-concurrent-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/cypher-ast-factory-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jackson-jaxrs-json-provider-2.15.2.jar
                                                                 [classpath] /usr/share/neo4j/lib/scala-library-2.13.11.jar
                                                                 [classpath] /usr/share/neo4j/lib/jakarta.ws.rs-api-2.1.6.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-collections-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-graph-algo-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/shiro-event-1.12.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-import-tool-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-expressions-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-lucene-index-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-planner-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-codegen-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-schema-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-io-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/javassist-3.25.0-GA.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-codec-http-4.1.100.Final.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-csv-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-common-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-browser-5.12.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-util-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jackson-jaxrs-base-2.15.2.jar
                                                                 [classpath] /usr/share/neo4j/lib/jPowerShell-3.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-slotted-runtime-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-bolt-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-transport-native-kqueue-4.1.100.Final-osx-aarch_64.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-transport-classes-epoll-4.1.100.Final.jar
                                                                 [classpath] /usr/share/neo4j/lib/netty-codec-4.1.100.Final.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-interpreted-runtime-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jackson-module-jaxb-annotations-2.15.2.jar
                                                                 [classpath] /usr/share/neo4j/lib/jakarta.xml.bind-api-2.3.2.jar
                                                                 [classpath] /usr/share/neo4j/lib/jetty-server-10.0.17.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-expression-evaluator-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/jersey-container-servlet-core-2.34.jar
                                                                 [classpath] /etc/neo4j/*
                                                                 [classpath] /usr/share/neo4j/lib/txw2-2.3.2.jar
                                                                 [classpath] /usr/share/neo4j/lib/shiro-config-core-1.12.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/eclipse-collections-11.1.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/neo4j-cypher-logical-plans-5.14.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/scala-collection-contrib_2.13-0.3.0.jar
                                                                 [classpath] /usr/share/neo4j/lib/scala-reflect-2.13.11.jar
                                                                
                                                                --------------------------------------------------------------------------------
                                                                                                [ Library path ]                                
                                                                --------------------------------------------------------------------------------
                                                                /usr/java/packages/lib
                                                                /usr/lib64
                                                                /usr/lib64
                                                                /usr/lib
                                                                /usr/lib
                                                                
                                                                --------------------------------------------------------------------------------
                                                                                             [ System properties ]                              
                                                                --------------------------------------------------------------------------------
                                                                jdk.tls.rejectClientInitiatedRenegotiation = true
                                                                sun.jnu.encoding = UTF-8
                                                                sun.arch.data.model = 64
                                                                user.timezone = UTC
                                                                sun.java.launcher = SUN_STANDARD
                                                                user.country = US
                                                                sun.boot.library.path = /usr/lib/jvm/java-17-openjdk-17.0.9.0.9-2.el8_8.x86_64/lib
                                                                sun.java.command = org.neo4j.server.CommunityEntryPoint --home-dir=/var/lib/neo4j --config-dir=/etc/neo4j
                                                                jdk.debug = release
                                                                io.netty.tryReflectionSetAccessible = true
                                                                sun.cpu.endian = little
                                                                user.home = /root
                                                                user.language = en
                                                                file.separator = /
                                                                sun.stdout.encoding = UTF-8
                                                                jdk.tls.ephemeralDHKeySize = 2048
                                                                sun.management.compiler = HotSpot 64-Bit Tiered Compilers
                                                                user.name = root
                                                                jdk.nio.maxCachedBufferSize = 1024
                                                                path.separator = :
                                                                file.encoding = UTF-8
                                                                jnidispatch.path = /root/.cache/JNA/temp/jna12570155067950948984.tmp
                                                                jna.loaded = true
                                                                user.dir = /root
                                                                native.encoding = UTF-8
                                                                sun.io.unicode.encoding = UnicodeLittle
                                                                log4j2.disable.jmx = true
                                                                
                                                                --------------------------------------------------------------------------------
                                                                                      [ (IANA) TimeZone database version ]                      
                                                                --------------------------------------------------------------------------------
                                                                  TimeZone version: 2023c (available for 603 zone identifiers)
                                                                
                                                                --------------------------------------------------------------------------------
                                                                                            [ Network information ]                             
                                                                --------------------------------------------------------------------------------
                                                                Interface eth0:
                                                                    address: fe80:0:0:0:8470:1747:d191:ff41%eth0
                                                                    address: 10.60.20.46
                                                                Interface lo:
                                                                    address: 0:0:0:0:0:0:0:1%lo
                                                                    address: 127.0.0.1
                                                                
                                                                --------------------------------------------------------------------------------
                                                                                         [ Native access information ]                          
                                                                --------------------------------------------------------------------------------
                                                                Native access details: Linux native access is available.
                                                                
                                                                --------------------------------------------------------------------------------
                                                                                                [ DBMS config ]                                 
                                                                --------------------------------------------------------------------------------
                                                                DBMS provided settings:
                                                                db.tx_log.rotation.retention_policy=2 days 2G
                                                                server.bolt.advertised_address=10.60.20.46:7687
                                                                server.bolt.enabled=true
                                                                server.bolt.listen_address=0.0.0.0:7687
                                                                server.bolt.tls_level=DISABLED
                                                                server.directories.data=/data/neo4j/data
                                                                server.directories.import=/var/lib/neo4j/import
                                                                server.directories.lib=/usr/share/neo4j/lib
                                                                server.directories.logs=/var/log/neo4j
                                                                server.directories.neo4j_home=/var/lib/neo4j
                                                                server.directories.plugins=/var/lib/neo4j/plugins
                                                                server.http.advertised_address=10.60.20.46:7474
                                                                server.http.enabled=true
                                                                server.http.listen_address=0.0.0.0:7474
                                                                server.https.enabled=false
                                                                server.jvm.additional=-XX:+UseG1GC
-XX:-OmitStackTraceInFastThrow
-XX:+AlwaysPreTouch
-XX:+UnlockExperimentalVMOptions
-XX:+TrustFinalNonStaticFields
-XX:+DisableExplicitGC
-Djdk.nio.maxCachedBufferSize=1024
-Dio.netty.tryReflectionSetAccessible=true
-Djdk.tls.ephemeralDHKeySize=2048
-Djdk.tls.rejectClientInitiatedRenegotiation=true
-XX:FlightRecorderOptions=stackdepth=256
-XX:+UnlockDiagnosticVMOptions
-XX:+DebugNonSafepoints
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
-Dlog4j2.disable.jmx=true
                                                                server.logs.config=/etc/neo4j/server-logs.xml
                                                                server.logs.user.config=/etc/neo4j/user-logs.xml
                                                                server.windows_service_name=neo4j
                                                                Directories in use:
                                                                server.directories.data=/data/neo4j/data
                                                                server.directories.dumps.root=/data/neo4j/data/dumps
                                                                server.directories.import=/var/lib/neo4j/import
                                                                server.directories.lib=/usr/share/neo4j/lib
                                                                server.directories.licenses=/var/lib/neo4j/licenses
                                                                server.directories.logs=/var/log/neo4j
                                                                server.directories.neo4j_home=/var/lib/neo4j
                                                                server.directories.plugins=/var/lib/neo4j/plugins
                                                                server.directories.run=/var/lib/neo4j/run
                                                                server.directories.script.root=/data/neo4j/data/scripts
                                                                server.directories.transaction.logs.root=/data/neo4j/data/transactions
                                                                
                                                                --------------------------------------------------------------------------------
                                                                                                 [ Packaging ]                                  
                                                                --------------------------------------------------------------------------------
                                                                Could not read packaging info: java.nio.file.NoSuchFileException: /var/lib/neo4j/packaging_info
                                                                
2024-01-08 02:18:22.240+0000 INFO  [o.n.d.i.DefaultIdentityModule] Found ServerId on disk: ServerId{74a11739} (74a11739-2d17-42b4-b8c8-b98429075eb0)
2024-01-08 02:18:22.242+0000 INFO  [o.n.d.i.DefaultIdentityModule] This instance is ServerId{74a11739} (74a11739-2d17-42b4-b8c8-b98429075eb0)
2024-01-08 02:18:23.493+0000 INFO  [o.n.s.CommunityNeoWebServer] ======== Neo4j 5.14.0 ========
2024-01-08 02:18:23.519+0000 INFO  [o.n.d.d.DatabaseLifecycles] Creating 'DatabaseId{00000000[system]}'.
2024-01-08 02:18:23.605+0000 INFO  [a.ApocConfig] successfully registered ApocConfig for @Context
2024-01-08 02:18:23.609+0000 INFO  [a.Pools] successfully registered Pools for @Context

Reproduce steps
Just installing the Neo4j database on the machine, loaded data via python driver and performed queries via driver or cypher.

Expected behavior: no database crash

Actual behavior
The problem has recurred 3 times during the past month, with interval of approximately 5-7 days. The downtime lasted for various of time, from hours to days with below observations:

  • continuous close to 100% CPU utilization
  • continuous disk throughput of steady 240MiB/s on pure reading (currently disk data is around 7GiB only)
  • during the down time, occasional logs of Detected VM stop-the-world pause: {pauseTime=245207, gcTime=0, gcCount=0} in debug.log

During the period when CPU peeks to 100%, I can't event ssh to the instance. The only way to recover was to restart the host machine. In the latest crash, though CPU recovered after 11 hours, database is still unavailable though neo4j status shows running status.

I referred to the closed issue on Detected VM stop-the-world pause due to limited JVM memory. But this does resolve my problem on the continuous massive I/O operation and high CPU utilization. Any hints on what caused the situation without external heavy load of write/read operations?

@ma3s7ro
Copy link

ma3s7ro commented Feb 28, 2024

@amanda-tian-awx Do you find any solution with disk and CPU usage on GCP + Rocky Linux?

@ojhughes
Copy link

This log entry gives you a hint

024-01-08 02:18:21.925+0000 WARN  [o.n.k.i.JvmChecker] The max heap memory has not been configured. It is recommended that it is always explicitly configured, to ensure the system has a balanced configuration. Until then, a JVM computed heuristic of 958398464 bytes is used instead. If you are running neo4j server, you need to configure server.memory.heap.max_size in neo4j.conf. If you are running neo4j embedded, you have to launch the JVM with -Xmx set to a value. You can run neo4j-admin server memory-recommendation for memory configuration suggestions.
2024-01-08 02:18:21.926+0000 WARN  [o.n.k.i.JvmChecker] The initial heap memory has not been configured. It is recommended that it is always explicitly configured, to ensure the system has a balanced configuration. Until then, a JVM computed heuristic of 60817408 bytes is used instead. If you are running neo4j server, you need to configure server.memory.heap.initial_size in neo4j.conf. If you are running neo4j embedded, you have to launch the JVM with -Xms set to a value. You can run neo4j-admin server memory-recommendation for memory configuration suggestions.
2024-01-08 02:18:22.039+0000 WARN  [o.n.i.p.PageCache] The server.memory.pagecache.size setting has not been configured. It is recommended that this setting is always explicitly configured, to ensure the system has a balanced configuration. Until then, a computed heuristic value of 1434413056 bytes will be used instead. Run `neo4j-admin memory-recommendation` for memory configuration suggestions.

Please follow the Neo4j documentation about memory configuration. The lack of heap space is likely causing the GC pauses

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants