运行Cassandra 4.0.8时Java崩溃

kwvwclae  于 2023-03-16  发布在  Java
关注(0)|答案(1)|浏览(169)

I'm running Cassandra 4.0.8 on CentOS Stream release 8, using Java 11.0.17. When I cleanup after a bootstrap some of my nodes crash. Basically it seems to crash when streaming and compacting a lot of data.
I believe to be using the latest Java 11 VM and the latest Cassandra 4 versions.
Would anyone have any suggestions on how to fix this problem. I cannot reproduce the problem on demand, it happens sometimes when there is lots to do.!!
Thanks for your help,
Jean
Here is an extract of the core dump:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007fe388431daa, pid=3006650, tid=3008039
#
# JRE version: Java(TM) SE Runtime Environment 18.9 (11.0.17+10) (build 11.0.17+10-LTS-269)
# Java VM: Java HotSpot(TM) 64-Bit Server VM 18.9 (11.0.17+10-LTS-269, mixed mode, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
# Problematic frame:
# v  ~StubRoutines::updateBytesCRC32
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to /home/cassandra/core.3006650)
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -ea -da:net.openhft... -XX:+UseThreadPriorities -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+AlwaysPreTouch -XX:-UseBiasedLocking -XX:+UseTLAB -XX:+ResizeTLAB -XX:+UseNUMA -XX:+PerfDisableSharedMem -Djava.net.

preferIPv4Stack=true -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSWaitDuration=10000 -XX :+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways -XX:+CMSClassUnloadingEnabled -Djdk.attach.allowAttachSelf=true --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=AL L-UNNAMED --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add- exports=java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.server =ALL-UNNAMED --add-exports=java.sql/java.sql=ALL-UNNAMED --add-opens=java.base/java.lang.module=ALL-UNNAMED --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens =java.base/jdk.internal.reflect=ALL-UNNAMED --add-opens=java.base/jdk.internal.math=ALL-UNNAMED --add-opens=java.base/jdk.internal.module=ALL-UNNAMED --add-opens=java.base/jdk.internal.util.jar=ALL-UNNAMED --add-opens=jdk.ma nagement/com.sun.management.internal=ALL-UNNAMED -Dio.netty.tryReflectionSetAccessible=true -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/home/cassandra/apache-cassandra/logs/gc.log:time,uptime,p id,tid,level:filecount=10,filesize=10485760 -Xms7961M -Xmx7961M -Xmn400M -XX:+UseCondCardMark -XX:CompileCommandFile=/home/cassandra/apache-cassandra/conf/hotspot_compiler -javaagent:/home/cassandra/apache-cassandra/lib/jamm -0.3.2.jar -Dcassandra.jmx.remote.port=7199 -Dcom.sun.management.jmxremote.rmi.port=7199 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password -Djava .library.path=/home/cassandra/apache-cassandra/lib/sigar-bin -Dcassandra.libjemalloc=/usr/lib64/libjemalloc.so.2 -XX:OnOutOfMemoryError=kill -9 %p -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/home/cassandra/ap ache-cassandra/logs -Dcassandra.storagedir=/home/cassandra/apache-cassandra/data org.apache.cassandra.service.CassandraDaemon

Host: Intel(R) Core(TM) i5-6260U CPU @ 1.80GHz, 4 cores, 31G, CentOS Stream release 8
Time: Tue Mar 14 15:01:44 2023 CET elapsed time: 2430.690820 seconds (0d 0h 40m 30s)

---------------  T H R E A D  ---------------

Current thread (0x00007f2b75aaee00):  JavaThread "CompactionExecutor:8" daemon [_thread_in_Java, id=3008039, stack(0x00007f25d5ddf000,0x00007f25d5e20000)]

Stack: [0x00007f25d5ddf000,0x00007f25d5e20000],  sp=0x00007f25d5e1c6f0,  free space=245k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

v ~StubRoutines::updateBytesCRC32

siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr: 0x00007f4556a2d000

Register to memory mapping:

RAX=0x00000000d8464a00 is an unknown value
RBX=0x000000061f4c1c90 is an oop: java.nio.DirectByteBufferR
{0x000000061f4c1c90} - klass: 'java/nio/DirectByteBufferR'
RCX=0x00007fe3a138e9c0: <offset 0x000000000146f9c0> in /usr/jdk-    11.0.17/lib/server/libjvm.so at 0x00007fe39ff1f000
RDX=0x0000000000000082 is an unknown value
RSP=0x00007f25d5e1c6f0 is pointing into the stack for thread: 0x00007f2b75aaee00
RBP=0x00007f25d5e1c6f0 is pointing into the stack for thread: 0x00007f2b75aaee00
RSI=0x00007f4556a2cfd0 points into unknown readable memory: 0x4f1f18b0cedc8a28 | 28 8a dc ce b0 18 1f 4f
RDI=0x00000000d18d1b56 is an unknown value
R8 =0x0000000029edee07 is an unknown value
R9 =0x0000000000000a19 is an unknown value
eyh26e7m

eyh26e7m1#

与LHWizard所说的相呼应,在查看您的问题时,您的堆大小看起来略小于8GB,这意味着您得到的是默认的计算大小,这可能不够。
由于您似乎正在使用CMS垃圾收集器,堆大小(在conf/jvm-server.options文件中指定)应该如下所示:

-Xmx16GB
-Xms16GB
-Xmn6GB

Xmx是最大堆大小。Xms是初始堆大小,但是由于调整堆大小会影响性能,所以使用Cassandra时,您通常希望将最大堆大小和初始堆大小设置为相同。同样,使用Cassandra和CMS时,您应该将堆的25%到50%作为新一代堆大小的目标,这就是我使用Xmn6GB的原因。
请注意,您应该在jvm11-server.options中研究G1垃圾收集器的配置选项。CMS收集器在Java 9中被弃用,在Java 14中被删除。如果您决定走这条路,您只需要担心XmxXms(您不应该使用G1GC设置新的代大小)。

相关问题