[Cassandra] 记一次Cassandra集群中一个节点CPU满载问题排查过程

悠悠 2022-02-20 06:57 600阅读 0赞

现象：Cassandra集群中一个节点CPU满载，Cassandra连接超时报错。该服务器CPU被Cassandra吃完。

排查过程：

1. top命令查看CPU占用最高的进程，确认为Cassandra

[root@VM_centos ~]# top
    20772 cassand+  20   0 8299168 4.484g 126280 S 376.6 58.7  23:49.92 java

2. jstack -l PID命令查看线程活动情况

[cassandra@VM_129_3_centos tmp]$ jstack -l 20772
    2019-04-12 20:29:23
    Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.172-b11 mixed mode):
    
    "SharedPool-Worker-134" #396 daemon prio=5 os_prio=0 tid=0x00007f907104c1d0 nid=0x58f3 waiting on condition [0x00007f8fbd6b3000]
       java.lang.Thread.State: WAITING (parking)
    	at sun.misc.Unsafe.park(Native Method)
    	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
    	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:87)
    	at java.lang.Thread.run(Thread.java:748)
    
       Locked ownable synchronizers:
    	- None
    ...
    
    "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f90701b2640 nid=0x5193 in Object.wait() [0x00007f9074065000]
       java.lang.Thread.State: WAITING (on object monitor)
    	at java.lang.Object.wait(Native Method)
    	- waiting on <0x00000006d918e910> (a java.lang.ref.Reference$Lock)
    	at java.lang.Object.wait(Object.java:502)
    	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
    	- locked <0x00000006d918e910> (a java.lang.ref.Reference$Lock)
    	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
    
       Locked ownable synchronizers:
    	- None
    
    "VM Thread" os_prio=0 tid=0x00007f90701a9200 nid=0x5191 runnable 
    
    "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f907001c630 nid=0x5127 runnable 
    
    "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f907001db40 nid=0x5128 runnable 
    
    "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f907001f050 nid=0x5129 runnable 
    
    "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f9070020560 nid=0x512a runnable 
    
    "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f9070062c20 nid=0x518f runnable 
    
    "VM Periodic Task Thread" os_prio=0 tid=0x00007f9070345590 nid=0x519d waiting on condition 
    
    JNI global references: 1050
    
    [cassandra@VM_129_3_centos tmp]$ jstack -l 20772

10进制的进程号20772转成16进制是0x5124，在上面的结果中找到它

3. 查看堆内存jmap -heap PID

[cassandra@VM_129_3_centos tmp]$ jmap -heap 20772
    Attaching to process ID 20772, please wait...
    Debugger attached successfully.
    Server compiler detected.
    JVM version is 25.172-b11
    
    using parallel threads in the new generation.
    using thread-local object allocation.
    Concurrent Mark-Sweep GC
    
    Heap Configuration:
       MinHeapFreeRatio         = 40
       MaxHeapFreeRatio         = 70
       MaxHeapSize              = 4294967296 (4096.0MB)
       NewSize                  = 419430400 (400.0MB)
       MaxNewSize               = 419430400 (400.0MB)
       OldSize                  = 3875536896 (3696.0MB)
       NewRatio                 = 2
       SurvivorRatio            = 8
       MetaspaceSize            = 21807104 (20.796875MB)
       CompressedClassSpaceSize = 1073741824 (1024.0MB)
       MaxMetaspaceSize         = 17592186044415 MB
       G1HeapRegionSize         = 0 (0.0MB)
    
    Heap Usage:
    New Generation (Eden + 1 Survivor Space):
       capacity = 377487360 (360.0MB)
       used     = 348288016 (332.15333557128906MB)
       free     = 29199344 (27.846664428710938MB)
       92.26481543646918% used
    Eden Space:
       capacity = 335544320 (320.0MB)
       used     = 318190024 (303.4496536254883MB)
       free     = 17354296 (16.55034637451172MB)
       94.82801675796509% used
    From Space:
       capacity = 41943040 (40.0MB)
       used     = 30097992 (28.70368194580078MB)
       free     = 11845048 (11.296318054199219MB)
       71.75920486450195% used
    To Space:
       capacity = 41943040 (40.0MB)
       used     = 0 (0.0MB)
       free     = 41943040 (40.0MB)
       0.0% used
    concurrent mark-sweep generation:
       capacity = 3875536896 (3696.0MB)
       used     = 79467816 (75.78641510009766MB)
       free     = 3796069080 (3620.2135848999023MB)
       2.050498244050261% used
    
    28574 interned Strings occupying 5184216 bytes.
    [cassandra@VM_129_3_centos tmp]$

未完待续...

注意事项：

如果jstack -l PID命令返回“Unable to open socket file: ”，则需要参考[Cassandra官方文档][Cassandra]指定CASSANDRA\_HEAPDUMP\_DIR解决。

参考：[https://alexzeng.wordpress.com/2013/05/25/debug-cassandrar-jvm-thread-100-cpu-usage-issue/][https_alexzeng.wordpress.com_2013_05_25_debug-cassandrar-jvm-thread-100-cpu-usage-issue]

[Cassandra]: https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/config/configHeapDump.html
[https_alexzeng.wordpress.com_2013_05_25_debug-cassandrar-jvm-thread-100-cpu-usage-issue]: https://alexzeng.wordpress.com/2013/05/25/debug-cassandrar-jvm-thread-100-cpu-usage-issue/