记服务器的cpu满载的一次排查过程1 问题描述2 排查思路3 通过打印线程Dump排查

69 阅读 0 评论 46 点赞

我是靠谱客的博主大气水杯，最近开发中收集的这篇文章主要介绍记服务器的cpu满载的一次排查过程1 问题描述2 排查思路3 通过打印线程Dump排查，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

The article summary

1 问题描述
2 排查思路
- 2.1 查看整机的状态
- 2.2 其它排查方法
3 通过打印线程Dump排查

1 问题描述

补充参考链接：https://www.cnblogs.com/likethis/p/11687711.html
今天在对代码测试，将代码打包部署到测试服务器后，出现cpu满载（几乎接近100%），然后就开始了自己的排查之路，因为现在服务器上的代码已经修复，有些场景不能复现。

2 排查思路

2.1 查看整机的状态

使用top 命令，查看整机状态。
在这里插入图片描述
上面的这张图是已经修复好的代码。主要看的参数是这几个：

load average：0.39, 0.45, 0.43：分别表示服务启动1分钟、5分钟、15分钟的平均负载。现在服务已经修复好，都不是很大，比较正常（记得自己排查的时候，各项参数都快接近10了）。
%Cpu（s）：该行的数据表示cpu的使用情况，13.2us表示使用的率，86.1id表示空闲率。
再看下面具体的进程，看哪一个进程的cpu占用率高，找到这个高的进程id，通过命令top -Hp 进程id，如：top -Hp 1420，就可以看到进程中具体的线程的使用情况。效果如下：

到这里我大概就知道导致cpu满载的原因（上面的线程都是kafka的线程），但是具体是什么原因（具体代码）导致的还不能够确定

2.2 其它排查方法

通过上面的方式基本能确定出错的大概原因，如果还是不能，还可以能过下面的方式。

使用 vmstat -n 2 3查看cpu 在这里插入图片描述

vmstat -n 2 3
2: 每两秒获取一次
3: 共获取三次
us:用户使用占比
sy:系统使用占比
id:闲置占比（id是idle的缩写，是空闲的意思）

使用free -m查看内存的使用情况，-m表示查询出来的单位是兆
使用df -h命令查看硬盘的使用情况

使用iostat -d 2 3查看磁盘io的情况
在这里插入图片描述

iostat -d 2 3: 打印出磁盘io信息
tps: 每分钟提交事务的次数,也就是向磁盘写数据的次数(增册改)
qps: 每分钟查询磁盘的次数
pv: PageView页面浏览的次数

到这里问题的大概方向基本能定位到，但是如何知道具体是哪些代码导致的还需要堆栈信息来排查。

3 通过打印线程Dump排查

jdk给我们提供了一些工具，帮助我们来排查问题。如jstack pid使用可以打印指定线程的信息，但是要执行该命令必须要到jdk的bin目录下执行才有效，那如何找到linux下的jdk安装目录及打印线程信息。

java -version命令查看jdk版本

[root@alpha-5g-1 ~]# java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)

which java:首先要申明一下which java是定位不到安装路径的。which java定位到的是java程序的执行路径。
```
[root@alpha-5g-1 bin]# which java
/usr/bin/java
```

ls -lrt /usr/bin/java

[root@alpha-5g-1 bin]# ls -lrt /usr/bin/java
lrwxrwxrwx 1 root root 22 Aug 20  2019 /usr/bin/java -> /etc/alternatives/java

ls -lrt /etc/alternatives/java

[root@alpha-5g-1 bin]# ls -lrt /etc/alternatives/java
lrwxrwxrwx 1 root root 73 Aug 20  2019 /etc/alternatives/java -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el7_6.x86_64/jre/bin/java

我们只需要进入到bin目录下就可以

执行jstack 进程ip号，如：jstack 21293
也可以把堆栈信息输出到一个文件中jstack -l 线程id >> 文件名，如：jstack -l 21293 >> d:log.txt

		2020-07-01 10:00:10
	Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.91-b15 mixed mode):
	
	"DestroyJavaVM" #68 prio=5 os_prio=0 tid=0x000000001bdaf000 nid=0x54bc waiting on condition [0x0000000000000000]
	   java.lang.Thread.State: RUNNABLE
	
	   Locked ownable synchronizers:
		- None
	
	"http-nio-8083-AsyncTimeout" #66 daemon prio=5 os_prio=0 tid=0x000000001bdb1800 nid=0x5d00 waiting on condition [0x000000002118f000]
	   java.lang.Thread.State: TIMED_WAITING (sleeping)
		at java.lang.Thread.sleep(Native Method)
		at org.apache.coyote.AbstractProtocol$AsyncTimeout.run(AbstractProtocol.java:1149)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"http-nio-8083-Acceptor-0" #65 daemon prio=5 os_prio=0 tid=0x000000001bdb0000 nid=0x62a8 runnable [0x000000002108f000]
	   java.lang.Thread.State: RUNNABLE
		at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
		at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
		at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
		- locked <0x00000000db5f35c8> (a java.lang.Object)
		at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:482)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"http-nio-8083-ClientPoller-1" #64 daemon prio=5 os_prio=0 tid=0x000000001bdb6000 nid=0x5f84 runnable [0x0000000020f8e000]
	   java.lang.Thread.State: RUNNABLE
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
		at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
		- locked <0x00000000dc4874b0> (a sun.nio.ch.Util$2)
		- locked <0x00000000dc4874a0> (a java.util.Collections$UnmodifiableSet)
		- locked <0x00000000dc487350> (a sun.nio.ch.WindowsSelectorImpl)
		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
		at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:825)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"http-nio-8083-ClientPoller-0" #63 daemon prio=5 os_prio=0 tid=0x000000001bdb3800 nid=0x5ea4 runnable [0x0000000020e8e000]
	   java.lang.Thread.State: RUNNABLE
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
		at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
		- locked <0x00000000dc4832c0> (a sun.nio.ch.Util$2)
		- locked <0x00000000dc4832b0> (a java.util.Collections$UnmodifiableSet)
		- locked <0x00000000dc483160> (a sun.nio.ch.WindowsSelectorImpl)
		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
		at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:825)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"http-nio-8083-exec-10" #62 daemon prio=5 os_prio=0 tid=0x000000001a070000 nid=0x4990 waiting on condition [0x0000000020d8e000]
	   java.lang.Thread.State: WAITING (parking)
		at sun.misc.Unsafe.park(Native Method)
		- parking to wait for  <0x00000000dc39dea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"http-nio-8083-exec-9" #61 daemon prio=5 os_prio=0 tid=0x000000001a06e800 nid=0x4e68 waiting on condition [0x0000000020c8e000]
	   java.lang.Thread.State: WAITING (parking)
		at sun.misc.Unsafe.park(Native Method)
		- parking to wait for  <0x00000000dc39dea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"http-nio-8083-exec-8" #60 daemon prio=5 os_prio=0 tid=0x000000001a06e000 nid=0x4f80 waiting on condition [0x0000000020b8e000]
	   java.lang.Thread.State: WAITING (parking)
		at sun.misc.Unsafe.park(Native Method)
		- parking to wait for  <0x00000000dc39dea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"http-nio-8083-exec-7" #59 daemon prio=5 os_prio=0 tid=0x000000001a06d000 nid=0x3e14 waiting on condition [0x0000000020a8e000]
	   java.lang.Thread.State: WAITING (parking)
		at sun.misc.Unsafe.park(Native Method)
		- parking to wait for  <0x00000000dc39dea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"http-nio-8083-exec-6" #58 daemon prio=5 os_prio=0 tid=0x000000001a06c800 nid=0x536c waiting on condition [0x000000002098e000]
	   java.lang.Thread.State: WAITING (parking)
		at sun.misc.Unsafe.park(Native Method)
		- parking to wait for  <0x00000000dc39dea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"http-nio-8083-exec-5" #57 daemon prio=5 os_prio=0 tid=0x000000001a06b800 nid=0x62ac waiting on condition [0x000000002088e000]
	   java.lang.Thread.State: WAITING (parking)
		at sun.misc.Unsafe.park(Native Method)
		- parking to wait for  <0x00000000dc39dea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"http-nio-8083-exec-4" #56 daemon prio=5 os_prio=0 tid=0x000000001a06f800 nid=0x5d8c waiting on condition [0x000000002078f000]
	   java.lang.Thread.State: WAITING (parking)
		at sun.misc.Unsafe.park(Native Method)
		- parking to wait for  <0x00000000dc39dea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"http-nio-8083-exec-3" #55 daemon prio=5 os_prio=0 tid=0x000000001a064000 nid=0x4748 waiting on condition [0x000000002068e000]
	   java.lang.Thread.State: WAITING (parking)
		at sun.misc.Unsafe.park(Native Method)
		- parking to wait for  <0x00000000dc39dea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"http-nio-8083-exec-2" #54 daemon prio=5 os_prio=0 tid=0x000000001a06a000 nid=0x5a10 waiting on condition [0x000000002058f000]
	   java.lang.Thread.State: WAITING (parking)
		at sun.misc.Unsafe.park(Native Method)
		- parking to wait for  <0x00000000dc39dea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"http-nio-8083-exec-1" #53 daemon prio=5 os_prio=0 tid=0x000000001a069800 nid=0x2ae8 waiting on condition [0x000000002048f000]
	   java.lang.Thread.State: WAITING (parking)
		at sun.misc.Unsafe.park(Native Method)
		- parking to wait for  <0x00000000dc39dea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
		at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"NioBlockingSelector.BlockPoller-1" #52 daemon prio=5 os_prio=0 tid=0x000000001a06b000 nid=0x5f0c runnable [0x000000002038e000]
	   java.lang.Thread.State: RUNNABLE
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
		at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
		- locked <0x00000000db5f3a00> (a sun.nio.ch.Util$2)
		- locked <0x00000000db5f39f0> (a java.util.Collections$UnmodifiableSet)
		- locked <0x00000000db5f38a0> (a sun.nio.ch.WindowsSelectorImpl)
		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
		at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:298)
	
	   Locked ownable synchronizers:
		- None
	
	"qosGroup-4-C-1" #51 prio=5 os_prio=0 tid=0x000000001a068800 nid=0x5ff4 runnable [0x000000002028e000]
	   java.lang.Thread.State: RUNNABLE
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
		at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
		- locked <0x00000000db567820> (a sun.nio.ch.Util$2)
		- locked <0x00000000db567810> (a java.util.Collections$UnmodifiableSet)
		- locked <0x00000000db5676c0> (a sun.nio.ch.WindowsSelectorImpl)
		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
		at org.apache.kafka.common.network.Selector.select(Selector.java:470)
		at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
		at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
		- locked <0x00000000db576dd8> (a org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:209)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:148)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:136)
		at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:197)
		- locked <0x00000000db577028> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
		at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)
		at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
		at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
		at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
		at java.util.concurrent.FutureTask.run(FutureTask.java)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"qosGroup-3-C-1" #50 prio=5 os_prio=0 tid=0x000000001a061000 nid=0x5e54 runnable [0x000000002018e000]
	   java.lang.Thread.State: RUNNABLE
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
		at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
		- locked <0x00000000db5341c8> (a sun.nio.ch.Util$2)
		- locked <0x00000000db5341b8> (a java.util.Collections$UnmodifiableSet)
		- locked <0x00000000db534068> (a sun.nio.ch.WindowsSelectorImpl)
		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
		at org.apache.kafka.common.network.Selector.select(Selector.java:470)
		at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
		at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
		- locked <0x00000000db543700> (a org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:209)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:148)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:136)
		at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:197)
		- locked <0x00000000db543950> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
		at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)
		at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
		at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
		at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
		at java.util.concurrent.FutureTask.run(FutureTask.java)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"qosGroup-2-C-1" #49 prio=5 os_prio=0 tid=0x000000001a063800 nid=0x4b00 runnable [0x000000002008e000]
	   java.lang.Thread.State: RUNNABLE
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
		at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
		- locked <0x00000000db500b48> (a sun.nio.ch.Util$2)
		- locked <0x00000000db500b38> (a java.util.Collections$UnmodifiableSet)
		- locked <0x00000000db5009e8> (a sun.nio.ch.WindowsSelectorImpl)
		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
		at org.apache.kafka.common.network.Selector.select(Selector.java:470)
		at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
		at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
		- locked <0x00000000db5100a0> (a org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:209)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:148)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:136)
		at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:197)
		- locked <0x00000000db5102f0> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
		at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)
		at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
		at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
		at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
		at java.util.concurrent.FutureTask.run(FutureTask.java)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"qosGroup-1-C-1" #48 prio=5 os_prio=0 tid=0x000000001a068000 nid=0x3d74 runnable [0x000000001ff8e000]
	   java.lang.Thread.State: RUNNABLE
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
		at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
		- locked <0x00000000db4ca5e0> (a sun.nio.ch.Util$2)
		- locked <0x00000000db4ca5d0> (a java.util.Collections$UnmodifiableSet)
		- locked <0x00000000db4ca480> (a sun.nio.ch.WindowsSelectorImpl)
		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
		at org.apache.kafka.common.network.Selector.select(Selector.java:470)
		at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
		at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
		- locked <0x00000000db4d9b98> (a org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:209)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:148)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:136)
		at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:197)
		- locked <0x00000000db4d9de8> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
		at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)
		at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
		at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
		at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
		at java.util.concurrent.FutureTask.run(FutureTask.java)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"qosGroup-0-C-1" #47 prio=5 os_prio=0 tid=0x000000001a066800 nid=0x2848 runnable [0x000000001fd8e000]
	   java.lang.Thread.State: RUNNABLE
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
		at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
		- locked <0x00000000db496fb8> (a sun.nio.ch.Util$2)
		- locked <0x00000000db496fa8> (a java.util.Collections$UnmodifiableSet)
		- locked <0x00000000db496e58> (a sun.nio.ch.WindowsSelectorImpl)
		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
		at org.apache.kafka.common.network.Selector.select(Selector.java:470)
		at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
		at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
		- locked <0x00000000db4a6510> (a org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:209)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:148)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:136)
		at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:197)
		- locked <0x00000000db4a6760> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
		at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)
		at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
		at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
		at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
		at java.util.concurrent.FutureTask.run(FutureTask.java)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"nefGroup-4-C-1" #46 prio=5 os_prio=0 tid=0x000000001a062800 nid=0x4d70 runnable [0x000000001fc8e000]
	   java.lang.Thread.State: RUNNABLE
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
		at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
		- locked <0x00000000db4635f0> (a sun.nio.ch.Util$2)
		- locked <0x00000000db4635e0> (a java.util.Collections$UnmodifiableSet)
		- locked <0x00000000db463490> (a sun.nio.ch.WindowsSelectorImpl)
		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
		at org.apache.kafka.common.network.Selector.select(Selector.java:470)
		at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
		at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
		- locked <0x00000000db472b08> (a org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:209)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:148)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:136)
		at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:197)
		- locked <0x00000000db472d58> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
		at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)
		at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
		at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
		at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
		at java.util.concurrent.FutureTask.run(FutureTask.java)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"nefGroup-3-C-1" #45 prio=5 os_prio=0 tid=0x000000001a067000 nid=0x5478 runnable [0x000000001fb8e000]
	   java.lang.Thread.State: RUNNABLE
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
		at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
		- locked <0x00000000db430008> (a sun.nio.ch.Util$2)
		- locked <0x00000000db42fff8> (a java.util.Collections$UnmodifiableSet)
		- locked <0x00000000db42fea8> (a sun.nio.ch.WindowsSelectorImpl)
		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
		at org.apache.kafka.common.network.Selector.select(Selector.java:470)
		at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
		at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
		- locked <0x00000000db43f540> (a org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:209)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:148)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:136)
		at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:197)
		- locked <0x00000000db43f790> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
		at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)
		at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
		at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
		at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
		at java.util.concurrent.FutureTask.run(FutureTask.java)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"nefGroup-2-C-1" #44 prio=5 os_prio=0 tid=0x000000001a062000 nid=0x4d30 runnable [0x000000001fa8e000]
	   java.lang.Thread.State: RUNNABLE
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
		at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
		- locked <0x00000000db3fc950> (a sun.nio.ch.Util$2)
		- locked <0x00000000db3fc940> (a java.util.Collections$UnmodifiableSet)
		- locked <0x00000000db3fc7f0> (a sun.nio.ch.WindowsSelectorImpl)
		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
		at org.apache.kafka.common.network.Selector.select(Selector.java:470)
		at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
		at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
		- locked <0x00000000db40bea8> (a org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:209)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:148)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:136)
		at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:197)
		- locked <0x00000000db40c0f8> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
		at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)
		at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
		at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
		at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
		at java.util.concurrent.FutureTask.run(FutureTask.java)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"nefGroup-1-C-1" #43 prio=5 os_prio=0 tid=0x000000001a065000 nid=0x32e4 runnable [0x000000001f98e000]
	   java.lang.Thread.State: RUNNABLE
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
		at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
		- locked <0x00000000db3c9348> (a sun.nio.ch.Util$2)
		- locked <0x00000000db3c9338> (a java.util.Collections$UnmodifiableSet)
		- locked <0x00000000db3c91e8> (a sun.nio.ch.WindowsSelectorImpl)
		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
		at org.apache.kafka.common.network.Selector.select(Selector.java:470)
		at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
		at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
		- locked <0x00000000db3d88e0> (a org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:209)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:148)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:136)
		at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:197)
		- locked <0x00000000db3d8b30> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
		at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)
		at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
		at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
		at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
		at java.util.concurrent.FutureTask.run(FutureTask.java)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"nefGroup-0-C-1" #42 prio=5 os_prio=0 tid=0x000000001a065800 nid=0x5b00 runnable [0x000000001f88e000]
	   java.lang.Thread.State: RUNNABLE
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
		at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
		at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
		- locked <0x00000000dae3e178> (a sun.nio.ch.Util$2)
		- locked <0x00000000dae3d0d0> (a java.util.Collections$UnmodifiableSet)
		- locked <0x00000000dae3a040> (a sun.nio.ch.WindowsSelectorImpl)
		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
		at org.apache.kafka.common.network.Selector.select(Selector.java:470)
		at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
		at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
		- locked <0x00000000db158560> (a org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:209)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:148)
		at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:136)
		at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:197)
		- locked <0x00000000db1f5940> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
		at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)
		at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
		at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
		at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
		at java.util.concurrent.FutureTask.run(FutureTask.java)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"Druid-ConnectionPool-Destroy-111832087" #37 daemon prio=5 os_prio=0 tid=0x000000001bfb7000 nid=0x5a30 waiting on condition [0x000000001dfbf000]
	   java.lang.Thread.State: TIMED_WAITING (sleeping)
		at java.lang.Thread.sleep(Native Method)
		at com.alibaba.druid.pool.DruidDataSource$DestroyConnectionThread.run(DruidDataSource.java:2540)
	
	   Locked ownable synchronizers:
		- None
	
	"Druid-ConnectionPool-Create-111832087" #36 daemon prio=5 os_prio=0 tid=0x000000001bfb6800 nid=0x6104 waiting on condition [0x000000001debf000]
	   java.lang.Thread.State: WAITING (parking)
		at sun.misc.Unsafe.park(Native Method)
		- parking to wait for  <0x00000000833b9ba0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
		at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2443)
	
	   Locked ownable synchronizers:
		- None
	
	"Abandoned connection cleanup thread" #35 daemon prio=5 os_prio=0 tid=0x000000001bfb5800 nid=0x4fd4 in Object.wait() [0x000000001ddbf000]
	   java.lang.Thread.State: TIMED_WAITING (on object monitor)
		at java.lang.Object.wait(Native Method)
		at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
		- locked <0x0000000083629340> (a java.lang.ref.ReferenceQueue$Lock)
		at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- <0x000000008350ffe0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
	
	"commons-pool-evictor-thread" #33 prio=5 os_prio=0 tid=0x000000001bfb5000 nid=0x570c waiting on condition [0x000000001dabe000]
	   java.lang.Thread.State: TIMED_WAITING (parking)
		at sun.misc.Unsafe.park(Native Method)
		- parking to wait for  <0x0000000083515b68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
		at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
		at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"container-0" #31 prio=5 os_prio=0 tid=0x000000001bfb4000 nid=0x5d6c waiting on condition [0x000000001d9be000]
	   java.lang.Thread.State: TIMED_WAITING (sleeping)
		at java.lang.Thread.sleep(Native Method)
		at org.apache.catalina.core.StandardServer.await(StandardServer.java:427)
		at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer$2.run(TomcatEmbeddedServletContainer.java:192)
	
	   Locked ownable synchronizers:
		- None
	
	"ContainerBackgroundProcessor[StandardEngine[Tomcat]]" #30 daemon prio=5 os_prio=0 tid=0x000000001bfb9800 nid=0x2a80 waiting on condition [0x000000001d8bf000]
	   java.lang.Thread.State: TIMED_WAITING (sleeping)
		at java.lang.Thread.sleep(Native Method)
		at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1359)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"RMI Scheduler(0)" #19 daemon prio=5 os_prio=0 tid=0x000000001a1ff000 nid=0x604c waiting on condition [0x000000001b2ee000]
	   java.lang.Thread.State: TIMED_WAITING (parking)
		at sun.misc.Unsafe.park(Native Method)
		- parking to wait for  <0x000000008233f3f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
		at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
		at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"RMI TCP Accept-0" #16 daemon prio=5 os_prio=0 tid=0x0000000019fd6000 nid=0xce8 runnable [0x000000001ac3e000]
	   java.lang.Thread.State: RUNNABLE
		at java.net.DualStackPlainSocketImpl.accept0(Native Method)
		at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
		at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
		at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
		- locked <0x000000008233f790> (a java.net.SocksSocketImpl)
		at java.net.ServerSocket.implAccept(ServerSocket.java:545)
		at java.net.ServerSocket.accept(ServerSocket.java:513)
		at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
		at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
		at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"RMI TCP Accept-58476" #15 daemon prio=5 os_prio=0 tid=0x0000000019f74000 nid=0x57cc runnable [0x000000001ab3f000]
	   java.lang.Thread.State: RUNNABLE
		at java.net.DualStackPlainSocketImpl.accept0(Native Method)
		at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
		at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
		at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
		- locked <0x000000008233fc38> (a java.net.SocksSocketImpl)
		at java.net.ServerSocket.implAccept(ServerSocket.java:545)
		at java.net.ServerSocket.accept(ServerSocket.java:513)
		at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
		at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"RMI TCP Accept-0" #14 daemon prio=5 os_prio=0 tid=0x0000000019f6d000 nid=0x3c20 runnable [0x000000001aa3e000]
	   java.lang.Thread.State: RUNNABLE
		at java.net.DualStackPlainSocketImpl.accept0(Native Method)
		at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
		at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
		at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
		- locked <0x000000008232b990> (a java.net.SocksSocketImpl)
		at java.net.ServerSocket.implAccept(ServerSocket.java:545)
		at java.net.ServerSocket.accept(ServerSocket.java:513)
		at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
		at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
		at java.lang.Thread.run(Thread.java:745)
	
	   Locked ownable synchronizers:
		- None
	
	"Service Thread" #12 daemon prio=9 os_prio=0 tid=0x0000000019e3b800 nid=0x1590 runnable [0x0000000000000000]
	   java.lang.Thread.State: RUNNABLE
	
	   Locked ownable synchronizers:
		- None
	
	"C1 CompilerThread2" #11 daemon prio=9 os_prio=2 tid=0x0000000019dab800 nid=0x5904 waiting on condition [0x0000000000000000]
	   java.lang.Thread.State: RUNNABLE
	
	   Locked ownable synchronizers:
		- None
	
	"C2 CompilerThread1" #10 daemon prio=9 os_prio=2 tid=0x0000000019da8800 nid=0x1fe0 waiting on condition [0x0000000000000000]
	   java.lang.Thread.State: RUNNABLE
	
	   Locked ownable synchronizers:
		- None
	
	"C2 CompilerThread0" #9 daemon prio=9 os_prio=2 tid=0x0000000019da7800 nid=0x54a8 waiting on condition [0x0000000000000000]
	   java.lang.Thread.State: RUNNABLE
	
	   Locked ownable synchronizers:
		- None
	
	"JDWP Command Reader" #8 daemon prio=10 os_prio=0 tid=0x00000000194a8000 nid=0x4338 runnable [0x0000000000000000]
	   java.lang.Thread.State: RUNNABLE
	
	   Locked ownable synchronizers:
		- None
	
	"JDWP Event Helper Thread" #7 daemon prio=10 os_prio=0 tid=0x00000000194a5000 nid=0x52d4 runnable [0x0000000000000000]
	   java.lang.Thread.State: RUNNABLE
	
	   Locked ownable synchronizers:
		- None
	
	"JDWP Transport Listener: dt_socket" #6 daemon prio=10 os_prio=0 tid=0x0000000019498800 nid=0x5670 runnable [0x0000000000000000]
	   java.lang.Thread.State: RUNNABLE
	
	   Locked ownable synchronizers:
		- None
	
	"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x000000001944a800 nid=0x6298 waiting on condition [0x0000000000000000]
	   java.lang.Thread.State: RUNNABLE
	
	   Locked ownable synchronizers:
		- None
	
	"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000000019496800 nid=0x6164 runnable [0x0000000000000000]
	   java.lang.Thread.State: RUNNABLE
	
	   Locked ownable synchronizers:
		- None
	
	"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x0000000017ac2000 nid=0x5a84 in Object.wait() [0x000000001943f000]
	   java.lang.Thread.State: WAITING (on object monitor)
		at java.lang.Object.wait(Native Method)
		- waiting on <0x0000000082373230> (a java.lang.ref.ReferenceQueue$Lock)
		at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
		- locked <0x0000000082373230> (a java.lang.ref.ReferenceQueue$Lock)
		at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
		at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
	
	   Locked ownable synchronizers:
		- None
	
	"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x0000000002bc9000 nid=0x5450 in Object.wait() [0x000000001933e000]
	   java.lang.Thread.State: WAITING (on object monitor)
		at java.lang.Object.wait(Native Method)
		- waiting on <0x000000008228da98> (a java.lang.ref.Reference$Lock)
		at java.lang.Object.wait(Object.java:502)
		at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
		- locked <0x000000008228da98> (a java.lang.ref.Reference$Lock)
		at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
	
	   Locked ownable synchronizers:
		- None
	
	"VM Thread" os_prio=2 tid=0x0000000017a97800 nid=0x3ee0 runnable 
	
	"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x0000000002aed000 nid=0x56c8 runnable 
	
	"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000000002aef800 nid=0x6380 runnable 
	
	"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000000002af1000 nid=0x4890 runnable 
	
	"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000000002af2800 nid=0x584c runnable 
	
	"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x0000000002af5000 nid=0x6170 runnable 
	
	"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x0000000002af6000 nid=0x4518 runnable 
	
	"VM Periodic Task Thread" os_prio=2 tid=0x0000000019fd7800 nid=0x5774 waiting on condition 
	
	JNI global references: 21284
	
	```

将信息打印出来就可以查看信息，一般看线程的状态，及锁的获取释放会不会有死锁的情况，如果线程数比较多，看起来还是有点费力的。打印出来的堆栈信息每一个选项什么意思，我就不一一解释了，百度一下就可以，比较简单。下面我把有问题的线程的信息帖上图。
图一：图一想要获取的锁被图二的线程锁住了

图二：

图三：图三想要获取的锁被图四的线程锁住了

图四：

问题说明：kafka消费者消费消息的时间间隔太小了，设置大一点就好了。