中间件Tomcat进程服务无法关闭问题分析.docx

Ta10nK1n9

云和恩墨

181

7页

1次

2022-09-19

5墨值下载

问题现象

Tomcat

进行在执行

shutdown.sh

脚本后无法停止，

JMX

端口一直被占用，

导致再次启动

Tomcat

服务时会报出

JMX

端口已被占用的错误信息，经过排查

发现出现该问题的应用都使用了

Dubbo

框架，涉及到

Dubbo

优雅停机服务问题。

错误日志

Tomcat

运行

shutdown.sh

之后，

catalina.out

输出错误日志如下：

17-Jul-2020 17:47:22.348 WARNING [localhost-startStop-2]

org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [iot-

app] appears to have started a thread named [DubboRegistryFailedRetryTimer-thread-1] but has failed to

stop it. This is very likely to create a memory leak. Stack trace of thread:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchroni

zer.java:2039)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecu

tor.java:1081)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecu

tor.java:809)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

java.lang.Thread.run(Thread.java:748)

17-Jul-2020 17:47:22.349 WARNING [localhost-startStop-2]

org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [iot-

app] appears to have started a thread named [DubboSaveRegistryCache-thread-1] but has failed to stop it.

This is very likely to create a memory leak. Stack trace of thread:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchroni

zer.java:2039)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

java.lang.Thread.run(Thread.java:748)

17-Jul-2020 17:47:22.350 WARNING [localhost-startStop-2]

org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [iot-

app] appears to have started a thread named [DubboRegistryFailedRetryTimer-thread-1] but has failed to

stop it. This is very likely to create a memory leak. Stack trace of thread:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSyn

chronizer.java:2078)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecu

tor.java:1093)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecu

tor.java:809)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

java.lang.Thread.run(Thread.java:748)

问题原因

看

com.alibaba.dubbo.remoting.transport.netty.NettyClient

中的两段代码：

因

ChannelFactory

的关闭有

DirectMemory

泄露，采用静态化规避

//https://issues.jboss.org/browse/NETTY-424

private static final ChannelFactory channelFactory = new

NioClientSocketChannelFactory(Executors.newCachedThreadPool(new

NamedThreadFactory("NettyClientBoss", true)), Executors.newCachedThreadPool(new

NamedThreadFactory("NettyClientWorker", true)), Constants.DEFAULT_IO_THREADS);

上面是为

netty

创建

boss

线程池和

worker

线程池。注释里提到了一个

netty

内存

泄露的

bug

，在大量创建和关闭

channelFactory

时会引起堆外内存的大量泄露，

zer.java:2039)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

java.lang.Thread.run(Thread.java:748)

17-Jul-2020 17:47:22.350 WARNING [localhost-startStop-2]

org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [iot-

app] appears to have started a thread named [DubboRegistryFailedRetryTimer-thread-1] but has failed to

stop it. This is very likely to create a memory leak. Stack trace of thread:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSyn

chronizer.java:2078)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecu

tor.java:1093)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecu

tor.java:809)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

java.lang.Thread.run(Thread.java:748)

问题原因

看

com.alibaba.dubbo.remoting.transport.netty.NettyClient

中的两段代码：

因

ChannelFactory

的关闭有

DirectMemory

泄露，采用静态化规避

//https://issues.jboss.org/browse/NETTY-424

private static final ChannelFactory channelFactory = new

NioClientSocketChannelFactory(Executors.newCachedThreadPool(new

NamedThreadFactory("NettyClientBoss", true)), Executors.newCachedThreadPool(new

NamedThreadFactory("NettyClientWorker", true)), Constants.DEFAULT_IO_THREADS);

上面是为

netty

创建

boss

线程池和

worker

线程池。注释里提到了一个

netty

内存

泄露的

bug

，在大量创建和关闭

channelFactory

时会引起堆外内存的大量泄露，

of 7

5墨值下载

关注

评论